Soft Skills Engineering - Episode 79: Story Point Misses and Measuring Productivity

Episode Date: October 19, 2017

This week Jamison and Dave answer these questions: It seems like my teams always miss their story point commitments. Is this normal? How do you change it? How do you actually measure developer productivity? The article comparing research on productivity in static and dynamic type systems is here. It is a great read. Jamison also mentions Goodhart’s Law. Read more about it here.

It takes more than great UML diagrams to be a great software engineer. This is episode 79 of the Soft Skills Engineering Podcast. I'm your host, Jameson Dance. I'm your host, Dave Smith. Are UML diagrams a joke to you, Dave? Do you not understand how serious this is? No, sir. They're not a joke.

You put a triangle where you should have put a diamond and people died. oh my gosh oh uml i'm pretty sure i read that book and then promptly forgot all of it and now my diagrams are they're boxes with lines and that's sometimes they have lines in the boxes too but no diamonds or arrows no no oh boy it's like abstract uml it's like an impressionist uml yeah exactly yeah all of the boxes are made up of tiny little boxes look when you master the art form you can you are then entitled to break the rules yeah that's true you can play with it a little bit once you have mastered it like i clearly have this show is an advice show for software developers and other technical people

about non-technical things as you can clearly tell from the intro and we have a question do you want to take it away dave yeah this comes from a listener named jameson jameson is this you uh it depends on whether it makes me look good or bad okay well i'll let you judge that after i read the question okay it says every team i've ever been on fails horribly at delivering the story points they commit to for each sprint this feels really bad how do we fix this uh that's gotta be old slacker jameson oh yeah me not the one that you might want to hire no no that's that's all lay about jameson everybody knows he just spends his time down at the dump drinking moonshine and not hitting his story points uh yeah that is me this

is the first time that we've openly asked ourselves a question yeah i would like to apologize to all the listeners whose questions we are not answering in order to answer my own question but dave and i just got started talking about this and and i really wanted to talk more about it on the air because i feel like while bad it is not uncommon yes or maybe i don't even know if while bad is the right thing to say here it's definitely not uncommon though well the part where you feel bad is bad yeah that is bad um i remember one one oh go ahead that's gonna say but you could fix the feel bad part by just changing your expectations a little bit and say when we fail to meet our commitments for the sprint that's when we should feel good problem solved i thought you

would say by just not caring about not meeting your commitments for the sprint oh that's old junkyard jameson talking if you said that i would say that's what i did yeah i responded with nihilism where story points it's like did you ever watch whose line is it anyway oh yeah absolutely you know how he says like welcome to whose line is it anyway where the points don't matter and everything's made up or something like that yes you are the second person to quote that line to me this week about story points uh yes it literally was about story points i was in like a sprint planning meeting a couple days ago and someone said that on my team that's been in my head for years and i'm sure i'm not the only well i know i'm not the only person

there's at least one other person i'm sure there's many there are many more people though yeah but the crazy thing is the timing it just popped out for you and this other guy in the same week maybe i am this other man also oh i didn't know you do look alike okay anyway sorry you were saying points don't matter yeah well i guess there are a lot of ways you could deal with this one one way is to try and get better at estimation and another way is to say this is clearly a hoop we just jumped through that doesn't mean anything and i'm just gonna not care as much uh and i've tried both ways and the end result is the same so far meaning you still don't deliver on your commitments whether you care yeah yeah

and i still feel bad either i feel bad about checking out of this meeting every week or i feel bad about failing to get better at estimation so if feeling bad is a given what can you do to mitigate the rest of your broken life i don't know i'm hoping you could tell me well i'll quote my former cto who said to me uh that he thinks that story points are the greatest invention in software estimation of all time he knows or believes something that i do not know or believe well did you ever have to estimate software by hour uh yeah actually i did and did that go better or worse or the same as story points it went the same which is every time you made an estimate it was off by a factor of 10 yeah it was it was way off and then we just failed oh look my my my

completely wrong estimates are still off by a factor of 10 with different units yeah exactly it was and the the the decision to use hours instead of story points was that basically it all comes down to time anyways so why don't we just cut out the middleman and and then it didn't make it any better cutting out the middleman usually works i don't get it that was the end man you cut out the end oh no huh can you can you explain why your cto felt this just disassociating estimates with time he's that's the reason i usually hear yeah he basically said that something some switch flips in a developer's mind when the developer is trying to determine how much time it would take to implement something compared to just estimating the

complexity of something and by putting a point number on the complexity you get more accurate time estimates when considered in aggregate so basically he said as a team everyone estimates their stories they commit to a set of stories they deliver and then as you deliver those story points can become a velocity which will predict future deliveries in aggregate for the team over time you know any single data point can have high variance either a single person or a single sprint for the whole team but that over time the velocity can serve as a useful predictor of total software delivery yeah that's the dream and while you're saying that i realized that one of the issues is that i've very rarely been on a team that will then calibrate how much they

decide to do in the next sprint based on how many points they got done in the previous sprint which is like the core of the approach i know but i know but we just you know we delivered 25 story points last sprint but and the seven sprints before that but i'm pretty sure we're gonna do 80 this time but here's what we would like to get done this work is all important so we will do it in this sprint right instead of instead of however long it's actually gonna take yeah yeah that's that's a thing i think i i could have done better on teams by being that negative person on the team who says yeah yeah we can't do this team yeah exactly that's the other thing no one wants to don't believe in yourselves you're all delusional let's just start with that

it's kind of kind of a crappy way to open a statement i so i have gotten better at estimation in general but not to the point where it's just like laser accurate about how long it'll take me to do something i have a much better feel for how for when something will just take me longer than it seems like it should take but i'm still routinely off by a lot on how long it actually ends up taking me yeah and when you're off it's not like oh i was 10 off it's more like orders of magnitude right yeah yeah like i just finished a thing that took twice as long as i thought it would take and there were a ton of unknowns it's a new platform new technology all around so it makes sense it would take longer

but i didn't know how much longer because i didn't know what working with these tools would be like yeah yeah have you have you ever gone to your product manager or team lead and said we need to build a small proof of concept to gain full stack familiarity with the things that we're going to be touching before we can make an accurate estimate uh yes actually yeah we we did that i would call it a medium scale proof of concept and it took it so it wasn't small scale it was it was entirely ui focused but the ui all worked and it took like a month or two oh that's that's a long time for a proof of concept yeah yeah the point was we would get this proof of concept we would show it to people and and it would work richly so it'd be a lot

easier to test with users okay and and get feedback on without the overhead of building a back end to support all the all the tricky data flow issues that came into into play it was kind of a waste of time honestly oh really yeah i mean we learned the tools we were using but we kind of already knew those they weren't completely new to us in this project and we ended up just like rebuilding everything including the proof of concept stuff oh ouch okay so it was one month of investment how many engineers uh it was like three so so basically a quarter of a year in terms of salary invested for some learning but did you get valuable user feedback um well yeah we we did i think the dream was this would result in a more firmly defined set of features

because it was at the very very early stages of a product we just had no idea what it would look like or be and we did this instead of starting to build the real thing right away to try and nail down some stuff and we got information maybe it wasn't a complete failure actually now that i'm thinking back on it but for purposes of estimation it wasn't that useful is what you know it wasn't useful for estimation i guess it was useful in that we were able to throw it away and we would not have thrown away the real product if we had started building it right away and we did learn a bunch of stuff from it that influenced how we built the real product do you feel like you went down a safer more long-term viable path that you maybe wouldn't

have gone down from uh no honestly total waste huge waste no it was a it was a meat it was not a big enough win for me to say this is amazing and i want to okay this is the right way to build software okay but i don't know it was a total failure okay anyways so back to the question dave have you seen this in your software experience yeah i would say almost every team almost every sprint if the team decides to commit to a set of deliverables in a one two or three week period they almost all the time fall short of it and isn't that crazy in two or three weeks as an industry we can't deliver what we promise well think about we talked about goals a couple weeks ago um do you ever make personal goals and then fail to achieve them i wonder how much of

this is aspirational instead of measurement not commitments to get things done but things to motivate the team or or the ideal version of what you would like to get done i guess i think we do idealize it and i think sometimes we when we're estimating we remember that one time that we were super productive and we cranked out a feature over the course of a day or two and it was amazing and we felt great and then we use that same personal velocity when we're considering story points and estimates for future estimation so it's like that one time i was able to be really focused and deliver something in two days this feels about like that but it turns out that that two-day thing was incredibly rare in your life and that normally there are interruptions there are

meetings and other things that get in the way of actually staying focused like that so that tends to spread out to four or five days and so i think i think we are aspirational in that one way but i think you're thinking of it in a slightly different way yeah i don't know well have you been on a team or seen a team that has has regularly hit their story point commitments yeah only one only one team and it was about can you tell the legend of this team i'm trying to think of a good agile team back when all the fields were green before brown fields so i worked on a team at my last company for about six months i worked on it for about a year but there was a good six month period where we nailed our commitments every sprint we did two

week sprints we did story point estimation up front before each sprint and we delivered like 100 of the of the story points committed and then after i left that team and another lead took over they continued that and i think they did like 13 or 14 or more sprints in a row of meeting their their commitments 100 and there was a there was a combination of factors there that that i noticed that they did that i tried to get other teams to do and then we had about eight or nine teams at the time and the other teams really didn't did not meet that same standard and i would say there were a few things that set them apart the first one was that they were pretty realistic in their estimates meaning they looked at past velocity when they were considering how many points to

take on for the upcoming sprint then in their daily stand-up they would ask really important questions like are any of the stories remaining that we've committed to at risk for my sprint for this current sprint and if so the team would pounce on that instead of just saying hey were you busy yesterday good you were all right next developer hey were you busy yesterday good you know instead they would go story by story and say is this story going to make it in this sprint yes or no is this story gonna make it in the sprint yes or no that's a great point i have seen and i've i've done and i've seen other people do just defensive sprints or defensive stand-ups where your job is to like justify your existence exactly like were you busy enough to look good in stand-up

i swear i worked real hard yesterday no blockers yeah yeah i've seen people say those words with different words yeah i i really dislike that kind of stand-up to me the the team's goal whether it's set of stories or something else should be the thing that's reporting status and the developers are just the mouthpiece for that thing they represent that thing so it's like and sometimes the person representing that thing is a qa team member or product manager like say the story's been coded up and now it's in testing the qa person should report on that story and they should say whether it's at risk for the sprint deliverable or not you know and this team did a really good job of that i felt like and um other teams i never could really put my finger on why they weren't

able to to meet that same high standard but they didn't so that is the legend as as the team changed and people moved off that team to other teams did they did they bring that ability to other teams or was it just something unique about that group of people that's a good question because i'll get diffused oh i see like did it did it remain with the team or was it something about a person on that team or something yeah yeah like did someone leave and then it stopped or i need to follow up with them because that was uh i left that company about a year ago and um at the time they were still running cranking along they've probably had some organizational changes since then i should follow up and see another question is did that team end up impacting the product or

the business a lot more than other teams see and that's or were they just more regular yeah that's that is the very interesting question because like in the end all the teams delivered high value for the company but this team was consistently predictable on a sprint boundary yeah and so it kind of didn't matter like the because the other team still delivered the value to customers it was needed they built the stuff that was needed by the time it was needed but within a single sprint they were just utterly unable to get to that 100 percent mark every time yeah but maybe it didn't matter in the end right because like they did deliver stuff and the company's doing great you know yeah yeah so it's this is such an elusive subject i think

um so uh it sounds like there might be some things you could do to help your sprints be more accurate you know you mentioned uh divorcing estimates from time um focusing your stand-ups on the status of the sprint not on if you worked hard or not yesterday being real good at your job i'm really good at my job no blockers yeah i should definitely not be fired yeah oh man i wonder how many stand-ups could be replaced with that that's that's probably a good thing to look out for as a team lead is when people are saying that and then you get to figure out why they're saying it and what they should say instead and help them figure that out i mean how much of it came down there's so many ways you could do this right you could you could be

very conservative in your estimates and then you will yeah yeah exactly like if you commit to like one story point for our eight developers for this sprint yeah you're gonna get a hundred percent every time and this team i'm talking about they didn't do that they weren't complete you know slime bags with yeah with the accounting but they also didn't estimate super high either so it was like some of these story point these other teams that failed to meet their commitments also delivered a high number of points but i never i never cross-referenced points because every team had a slightly different calibration so it was kind of apples and oranges to compare them like that yeah you don't want to get into like promoting people based on how many points their

team gets done no we'll talk about that in our next question teaser oh that is true foreshadowing it sounds like you're almost saying um yes they were good at this thing but you don't know that it affected their productivity as a whole yeah but to answer your specific question of i always feel bad because my teams never commit never deliver their fully committed story points in a sprint this team figured that out but what you also said is that's the only team so what i should really say is hey team it doesn't matter no one does this so don't don't worry about it yeah we actually so one of the teams we changed their approach and we said rather than focusing on commitment focus on

velocity and the question isn't can you commit to and deliver all the points this sprint the question is how high can you push your velocity each sprint and we changed we completely changed that for them and we did it as an experiment and uh and once again because these things are basically impossible to measure i don't really know what outcome it had but the team didn't feel bad about missing their commitments because they simply didn't have commitments and instead they were just like let's go big let's deliver as many points as we can this sprint huh and when i say they didn't have commitments that's not to say that they weren't committed to meeting certain business objectives they just didn't have sprint boundary numbers that

they had to target each sprint you know they still had big deliverables like hey we have this customer who needs this by a certain date and they had to hit those targets but on a given sprint it was like hey maximize your velocity and software development is so tricky because this is all looking at how you build it through the framework of agile but people build stuff in a lot of different ways and and say you land on the perfect way to agile software develop i don't does that even mean that you'll build better products than people who just like yolo and sit down and type for a whole week without yeah yeah i don't know well i feel better awesome that was all we were really going for anyway uh old lay about jameson feels better

you were right all along yeah nothing matters just lay about celebrate at the dump okay dave i can tell you that we have answered this question uh-huh and we have uh fully because i know deeply the mind of lay about jameson should we move on to the next question yes we should uh this one's for you to read okay this is from listener john we asked we asked people how to pronounce their names um and john said pronounce it biblically uh so i think i'll put some echo on this john maybe john the beloved yeah great show first time listener first time asker a big time fan do you have those foam hands foam fingers and they have just our faces on them that would be cool oh man that would be so embarrassing that's what fans do if i understand uh fans correctly

you do i am wondering how productivity would actually be measured how can we adjudicate with measurable results between the whippersnapper who wants to use haskell and the pragmatic tech lead maybe this is getting into the hard side of soft skills but talk of productivity always sounds so hand wavy are we talking self-reported developer happiness surveys story point velocities dollars per developer hour is measuring such things even worth it in any case i want to hear what you have to think about the elusive p word thanks thanks john the beloved yeah saint saint sorry saint john i don't know which one saint john the beloved yeah keep adding every time we say john's name we need to add more superlatives to it great as they do in the bible that's right that's i'm

pretty sure saint john the epically beloved awesome yeah disciple my last job there was it's funny he mentions haskell and the pragmatic tech lead because that exact debate took place should we use haskell the pragmatic tech lead was like should we have a company and uh we ended up not using haskell but okay i always wonder what would it be like to actually fail no plenty of people get stuff done with haskell um they're called academics i'm sorry we've now offended dozens of people this is an interesting question and actually productivity did come up in this debate because

one of the strong arguments for haskell is you spend less time debugging because the the strong and powerful type system helps avoid a whole class of errors and i i believe that is the case and see it in my work and i i use some languages that have stronger type systems to to spend less time debugging but there's not really a great measurement of that in fact uh there's a great article this is about static versus dynamic typing not productivity in general but it's a review article that looks at all these studies of productivity in static and dynamic languages and it's basically a wash there there are an equal number of studies supporting either side and how either side will make you build better software faster that it's hard to tell one way or the other there's an equal

number of fake studies that tell you uh yeah not fake but they're not usually the most rigorous because it's hard to measure right exactly this is what the question asker is asking um if you set up people to build something in ruby and haskell how do you figure out which one is more productive um and that is the question we're trying to answer yeah yeah we've recursed um well should we just go back to the quote at the beginning of the question yeah so uh most dank saint john the beloved i have not been in companies where there's an explicit measure of productivity it's always been a gut feel which is bad and good for different reasons but i've never seen it

attempted in in the wild um i know it has been a lot have you ever worked somewhere that has like explicit numerical measures of productivity of some kind i mean beyond beyond sprint planning stuff yeah it's not really uh that doesn't seem quite the same though none that are stated overtly yeah but sometimes managers latch on to certain metrics here and there i okay i have seen them used when people feel that there's a performance problem right um then somebody will pull up source control and be like oh this person only made four commits in the past month or something like that yes i've seen that but i haven't i haven't seen them on a dashboard that's like oh uh ask chief pilot mostank saint john the beloved closed five tickets and and chief pilot lowly

steve only closed three so i better oh man there was a good game there was a company recently that will their whole company model was you pay them a monthly fee give them access to your git repository and they will generate reports for you about your people yep yeah i've seen that and i tried a demo of it just to see what they were doing and uh it was really interesting because they would identify like drops like changes like this person used to write a bunch of code and now they've suddenly stopped things you might want to look into but they also reported like i think what a simple-minded manager would would consider to be basically a top employee report and uh i think that the real world of software development is just so much more

complicated than that that that kind of metric just does not work it falls apart in really important ways and it will actually penalize some of your most valuable engineers yeah because it's wholly dependent on source control which means it's only the code which means it misses um most of what we do as a job exactly uh which is which is thinking and talking and helping other people and and um it takes a lot more than just writing code oh yeah i mean there are people where you see on every team who are holding the team together right like they're answering all these hard questions they're connecting the dots for people they're helping integrate stuff but they're not necessarily cranking out code and bug fixes and their work isn't necessarily

reflected in git or in your in your issue tracker and yet they are like the glue and if you take away the glue the whole thing falls apart right yeah and so there is a law called goodhart's law that states when the measure becomes the target it ceases to be a good measure i think that applies a lot to developer productivity because any metric you come up with will will be gamed if you do it as lines of code people will write long verbose functions just so they look more productive and they might not even do it deliberately it might it just is a subtle influence in their behavior yes and also let me just let me just insert here the corollary to that law is that just because it has become a target doesn't mean that your employees are bad for seeking that

target they might think that it's what you want them to do so they're trying to be good employees and doing what you want by making longer function names or whatever right so yeah if you do a number of issues closed you suddenly get very granular issues yeah yeah and and if you incentivize people like the glue person you talked about if they are disincentivized to answer questions then they're not going to help the team as much and they'll be doing things they're not as valuable in exactly i said before we started recording that i think there's a strong case for nihilism here where it's impossible to measure productivity and you can't do anything which is unsatisfying because somehow you have to you have to know is this person doing a good job are they getting

better how can i help them get better uh will this person be a good hire like judging productivity is something you kind of need to do on a team to to promote to fire to hire to just make the team work well if we haven't figured out how to do it what do you do yeah yeah i mentioned earlier it usually seems like a gut feel thing it does and look i'll tell you when it comes down to measuring developer productivity the best thing you can have is someone who is super connected to the people to be reporting on that now the problem with this is it doesn't scale up so it's very hard to compare across orgs or across teams even but when you're asking someone like who are the most productive developers on your team the people

closest to the action the manager or lead of that team their job is to know this information and they'll gather it from lots of data points sometimes it's sometimes it is something that could be represented as a metric like this person just gets a lot of work done but sometimes there's caveats there like but the code they produce has a lot of bugs you know yeah yeah um and so that is the true story of developer productivity you can't put a single number on it because some it's actually a set of trade-offs like yeah this developer is fast at producing new features but also has low quality code that requires lots of follow-up this other developer you know does the opposite like they produce features really slowly but they always work the first time

you know yeah and this developer is good at getting people getting other developers to produce features more quickly you know so it's like why would you there is no one number because it takes these multiple different kinds of people on a team to be successful yeah i'm thinking back through all the people that i work with and i feel like there are there there are like one or two cases where i where i think of a person and think yes somehow they just were way more productive than other developers i've worked with but they are pretty rare and the rest of them that were talented developers are like you said they have very different strengths and weaknesses and complement each other in different ways yeah exactly that's been my experience too the danger

with that is it's subject to all the biases that people are subject to i mean the danger with measures is they're easy to game they influence people's behavior in ways you might not intend but brains do that already in in different ways i i think if you if you believe that productivity is hard to measure and you go by what you know about people then you owe it to yourself to work hard to make sure you're not being unfairly biased so i have a little story about this because i was in a similar situation as saint john the dank and when i was right out of college i was working in my first job and my one year anniversary was coming up and i knew that that meant it was time for performance reviews. And I wanted to know, like, how can I show my boss that I'm doing a

good job and that I deserve a big, fat, juicy raise? And so I went to my boss and I said, hey, you know, didn't really mention that my one-year review is coming up, but I'm sure he realized what I was doing. And I said, how do I demonstrate that I am contributing positively and doing a good job you know and i said look if i was in sales i would just show you all the sales i brought in and it would be a single number that really reflects my entire contribution and it would be easy but as an engineer like i fix bugs i participate in meetings i do design reviews i do i write code i build features it's very very difficult to put a number on these things and my manager talked to me for like an hour about this just talk talk talk and at the end i was like

wow this this is so great he had all these good things to say but i walked away from that meeting and i realized he has no idea i tried to figure out what his thesis was and i realized he doesn't know he just kind of talked about all the different things and i think in the end that's how you measure productivity is it's actually a long um it's a long discussion because there's so many facets to it in engineering and trying to find one way to measure productivity is just kind of a fool's errand but chief pilot most dank saint john the beloved is not a fool no he knows what's all those titles i know yeah phd on the end there yeah if you work in an environment that has these numerical measures of productivity i think you just kind of need to acknowledge that they exist

and you might need to do some hoop jumping to look good on them but that they are flawed and doing your job well will not necessarily make you look good on those numbers and doing good on those numbers will not necessarily make you do your job well if you are trying to evaluate people i think we have a lot we had a lot to say about uh how you look at people's worth to the team and to the business and if you're looking for good titles we definitely had a lot oh yeah we definitely we're all over that i think we should come up with a formula that you can use that's like super simplistic but awesome like it's like points it's almost like story points but it's like developer value points and i think it's here it is no i got it it's the number of mechanical

keyboards you own it's it's disassociated from time uh that's it therefore it must be good i guess technically not because as you get older probably the number trends upwards unless you get into minimalism and then maybe you only have one i think it's no worse than many other measures of developer productivity how about that how about that okay if you have if you are trying to measure developer productivity it has to be better than the number of mechanical keyboards that they own that's the baseline that's the baseline yeah that's the baseline productivity measure and you need to beat that it's a pretty low bar but maybe not uh i don't know i don't know that it's that low i have a lot of mechanical keyboards

and i'm pretty productive that's all right you are have we answered the question absolutely all right best of luck to you uh sir i don't i don't want to say all the titles again all right what can people do if they would like their own wait oh no i forgot i was going to say this we need to come up with a measure of productivity for ourselves in the show oh yeah okay um i mean we could do number of questions answered it's usually two number of titles invented okay yeah uh i mean just number of times we say words number measure the number of words we say word count that's hard to measure though yeah someone has to count them yeah that's that sounds expensive duration okay really yeah it's very easy

to measure which means it's a very good measure of productivity that's right because it's low effort to create number of times you made me spew out my water because i was laughing number of times i edit out either of us saying um only i know that one though yeah do you actually do that oh yeah i had no idea yeah i do do i say um a lot no oh you do not oh it's you somebody else says um a lot you are killing our productivity score jameson no i think i was gonna say that well i guess that would be yeah it's like an inverse measure productivity how long it takes me to edit the podcast that could be a measure all right i think we've got a winning a winning metric here yes we'll combine

all those into a number and we'll report it to you next week and the units on that number will be what unitless oh it's a dimensionless ratio yeah it's yeah oh but the denominator is number of cumulative mechanical keyboards between the two of us okay sure yeah all right we'll do it next week we'll report um what can people do if they would like their questions to be featured next week or other weeks you can go to softskills.audio and click the top right of the page where it says ask a question fill out the form and enter whatever information you like you can give us your name or leave it off you could be anonymous or give us all your credit cards and social security numbers if you want to which doesn't matter because equifax already gave those out

that's a little topical humor for you uh equifax did not give out good ratings on itunes though to our podcast so if you would like to do something they have not done please do that yes um yeah share it with your friends we like it when people listen to our show i think we're done are we done i think so the metrics seem to suggest that we're finished uh it just ticked over to seven so we'll catch you next week thanks bye bye