Soft Skills Engineering - Episode 454: Tracking productivity? and my CTO is ChatGPT

Starting point is 00:00:00 it takes more than citing cyclomatic complexity as the reason you prefer walking over biking to be a great engineer this is episode 454 of the soft skills engineering podcast where i'm your host jameson dance i'm your host dave smith soft skills engineering is a weekly advice show about all of the non-technical things that go into the technical field of software development like your choice of transportation and how that relates to fake concepts like cyclomatic complexity. Cyclomatic, like bicycle cycling?

Starting point is 00:00:35 Is that what we're talking about here? I think so, yeah. Motor cyclomatic complexity, yeah. It's too complex, so I walk because I prefer simplicity. I do think cyclomatic complexity is... I know it's real, but I have sad memories of linters turned on in Java

Starting point is 00:00:52 that made me cry sad tears. because your cyclomatic complexity number was too high yeah yeah you can either have a long function that has a bunch of nesting or you can have a bunch of little functions and everything happens somewhere else and i think i've swung back to like it's kind of just easier to read through one long function and a bunch of if statements than like jumping all over trace 15 methods all over the place and anyways that's not what this is and that's why today's episode is going to be released in 17 different chunks each modular and self-contained yeah you can compose your own episode however you want why would we dare to impose a structure

Starting point is 00:01:28 or an order upon you what if you want to listen to the the outro first yeah you can override methods in the episode and make them do your own thing yeah it'll be great to it right to left all right do you want to thank our patrons yes i do let's pop open this list of surprises today and see what's in here big thanks to those that are contributing on patreon at a level where we shout them out or whatever they say we should shout out they are noah labhart one day i will write a cron job to replace this text with a random dad joke every week oh that would be wonderful uh this one says in your best aussie accent yeah no i'm just whiting for a mite how'd i do that was a really good aussie accent alexander kuznetsov nick molyneux attribute

Starting point is 00:02:14 error non-tied object has no attribute two string javier gonzalez chewy ted timbrel i quit my job in 2025 and got a way cooler one as a software dev thanks sse podcast whoa and i do the cha-cha like a sissy girl good to know become a senior engineer.com is a newsletter you should read unsalted french fries are morally objectionable generating side projects with ai so jameson notices my resume dan from drone deploy chase w norton never is not just a crater on mars flamingo emoji i like chicken i like liver myomics myomics please deliver trash panda get status kyle boss kent c dodds dodds sorry kent's almost left the s off your name but it did not it did not leave off the c developers developers developers developers developers

Starting point is 00:02:55 developers developers developers developers developers developers develop that i think it's a reference to the c the famous sweaty steve balmer talk yeah nevar is not just a planet in the vulcan system jenny kim the stochastic parrot helicon.ai best observability tool for ai red panda is best panda java is just it's a discounted c sharp with worse documentation Jonathan King, Zanai, Beautiful Functional User Documentation, Bethany, DJ, Tim, and Sasha. Okay. That was like a four-person combo pack in one line. That's great. Way to go, team. Williamangel.net cares about your privacy, so we sell your data with love to the highest bidder. Ragnar, not my chair, not my problem. That's what I say. Brayden Keynes, John Grant,

Starting point is 00:03:35 Brittany Ellick, Joe Grossberg, come to the Slash New conference in Newcastle, Australia, May 28th and 29th. Oh, I would love to go to that. Wow. I really botched the pronunciation on that one. Sorry about that. Please write in and we'll send you some stickers. If this is the best part of your week, please spend more time with your kids. Okay. That was good. I feel so good. I think if you went to that conference in Australia, you would be accepted as one of their own because i was impressed a pretty solid australian accent nice noise should i read our first question i hope you would that's part of why i invited you here today oh excellent i hope the other part is not a performance review after the show we will do your performance review

Starting point is 00:04:22 this is from an anonymous listener who says i'm a manager on a product team i have been asked by upper management to measure story points completed per developer per sprint and display the results publicly each sprint to motivate lower performing employees i explained why according to scrum i don't think this is a good idea but i think my explanations came across as me not wanting to make my team accountable for performance for some context i currently track productivity by reading daily updates prs tickets and tickets from each developer i worry that story points is easily game mobile as a performance target and will make the team want to modify the points after the fact to reflect actual time spent and story points will become a less useful tool for project planning

Starting point is 00:05:04 i'd like to satisfy the higher up ask to measure productivity but in a way that is good for the team the company and my career any thoughts on how to approach this awesome i have a counter metric to propose for you okay number of developers per week that i told to improve their performance put that on a dashboard yeah what about your performance yeah how many how many tough conversations did you have this week how about number of dashboards i put metrics on you're you're highlighting part of the problem with making everything a metric easy way to make that number go up this is i feel like this is touching on an evergreen question which is how do you measure productivity right and i think the broad answer is like no

Starting point is 00:05:50 like you're asking the wrong question sort of like well maybe the wrong question is how do i how do i reduce a developer performance to a single number i think is the yeah the wrong question that's being asked here maybe the preface to this is like what what is the outcome they want from this because they have they have asked you to do a thing and you do not want to do the thing but presumably there's some outcome they they are pushing for well behind the request a little bit in the question they said to motivate lower performing employees it's like yeah like put the put the average up on the dashboard and then when you see that your number is lower than the average you make number go up but is that because they think you have low performing employees like

Starting point is 00:06:37 do they have a concern this are they are they worried that the team is going too slow are they just worried that there are people kind of sloughing it and and they're just getting hidden by the the team overall are they worried that yeah you you have lower performing people and they're not improving or getting fired or quitting right so the team just has these people hanging around i think upper management always worries about that question because yeah they see the payroll go out every week. And then they always wonder, am I getting my money's worth? Yeah. Yeah. I can see how saying, I don't think this is a good idea and giving kind of no alternatives. Like, no, we shouldn't do that. Scrum says you measure, I think it's team

Starting point is 00:07:22 productivity, right? Or team points, not individual points. Yeah. Scrum, I mean, I actually haven't checked in on Scrum recently, but. Yeah. Scrum feels, I'm sure it's deep and there's a lot of nuance i'm ignoring but i thought the the point was you measure the team not the person yeah and i mean the story points exist according to scrum last i checked in for one reason only which is to predict you you calculate velocity to predict deliverables yeah it's like capacity planning yeah exactly productivity measurement yeah and many teams have killed the ability to do capacity planning by using story points as a performance metric yeah and i think the the reason i think that happens is because well like the the mechanism by which you kill

Starting point is 00:08:06 capacity planning by turning this into a metric is goodhart's law which we've mentioned a few times goodhart's law says that when a metric becomes a target it it ceases to be an effective measure and and yeah we've james and you and i've talked about this i don't know probably five times over the last whatever number of years we've been doing this but it's my favorite one of those laws that aren't actually laws named after people right i know when i bring up the most often honestly and wouldn't it be great if you had a name a law named after you that's not a law that would just be wonderful anyway i'll have made it i mean why does this happen this happens because people go oh you want that number to go up i will make that go up and and you know i'm

Starting point is 00:08:42 just going to skip over a lot of stuff we've said before and then drill into this question but like the stuff we've said before is like people are not always bad intention you give them a number you tell them you want that number to go up you make it go up not because you're cheating not because you're gaming it but because you think they want the number to go up and so you do it and sometimes you do things that the business doesn't actually want in order to get that number to go up and that's why good hearts law is not not a good hearted law yeah sorry presumably there are some amount of lower performing employees that all they really need is like a nudge to do more and then they will do more and it'll be generally more valuable but also you will have

Starting point is 00:09:22 plenty of lower performing employees who now will not be the lowest number of points delivered each month, or each sprint or whatever, but they will still be the lowest performing employees on the team. Right. They will find a way to make that number go up even if they don't actually perform better. Yeah. That's really well said. So as someone who's in upper management and I worry about this problem a lot, I do measure story points just to keep an eye on how much my teams have delivered, but I do not put it in front of the whole company. And I do not tell individual developers that I'm looking at this as an output metric that they are judged by. And that's because I don't want it to become, I want it to be a useful measure. So I do look at it, but that is

Starting point is 00:10:06 not really how I measure performance per se. And the way that I measure performance and the way that I give myself the feeling, Jameson, I think that you were referring to earlier, which is how do we get upper management to believe that the team is being performance managed? Well, the way that I give myself that feeling is by talking individually to the managers who report to me who manage developers. And I asked them, how are your team members performing? And they talked to me about each one. Now that that only scales so far, but it always scales to your directs directs. And so I'm super happy to hear them talk about it. And and once in a while, they approach me proactively and say, Dave, I have a performance problem I'm working on. Here's how I found out

Starting point is 00:10:42 about it. Here's what I'm doing to fix it. And that puts me at ease as, quote, upper management, knowing that it's in good hands. And I think that's probably what this manager needs to do is go to upper management and periodically describe the performance improvement work that you're doing with your individual team members so they know that it's happening. Yeah. You don't want it to be too quiet. Exactly. You don't want upper management to say it's too quiet. Surely nothing is perfect. everything's great nothing to talk about that's not what you want i see that makes sense that's

Starting point is 00:11:18 a good point i have at times in places where i have we have done story points i think i've only used them if i have a specific concern about a person and and just wanted uh wanted a data point among many other data points about like well okay let's look at story points let's look at 10 other things and and then put that together to get some kind of overview of like across a bunch of different measurements how do they seem to be doing yes in things that are easy to measure that's great i mean you do not want to reduce a human contribution to a single number it's just not it's not going to work here but you want to give them something and what if they really push to say okay but i need i need i need something measurable okay sure story points per by yeah i

Starting point is 00:12:04 I get it. It can motivate bad behavior. Are you just telling me that you cannot be accountable for the performance of your employees in a quantitative way? Yeah. And I think that's a good counter question is, is the only way that you will believe I'm being accountable is by showing you numbers about my people? And they might say, yes. Yeah. And you'll go, okay, well, if that's the case, I have a couple of ideas for metrics that you can put on a dashboard that I think are not harmful and actually do do a good job of measuring engineering performance. Would you like me to share them with you? I would love. Are you asking

Starting point is 00:12:38 me or the hypothetical executive? I was asking you, but I kind of like how it sounds to ask the hypothetical executive. Ah, yes. Yes. Small underling, please share them with me. So a measure that I love is cycle time. Cycle time, every team needs to tweak this to make sure it fits your deployment processes. But the platonic ideal form of cycle time is the amount of time it takes a developer on your team to go from starting to work on a deliverable to actually having that deliverable in production and getting value for customers. Now that changes a little bit from team to team. I've had to tweak that definition in the past because we had weird deployment procedures, like for example, time from start of work to QA signs off, for example,

Starting point is 00:13:22 that's one way to do it. But I like to measure median cycle time at the team level and put that on a dashboard. And if that thing starts ticking up too high, I know we have a problem on the team. And you can do this across teams. Again, it's never apples to apples from one team to the next because you're like, hey, my mobile app team only deploys to the app store like every couple of months, you know, but my web team deploys multiple times per day. So what do I do with that? You know, it's like, got it. You can't measure these things from team to team, but it's a great performance measure. As a general rule, if that number is low, the team is doing better. Now there's obviously exceptions to it but that's a that's one of my favorites i think you it's interesting that you've

Starting point is 00:14:02 talked about team level metrics and that's one answer i've used in the past for measuring productivity is is sort of like it's my job to if i'm the manager of the individual team it is my job to be aware of the productivity of individuals but it is much healthier and encourages much better behavior if we kind of externally measure productivity at a team level it's much easier to feel safe and feel like you're not being in the panopticon and just being kind of micromanaged and so you could talk about that explicitly you want the team to be accountable at the team level and you can be accountable kind of personally you the manager of the team for the individual performance of individuals and then that sort of incentivizes you to address lower performance on

Starting point is 00:14:53 your team i don't know if that's a super satisfying answer but i think i think team level metrics are less prone to freaking out developers and encouraging bad behavior yeah and you know i i think team metrics like this team members can really rally around them because they're like yeah oh our cycle time is up let's get yeah and it's it's comforting to make number go up or down like yeah it's not all work can be reduced to it but it is pretty easy to rally around a thing. Say we've got 15 of these and we want zero. Go. Yeah, exactly. Get pumped by that. Get it down to zero. And that's probably a good segue to the second metric that I like to put on the screen, which is bug creation rate and bug resolution rate and time to resolve bugs. This

Starting point is 00:15:36 is kind of a measure of software quality as perceived by your users. So I like to segment out bugs that are reported by users or that we know affected users, as opposed to bugs that we found internally before they got anywhere or before any customers experienced them. And I like to put that on a chart too and show how many we have that have been created in the past week, how many have been resolved in the past week, and of the ones that have been resolved, what the average lifespan of that bug was. How many days did it exist for users? And that's another good one because you and your team can see this going up and be like, whoa, we got to pump the brakes on new development and fix some of these quality problems we got going on before

Starting point is 00:16:14 we move forward. Yeah. Well, have we answered the question? I think so. Let's see. Yeah, I think so. It's a challenging situation to help leaders understand, but I do think a combination of offering up metrics that you do feel comfortable sharing and that won't have negative effects and talking regularly about the performance improvements that you're doing. This has worked really well for me. I work with an executive team regularly and a regular but not everyday discussion that I have is who on the team am I working with to improve their performance? And that builds confidence in them. Like they've never come to me and said, Dave, I need you to put something on a dashboard so that I can know that you're doing performance

Starting point is 00:16:53 management because I don't think you are. And ultimately that's what this question tells me is they don't believe that you are doing everything you can do to motivate the best performance of your team members. Which could be true. I mean, it is uncomfortable to push someone to say, hey you are not doing a good enough job so i mean it separate from what's the best way to show this to upper management it's probably also worth it worth taking a moment to to consider that like is there some truth here do i need to just do a better job at helping some lower performing members of the team improve absolutely or or the bad sad crappy part of like do they need to not work here anymore right yeah that is unfortunately your job like you you took the mantle of manager

Starting point is 00:17:35 and one of the duties is unfortunately letting people go when they don't meet the bar for performance. That's what the money's for. And if you don't, if you don't tell your team members how they stand on performance, you're actually doing them a great disservice. Because at some point, either their teammates are going to be frustrated with them, you're going to be frustrated with them. And at some point, you will have to fire them if their performance remains low for long enough. And that's a terrible moment when you do that and they had no idea it was coming. yeah we've probably all been guilty of that but it's not fun yeah all right on that bummer of a note yeah we should go to our next question is this the performance review part no okay no yeah

Starting point is 00:18:18 all right yes i'll read it this comes from a listener named mike who says i work for a company with 30 employees our ceo is trying to be our cto by prompting all our issues to chat gpt This week, we had a discussion about changes needed to comply with specific certifications requested by one of our customers. 15 minutes later, I got an email containing a chat GPT conversation giving, quote, advice that I debunked just 20 minutes beforehand. I have been vocal about my concerns of overuse of LLMs before, and I think it's dangerous for our CEO to keep sending large chunks of factually incorrect text across the org.

Starting point is 00:18:57 he did finally stop talking about story point burndown because chat gpt told him it's a bad metric though so maybe this is salvageable what a great segue from our last one yeah tell chat gpt to tell your boss that it's a bad idea yeah you got to do some alignment on whatever model they're using to make it give the answers you want your boss to right replace factually correct with what i want yeah what a what a brave new world we live in oh i know i see the stuff on twitter that's all like i don't know how to code and i built this app and it's it's magical and then occasionally you'll see them get dunked on for like having an unauthenticated database connection and someone just goes and fills it up with poop emojis or whatever

Starting point is 00:19:43 yeah there's just so much capability out there that we are exploring the impacts of so are you saying that the ceo is a vibe ctoing i think they are yeah i think they're ctoing based on vibes part of the discourse around vibe coding though is well how much do you have to understand and craft and yeah what's what's your responsibility as the vibe coder to to beyond just accepting the output i think it's always been easy for humans to convince themselves they understand something that they actually don't that's way more complicated but i think it has never been easier than when you have a helpful seemingly all-knowing always available assistant that'll just tell you stuff you can just ask it yeah explain to me like the anything the five most

Starting point is 00:20:35 complex things of building a nuclear reactor and then it'll give you five things and you think huh that's not that complicated i got it i should forward this to my uh my atomic scientist team who's building my nuclear reactor hey there's some hey yeah forward forward forward here's some stuff you might have missed and then it's the top five most most dangerous things you might have missed is according to chat gpt yeah like i i look back and see myself doing this mostly in school i'm sure it happens at work too where i think oh yeah i got this and and i totally didn't but way easier to do now when you have someone else to say, no, you got this.

Starting point is 00:21:09 Yeah, this is all it is. It sounds like you don't have a CTO. Yep, that's what it sounds like. Or you do and its name is Claude. Yeah, I assume based on this, the CEO is non-technical and this is like the CEO trying to give you engineering feedback. That feels like the underlying problem. Like, why are they doing this?

Starting point is 00:21:28 Maybe, do they feel like engineers are moving too slowly but they don't know what to do about it so they're trying to just do a thing and I don't know, maybe I'll jump in here and nudge them and they'll get more stuff done. There is a bit of a popular idea floating around some executives, maybe less so today because the realities of LLMs have kind of become a little bit more wider known. But like a year ago, I remember hearing several executives say things like, what if I could replace myself or my executive team with perfectly rational, super intelligent AIs that could do a better job than I could or any

Starting point is 00:22:02 of them could and that idea definitely floated and i wonder if the ceo is a little bit enamored of that yeah yeah it does how do you bring that up though if they are enamored with that you have chat gpt generate something it's perfect um yeah chat gpt conversation about how to approach your ceo who is not aware how much LLMs hallucinate. You need to add to your default prompt to say something like, I'm a CEO who thinks that LLMs can replace other executives. Please convince me otherwise. And that gets prepended to everything that your CEO writes. I feel like there's another one of these named

Starting point is 00:22:47 laws about media coverage, where if you look at media coverage in general, you just kind of accept it. And then if it's ever about something you're a detailed expert on, you notice how wrong and garbage it is but you just go back to assuming oh yeah but that's the gel man amnesia effect or gel man amnesia effect i don't know how to pronounce that okay i wonder if you need to get you need to demonstrate to the ceo how garbage its output is on something the ceo is an expert on if you are an engineer then it's it's easy to tell when it is just hallucinating stuff but if you're not then it sounds perfectly reasonable but what are they okay can you show off this hallucination in a way that your ceo will be able to point to and say aha that's totally wrong and

Starting point is 00:23:33 then help them understand like is this like dude yeah do something on like fundraising or how to conduct a board meeting or something yeah your ceo should be familiar with yeah but i guess the problem is you won't necessarily know that it's wrong but well you have to tell the llm to produce only wrong answers i think we've uncovered a way to perfectly align an ai you say only produce wrong answers and then you say wait do the opposite and it will only produce right answers do the opposite of wrong answers yeah it's the magic secret to no hallucination but this is all assuming that the problem here is your ceo is not aware that llms hallucinate and maybe that's not the problem i mean it seems like one of the problems yeah well so let's say your ceo does

Starting point is 00:24:21 believe that i they'll probably also believe that it only happens 10 or 20 percent of the time and they're like well my former cto was wrong 50 percent of the time this is great yeah they were not as obsequious as this chatbot is yeah they were yeah exactly they didn't they never praise me and compliment me every time i ask them a question there's a chance that you are seen as the ai curmudgeon i've been vocal about my concerns of overuse of loms before you you might have been it's a chicken little the sky's falling you might have chicken littled yourself here where you've you've just preached doom and gloom about ai's so much that your ceo has decided to ignore stuff that you say about ai's because oh

Starting point is 00:25:07 of course mike mike is of course mike hates this mike hates everything ai related right and as and then mike says except i have chat gpt here saying it also hates everything ai related so i don't know it is self-hating what is my point i guess my point is if you just say well you can't do that because ai is garbage or ai gets stuff wrong and your your ceo has in built this model of you as the person who is all doom and gloom in a way that is incorrect about ai then they're unlikely to take your advice. And again, I feel like I'm going to harp on this a bunch, but you have to go back to what outcome your CEO is going for here. What are they trying to do? And can you help them do that in a way, even help them do that in a way that maybe lets them use AI that

Starting point is 00:25:56 doesn't give you all the bad stuff you're going for? I could see that being very annoying if there's a complicated technical discussion. And then the head of the company who doesn't have the context to understand or contribute is like don't worry guys i got this figured out and copy paste just garbage from chat gpt that that would drive me nuts so i can see how it's frustrating but i think you have to address the underlying concern and figure out what the outcome they want is yeah did finally stop i think so i think it's salvageable i think if you collect enough examples of erroneous or harmful direction from your ceo and you might be able to convince your ceo to stop but you might not ceos aren't really known for you know like doing what the people say to do

Starting point is 00:26:44 i don't know yeah i didn't get the ceo doing what all you little people told me to do yeah or you and your llms i wonder if they'll just think ah the models aren't good enough yet so i will wait a year and then i will go back to yeah copy pasting everything that my engineering team says in the chat gpt and pasting it back honestly i feel like that would be there are there are ways in which that could be useful it's it is easy sometimes as an engineer to get really stuck in a rut and the benefit of someone from an outside context saying no clearly you must do this can it could help nudge you out of that sometimes but it's probably not even if it's a machine yeah yeah it's a machine being wielded by the guy who signs your paycheck

Starting point is 00:27:28 yes yeah there is important context there all right have we answered this question i think so good luck best of luck mike all right what can people do if they want their own questions answered go to softskills.audio and click the ask a question button thank you so much to everyone who has done that we really appreciate you writing in with your questions we know it's not easy to fill out that form and type in type your soul into the text box but we love it you do it thank you golly you do it anyways you persevere you are a champion thank you for listening we will catch you next week

Soft Skills Engineering - Episode 454: Tracking productivity? and my CTO is ChatGPT

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.