The Pragmatic Engineer - Developer productivity with Dr. Nicole Forsgren (creator of DORA, co-creator of SPACE)

Episode Date: February 19, 2025

Supported by Our Partners• WorkOS — The modern identity platform for B2B SaaS• CodeRabbit — Cut code review time and bugs in half• Augment Code — AI coding assistant that pro engineering teams love—How do you architect a live streaming system to deal with more load than it’s ever been done before? Today, we hear from an architect of such a system: Ashutosh Agrawal, formerly Chief Architect of JioCinema (and currently Staff Software Engineer at Google DeepMind.)We take a deep dive into video streaming architecture, tackling the complexities of live streaming at scale (at tens of millions of parallel streams) and the challenges engineers face in delivering seamless experiences. We talk about the following topics: • How large-scale live streaming architectures are designed• Tradeoffs in optimizing performance• Early warning signs of streaming failures and how to detect them• Why capacity planning for streaming is SO difficult• The technical hurdles of streaming in APAC regions• Why Ashutosh hates APMs (Application Performance Management systems)• Ashutosh’s advice for those looking to improve their systems design expertise• And much more!—Timestamps(00:00) Intro(01:28) The world record-breaking live stream and how support works with live events(05:57) An overview of streaming architecture(21:48) The differences between internet streaming and traditional television.l(22:26) How adaptive bitrate streaming works(25:30) How throttling works on the mobile tower side (27:46) Leading indicators of streaming problems and the data visualization needed(31:03) How metrics are set (33:38) Best practices for capacity planning (35:50) Which resources are planned for in capacity planning (37:10) How streaming services plan for future live events with vendors(41:01) APAC specific challenges(44:48) Horizontal scaling vs. vertical scaling (46:10) Why auto-scaling doesn’t work(47:30) Concurrency: the golden metric to scale against(48:17) User journeys that cause problems (49:59) Recommendations for learning more about video streaming (51:11) How Ashutosh learned on the job(55:21) Advice for engineers who would like to get better at systems(1:00:10) Rapid fire round—The Pragmatic Engineer deepdives relevant for this episode:• Software architect archetypes https://newsletter.pragmaticengineer.com/p/software-architect-archetypes • Engineering leadership skill set overlaps https://newsletter.pragmaticengineer.com/p/engineering-leadership-skillset-overlaps • Software architecture with Grady Booch https://newsletter.pragmaticengineer.com/p/software-architecture-with-grady-booch—See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠—Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email [e-mail]. Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

S is a satisfaction metric. Just ask people, right? Are you satisfied with the tools you use? Are you satisfied with something? Because people can usually tell you pretty quickly. Like, I've seen data across an entire tool chain that was pretty fast. Like, it was good for the company. But you asked the developers and they're like, this, this is nearly impossible.

Performance, you know, outcome metrics we definitely want. A is activity, all accounts, anything that's account, right? We usually have a lot of activity metrics. C is communication and collaboration. This can be looking at meeting schedules. It can also be looking at how well APIs are working or if they're breaking. And then E is efficiency and flow. Those types of things, those types of data points and metrics, give you additional insight.

Nicole Forsgrin is probably the best known expert on developer productivity. She is a co-creator at the widely used framework Dora and Space, lead author of the book, Accelerate. Nicole started her career as a software engineer at IBM, where she experienced firsthand the challenges of being given unreally deadlines by management who understood nothing about developer productivity. Seeing how developers burned out around her, she entered academia, researching DevOps, and developer productivity.

She then joined Chef Software and then co-founded a bootstrap startup called Dora. Dora was acquired by Google and Nicol worked on developer productivity here. She later joined GitHub as VP of Research and Strategy, and she is currently leading research initiatives of Microsoft. She's also working on a book related to developer productivity, one that we'll touch on in the podcast. In our conversation today, we cover Dora and Space, examples of using these frameworks and advice on how to make the most of them. measuring developer productivity, whether measuring PRs makes sense and why this area remains challenging to quantify. Traits of high-performing engineering teams and why time to onboarding can be very telling about the efficiency of a team and many more interesting topics. If you're looking for advice on making your engineering team more efficient or are interested in ways to measure developer productivity, this episode is for you.

If you enjoy this show, please subscribe to the podcast on any podcast platform and on YouTube. Thank you. This greatly helps the show to spread to even more listeners and viewers. With this, I'm very excited to have Nicole on the podcast. I wanted to put a pointed question at you. Yeah. You've been studying developer productivity, both at the theory level, but also very practical level. What is your take on the importance of PRs or diffs?

And the background is here that there's now a renewed conversation about this. There's been this viral threat going around on the concept of ghost engineers who might not be producing as much kind of like code on a monthly basis. And, you know, that's something that can be measured. And also companies like Uber and Meta, they do have tools for managers and directors to track the number of diffs per team. They don't say that they're going to break it down per engineer. It is possible.

And also there's a new DX core 4 framework, which is something that is going around. And they also suggest that the, you know, that the. number of diffs at the team level should be measured. I'm curious, how important is the diff frequency and how good of a predictor is it for productivity or unproductivity at a team at an individual level? I love this question. Thank you so much. I will say PRs and diffs, both are good signals and are terrible signals, which is probably not helpful, but let me kind of dig into this a little bit more. When we think about using the word productivity in particular, Many folks, especially in leadership, are thinking about the traditional economic definition of productivity, which is the rate of output, right?

What do you produce? And then sometimes that'll be a ratio, right? The rate of output per inputs. And so, as we know, that doesn't apply very well to software, or it's really hard to disentangle, right? Is it a feature? Is it a shipment? Is it a full product? Is it what?

It's different than in like farming. I grew up my grandfather that was a farmer, right? Like how much do you put in and then how many potatoes do you get out? Like that's fairly straightforward. And when we think about GDP, that's often what it's looking at. But in technology, things are just different because many times you'll get many, many, many more things when we think about cameras and phones or photos, right? We could take a handful of photos.

Now we can take millions of photos and it's basically free. And so many things in technology are just flipped. So with that context, I will say they are important in the PRs are important in a few ways. One way is that it does give you a relatively reasonable view into what work is getting done and what work can get done through the current system and processes that you have. With a couple important caveats and disclaimers, right? Many senior engineers don't have very high peer output or code commits. Now, for those of us who have been in software for a while, this isn't surprising, right?

Because as you become more and more senior, you're doing things like unblocking folks. You're doing things like architecture and design work. You're doing things like supporting reviews and mentoring and maybe like pair programming with juniors. And so your commits, your PRs, yours, finger quote, right, aren't showing up. This episode is brought to by DX, the engineering intelligence platform designed by leading researchers. Measuring developer productivity is a decades-old challenge, and many technology organizations still struggle to get it right. DX takes a scientific approach that offers complete clarity into productivity and its underlying factors. It combines data from development tools with self-reported data collected from developers,

so you can see both the what and the why. The insights from their platform cater to leaders at every level of the organization, organization from CTOs to platform teams to frontline managers. Learn why some of the world's most iconic companies like Dropbox, Played, Twilio, Versel, and Gusto rely on DX. Just visit getdX.com slash pragmatic engineer. That is GETDX.com slash pragmatic engineer. And then you're also pulled into recruitment, your leadership meeting, performance calibration, oh, there's this team on fire. We need you to help.

foul. Yes. The higher you are staff and principle level, it happens all the time. Exactly. Exactly. And so I think it's really important to think about not just what we're measuring, but in which context and who we're measuring because that's not, I wouldn't call that output.

I would call that one aspect of throughput, right? Because it can be really interesting and important to see if there are blockers, why is it? If there's friction, why is it? Is it an overly complicated code base? Is it really, really difficult to use tech stack and developer ecosystem? Is it that our PR assignment process is just, like, bad, right? Either it only goes to one person or it doesn't, like, get shared out very well.

And so one person or just the assignment is the blocker. So there are a handful things to look at there. It's bad as a just straight solo metric. but I think everything as bad as a solo metric. What we always want to do is take kind of a constellation of metrics that help us think more holistically about the context in which developers are working. What are the tradeoffs that they are making? So I've been working with some folks to kind of kick up.

We started the Eng Thrive effort at Microsoft. And it's a way to understand and start conversations at several levels in the business about how can we help our engineers thrive? And we use a handful of metrics across many dimensions. And it was inspired in part by space, the space framework. Because then we can look at qualitative feedback. Microsoft famously has NSAT, which is kind of like a satisfaction qualitative survey. That's like every six months, right?

I think I remember when I was there. It was still going on back in 10, 15 years ago. Yeah, some of them are quarterly now so that we can make sure We stay on top of any trends, any changes. But then we also want to capture some objective data from our systems because that can provide additional insight and you're not asking people for it all the time. You're not interrupting their work. And so there you're looking at, you know, examples of quality and outcome metrics.

You're looking at examples of any delays that may come up. Now, I'm just giving like broad examples. This isn't not necessarily what N-DRIVE uses, but, you know, like build time or deployment pieces or security outcomes or quality and stability outcomes, PRs and the data that are around PRs, right? So it can be number of PRs, it can be time for PR, it can be the time that it waits from review. Those can be helpful in this constellation, right? Because then you're also seeing and you can think about a story of why people are making decisions to do the work that they're doing and what are the impacts and implications. And I love that because it can help inspire

larger questions around when we're making investments. Now, say investments, whether it's money or capital or time, where should those go, right? How do we want to balance feature delivery and customer value delivery with internal investments? Because those are so tightly related, right? I like to joke. This is our hot AI moment, right? And everyone's like, we'll build all the AI things. Everything's AI. You have to go faster. It's real hard to build the AI things when like your pipeline's trash. Like it's super hard if your developer experience just isn't there. I agree. I also feel that the reason, I've been thinking why is this trending right now? Like people seem to be really touchy-feely about the fact that you can measure diffs.

This survey was specifically about remote engineers, you know, like, like the sense you're doing less. And I feel that what is happening is a lot of things that we're, what you're talking about and what people talk about in developer productivity. It's like you have a team that's doing well. Everyone is, you know, doing their work. And we're trying to figure out how to do better, what's happening. Where are people stuck? Where are the bottlenecks?

But there's another question, which is performance management, which is, I have a team and I'm a non-technical manager. Or I might be, you know, removed from the layers. I'm the director or CEO. And I just want some stats that tell me who's performing well and not. And there's an assumption for some reason for people who are not working. here that I can just look at some numbers, may this be outputter or some scores, and I'll see people scored. And my question to you, how important or is it possible to, you know, run a good engineering team without a manager or a lead who is hands on and kind of understands this

context and, you know, like knows that what's going on is kind of involved. Because I'm kind of seeing that there's a real desire from people, investors, founders who are removed from technical details to get some of these stats, you know, some of these developer frameworks so that they can tell without understanding anything, oh, this engineer is good and productive, that engineer is not? A, is this realistic? And B, is this something new? Or have the essence of business always been asking for something like this?

I'll start with that last one. Is this new? I'll say it's not new, right? Like we've been, we, the royal way. That's good. Business and researchers have been watching people work and trying to measure how people work and trying to assess how people work with summaries and numbers for a really long time, right? There's something in research called the Hawthorne effect, which is when just, I love it.

So this gentleman was working with, I want to say it was like some type of production for a process. And he would come into the office, he would, you know, like flip the lights on, like watch people work, count things and he would leave and like they would count things and the numbers would be different. Well, so he was like, it must be because the lights are on. Because I guess there's, if I'm remembering correctly, right? Like he turned the lights on. So they start turning the lights on, but it's not like consistent. It turns out like people are doing more like when you're there, when someone is there.

When they know they're being watched, right? Athletes tend to perform better when there's an audience, right? Us developers will definitely do it. As soon as we know what's the measure, we're like, all right, we're optimized for that. Absolutely. And I think this is where we have some real limitations around things. You touched on it a little bit when you said like we can measure PRs. So many times people will default to.

the measures that are easy and quick to operationalize. Number of commits, number of PRs, number of lines of code, number of what can we grab quickly? And I think this is one thing that prompted space, the space framework, is counts can be useful. Yes, if you have access to data that you can quickly operationalize and grab, sure, right? Inserting caveats around like what's the accuracy, the reliability of the data, how are you gathering it, are you doing cohort analysis? But also, that will only show you a few things. And the few things it will show you are the parts of the business that are easy to operationalize.

And it will ignore everything else. And so we often want to think about, you know, S is a satisfaction metric. Just ask people, right? Are you satisfied with the tools you use? Are you satisfied with something? Because people can usually tell you pretty quickly. Like, I've seen data across an entire tool chain

that was pretty fast. Like, it was good for the company. But you ask the developers, and they're like, this is nearly impossible. We are going to break soon. because so much of it was in systems, so you'd think it was automated, but every single handoff point had to be manual because they were not integrated. And that's one of the biggest weaknesses for operationalized data is missing that handoff.

The boundaries between systems and the boundaries between processes. So people can tell you pretty quickly. So performance, you know, outcome metrics we definitely want. A is activity, all accounts, anything that's account, right? We usually have a lot of activity metrics. C is communication and collaboration. This can be looking at meeting schedules. It can also be looking at how well APIs are working, right?

Or if they're breaking. And then ease efficiency and flow. And so that's where we're looking at time, handoffs. You know, those types of things, those types of data points and metrics give you additional insight. And if you can grab at least three different categories, you're going to be in a much better position. Now, you also asked, and I'm kind of. tying into the earlier question, which was, can you evaluate and understand how well a team's doing? Can an edge leader and an edge manager do it if they're not there? I think at that level, right,

when you've got a tech lead, when you've got an end leader, an edge manager, being absent and unaware of context does no one any favors, right? Yeah. At some level, though, I think there can be benefit to having a holistic set of metrics to help leaders guide decisions around which type is an example. What type of blockers or barriers or friction points are affecting most of the company? Would that be a good use of a central resource? What types of barriers show up that are probably out of the realm or scope of control for a development team to fix?

Right? Sometimes their documentation can be. I hear you. But if it's something like a central build system, no individual team is going to be able to fix that. Or it will be heroic effort, but the larger the company, the more impossible it is. Exactly. And so this is where I think there can be some value, right, especially when we're doing things like cohort analysis and kind of holistic measures and having it be a discussion where people can get curious and can talk through what they think they're seeing and then go confirm. because it's just not realistic for an executive to meet and be fully embedded with every single engineering team that exists and every single product team that exists and every single marketing team that exists to try to figure out, you know, get a good feel for the business.

This makes a lot of sense. And I really like about space how to me that was the first framework that kind of made it really clear that this thing is complex. And we can measure a lot of things and we should measure multiple things. And sometimes they will, you know, tell us different stories. but we need to look at the whole picture. But before we go too much into space and what happened before, before space, there was Dora. You mentioned your first start. It had a massive impact on the industry.

The book Accelerate, I think, had a huge impact as well. I see it on the shelves as almost every engineering leader. And if you've not read it, I think it's a great read to start with, at least. And in the Dora framework, there are four key metrics. the deployment frequency for D, lead time for changes, change failure rate, time to restore service. And still today, a lot of engineering leaders who I meet, they read the book, they kind of hold it up and say, like, I've read the book, I've cracked it, what we need to do to be a highly performing team. Here are these four things. We're going to measure it.

And if we have high scores on it, we'll be there. Now, clearly, things have Voltzance, but I'm interested, where do you see the usefulness of Dora? So teams who are optimizing for this and getting to this thing, you know, how far can they go? And what is the point where it's kind of time to look at other things? And, you know, like, for example, how space and other things evolved from here. Dora is a lot of things. Dora is how I kind of refer to the entire research program. But a lot of folks, just when they hear Dora, they think about the four metrics.

And so I think understanding that there are both of those are important. And here's why. The four metrics end up being, you know, through a lot of statistical tests over 10 years now, like even when I'm not doing the analysis, it ends up being a very good indicator, a really good signal for how well your development pipeline is working, right? Is it relatively fast? How many things can you deploy? What are the quality outcomes at the end of that? Right. And so for folks who are just starting out, that can be a pretty good place to start.

Now, I will say sometimes folks just hate me because they're like, well, this doesn't work here. You know, I'm in air gap systems. I'm in wherever. Find ways to accommodate for your environment. I do work with folks that work in air gap systems that use Dora. And so instead of the final- By air gap systems? Can you just tell us for ones of us who are not familiar with that term?

Yeah. So air gap systems are where you have some gap in your deployment process, usually right before like final deployment. where there is no communication between the two sides, right? There's no input. You can't just push and it deploys all the way through. So some folks that I work with that have air gap systems is the U.S. Navy, right? They do it. So sometimes it's for-

It kind of makes sense why they would have enough. Right. Many times it's for security purposes, but sometimes it's because you literally have to take something out to a ship or a submarine and install it there, right? That's a good way to ship on a ship. Yeah, right? And so there, you know, we say,

okay, well, then what is, what do we want to consider our deployment pipeline? When we want to think about optimizing and improving this, what should that be? Now, at some point, you do want to consider flying out to a ship or a submarine and installing it. Yeah. But for the teams on the ground, you can go until that pre-prot environment. Because that's within the scope of control. And that, for all intents and purposes, the way they're thinking about it, that is what they're thinking about. Now, this is the same is true for, like, iPhone and Android.

Android apps, you can't push to the app store every single day. That's not real, right? But you can go to pre-prod. And that can give you great insights. So don't get super caught up in like precise definitions because we can always accommodate. Now, your other question is how far should you go and what else is there? Now, sometimes folks also tell me that they hate Dora, the Dora for because it just tells them how bad they're doing. But that's where the rest of the research program comes in, right? Because

there's a BFD, a big friendly diagram that kind of points to the many things that can improve these four metrics, things like improving automated testing, things like improving continuous integration, things like improving, you know, your cloud deployments and your cloud usage. So it can help. It's a signal, but it's not the action, right? How well are we doing? And it can be good because as you generally start to improve, you start to improve. If there's a huge drop-off, you should probably take a look at your system. Right. So from a signal perspective, it's pretty good. It's not everything, though. And one thing I always try to point out is that Dora only goes from commit to production. Now, back when we were starting this, this was like 2013. That end-to-end software

systems had not been studied very much and measured because it's so, so difficult to measure them. And so we focused on that part of the engineering system because we thought we could have some really good impact, deliver some good insights, also because that's the part of the process that can be highly engineered, right? When we think about writing code, we want things to work better and be faster, but also sometimes we just need to take 30 minutes to go walk around outside to solve a problem. Sometimes we're sketching on a notebook to solve a problem. Sometimes we're having discussions around prioritization. Once you commit, though, you should have opportunities to optimize all of that system. It should be high predictability and low variability. And so that is also

a nice way to think about that because if your tool chain is really out of whack, if it requires a ton of manual intervention, if all of your tests and your builds are failing, if it's highly, highly variable, sometimes it's super fast and sometimes it absolutely breaks and you can't figure out why. It's a great place start because you can engineer the heck out of it. But it's not everything. Developer experience ends up being incredibly important. Now, it's related because if you can't get feedback on something for two weeks and then you find out that, you know, the thing you committed, you thought you were done with, you were no longer done with. I have to drop all of my work. I have to reset my environment. I have to get my head back in the right head space. I have to figure out everything that's

happening. So fast feedback is important, but the rest of the developer experience, the holistic developer experience is also important. Can I set up an environment quickly? Can I provision the resources that I need quickly? Can I write code and do local build and test quickly with good insights? And so that's where identifying the opportunities to really understand and get to, I like to look at this as a constraints problem, theory of constraints. Where are my biggest blockers? Because I can't fix everything and I sure can't fix everything in the next month. Do I understand correctly that Dora started with kind of developer productivity, if you will, as you said, like until we ship just to get some data from there.

And a lot of the research that has been happening since thinking about space, thinking out the DevX framework, thinking about some new things coming, are starting to look at not just this, but also the other part or get a piece of the puzzle. I'm just putting it here. As you mentioned, developer productivity and how all of this is, affecting one or the other.

Is this a fair way to say where the space is progressing? Or how do you see things evolving since Dora, together with space and some of the newer research that you're embarking on? Yeah. So a lot of people have studied various ways to improve the active writing software, right? I don't say development because development is a lot of things. But like when you're coding in an IDE, what are ways that we can improve that? But I think there has been kind of this continued evolution, both with the tools that we have,

with the processes we have, with the things that we've learned prior to think about this from an end-to-end type of a view, right? So not just commit and not just writing code. Although many people focus on that and they're the experts and they can tell you everything down to the millisecond of how long things take and like at what point you lose focus and you break concentration. So I think all of that, all of these individual pieces are very, very important. And for some folks, myself included, my team, we like to look at the full experience to kind of test what that looks like.

And I think it's fair to say that that is part of the evolution of space is, you know, we had learned so much about the, you know, commit to prod. And how else do we want to think about that? And I had been chatting with so many companies for so long that kept saying, well, how do I define a metric? then. How do I pick the right thing? Or once you're in Dora, and let's say Dora, you know, the analysis, some kind of constraint analysis tells you that build is the biggest problem or PR is the biggest problem. Then the question is, well, how do I measure it? And so space was useful here because it can measure the entire developer end-to-end tool chain. Or you can just come up with space metrics specific to PRs. How satisfied are people with the PR assignment process?

What's the outcome of PRs, right? Like what percentage of PRs get through with in, you know, a day, a certain time frame or a number of viewers. Activity metrics. How many PRs do people commit? How many PRs are they assigned? Communication, right? Like, what does the communication piece look like? And then efficiency and full, how long does it take?

Where are the delays? So you can apply space to a specific component as well. And it turns out when I went back and looked, Dora is an instance. It's an instantiation of space. Oh, wow. That's kind of reassuring. Yeah, so deploy frequency is a count, right? That's an activity metric. Lead time to deploy is efficiency and flow. And then change, fail and time to restore are both performance metrics.

So getting into developer experience, what, like, as you mentioned, this is becoming increasingly important, especially at mid-sized companies and larger companies, maybe startups don't care as much just yet. But what does this term mean to you? What is a place that has good. developer experience? I think developer experience is what a developer goes through, like, what's their lived experience? I'm very reciprocal in using the same word. But in research, we'd like to talk about, like, what's a lived experience, right? What does it feel like and what is it like to do the job day to day?

Now, many times people kind of conflate like a good developer experience with just using the word developer experience. but I do think it's important to think about them at least a little bit separately because the experience is just what the work is, right? Could be good, it could be bad. That's what it is. And then if we want to improve it, we can look for ways or we can identify ways that take away from or inject negative, like not wanted things in that developer experience,

which often looks like friction, which often looks like blockers and barriers, which often looks like confusion, right? If we can't find information, if we can't find the docs, if we can't find any of the things, then that gives us, you know, an improving developer experience. Or sometimes there's a degradation of developer experience. When we roll out, I'm sure people will think about this, right? When a company rolls out a new HR system or a new expense system or a new budgeting system, it's going to slow me down. And that might not be what people typically think of as my role as a developer. But if I need to ever interact with these systems with a type of regularity, you've got an initial learning curve, which is understandable.

But if that delay or if that friction remains for an extended period of time, that is going to affect the way I do my work. It will affect the time that I have available. Just as you say, an example that came to my mind, when I worked at Uber, when I started, things were pretty kind of free. You could access a lot of systems. And there was some logging of accessing sensitive data when you went to profile, you had to just write quickly while you're doing it and then boom. And then as Uber grew, obviously they needed more compliance. So what happened is when you wanted to access systems or even data as I was debugging customer issue, I needed to fill it out and my manager or my director had to approve it.

Now, suddenly stuff started happening where like, okay, I need to just fix this bug. And I couldn't because my director or my manager was traveling and it was like two days because then the skip level. And suddenly I lost context. I didn't get back. And I mean, this policy change that had nothing to do with developer. It was just really like hammering down. I never thought of it as the developer experience was there.

I was like, oh, there's just, you know, things. But the result was like things were tasks were getting done slower, especially these debugging tasks. It's an interesting way to think of it. I don't think we thought of it like that. And I love that you bring up not just the time delay, but also the cognitive implications, right? How does this affect your cognitive load in terms of having to come back to what? later and regain context, but also, maybe this is just me, but I'll joke that I just end up having to use computers and anger too often, right? Because when it keeps breaking, I can handle

so much frustration and still be at my best in terms of creativity and sometimes it inspires me. But at some point, I'm like, not today. We're done. Right? Like, I've got to jump over to some administrivia that needs done anyway. And it just doesn't require the type of problem solving and creativity and innovation that great work does. And I just, I can't do great work when I'm just surrounded by friction and interruptions and broken systems and unnecessary process. Some of it's good. But also, how can we find ways to minimize that process, right?

Security and compliance is super important. Are there ways where any environment that gets spun up automatically has all of the, or most of the preconditions in the settings done versus a developer, every single developer having to do it manually every single day and every single time, like multiple times a day. Now, in your experience, because you talk with a lot of different companies, large and small, you also work at some very well-known ones. Again, pretty pointed question. Do you think developer experience in general is better at some of the largest companies, you know, the Facebooks, the Microsofts, the metas, etc. Or at small startups, which, you know, like these big companies that I mentioned,

and they do invest a lot in developer productivity and sometimes smaller companies look at them at envy. They have platform teams. They build custom tools. They have teams that measure all these things and try to improve it. Startups, you know, small like two, three, five person startup has none of that. They just kind of do whatever they can. So do you see a difference between them? Because sometimes startup move pretty darn fast.

Yeah. I will say yes and no. And a joke I used to make all the time when people would come to me with door and they would say, well, this can't apply to me because I'm at a startup. There's no way this could apply to XM in a large company. There has to be something different. I'm like, well, startups tend to be faster. They have less bureaucracy.

They have less levels. Large companies have more funding. They have more resources. They have more ways to help improve what that developer experience looks like. So, TLDR, no excuses. But what do I see? I see a mix, right? So I've been at a large company that had Google, probably the best developer experience I've ever seen.

It was incredible. I was committee coding the first day. It was great. They have really taken time to invest in that and prioritize that and research that, right? I worked with some incredible researchers there that did great works here. Jaspin, Emerson Murphy Hill, he's at Microsoft now. They care deeply about that. And it's evident through almost every system that they have.

They do not tolerate friction. I've been at other large companies. and advised, especially I've seen, you know, as I kind of talk to large companies, some of them are very good and some of them are very bad. Like, it's, I don't want to say bad. It's not a value judgment, but it's incredibly difficult to get work done there. Why? Yeah.

Because many times they're working in large legacy systems, legacy code bases. They're working with layers of bureaucracy that slowly built up over time and now it just makes it incredibly difficult to do work. Yeah. Startups have often a slightly different set of constraints, right? So they can move fast. They can pick whatever tool they want. They can just go.

But that also means they have less infrastructure in place, right? They have to build that as they go. They have to kind of acquire that. If they have to go through security and compliance, there is not a team dedicated to security and compliance. At Dora, we were a tiny little like three-person startup that pretended to be like a 10-person startup. We did stock two compliance because some of our customers required that.

Getting a stock two and basically a two, two and a half three person company, that's a lot, right? Like that's. Congrats. I did that. We did that at Uber, but we had a massive team dedicated people. It stopped work. Now, shout out to Jez because he did the lion's share of that work. But there was no development happening through those, that week or two, right?

It just doesn't happen unless it was like emergency bug fixes or something. Yeah. So there's there, there, there are always tradeoffs wherever you go. One thing that keeps coming, I just keep asking it, because I, I, I've also been thinking and covering and, you know, being involved in developer productivity here and there for like many, many years now. Why is it so darn hard? And you've been doing this for a lot longer than I have. Why? It just feels, you know, like we're software engineers.

We write code. We have an output. Okay. We're people as well. but we're kind of pretty organized people. It feels eventually we should converge on something that's kind of, you know, like easy enough to understand. And the space should go down a little bit. You know, we should understand more and more.

There's less ambiguity. It doesn't seem to be happening. How are you seeing it? Why is it hard? I think it's hard for a lot of reasons, right? One is that most of the work that we do is, let's say, invisible, right? Like you can't walk down to a production line and see where something breaks. Yeah.

That's part of it. And the systems that we use and we work with are increasing in complexity, right? When we think about, like, cloud systems, that changed the way we had to think about distributed systems and orchestrating them and architecting them. Now, we have AI. AI also changes the way we need to think about the infrastructure and the pipelines and the tooling and the model management, right? how we think about that and how we account for that, just its existence and its emergence with continued growth changes the complexity of the work that we need to do and the supporting systems that we need.

Another thing that can be challenging is many times leaders and execs aren't feeling the pain, right? How many CEOs have been developers and are currently, so some of them are developers, but also how many of them are currently using our systems to do work? and using them for at least a week or two, right? They're not managing their calendars. They're not, which they shouldn't be. But if they had to feel that pain, I think the motivation would change, right?

This is so interesting because when I talk with David Singleton, who used to be the CT of Striper seven years, he told me that every few months, he takes a few days where he does development work, which I found very interesting because not many CTOs at his level do it. He had thousands of people underneath him. At the same time, Stripe is one that invests very heavily into platform teams, into developer tools that they had one of their AI systems early on to help developers. And I was thinking, and he told me like, look, if you're not doing this, he said what you're saying, but very few people do it.

So I think that's just a really interesting point. I think anyone listening who's in leadership position, especially in a C-level position, if you're not doing it, you're, you know, there's companies that are doing it a lot better and you might be missing some things. Right. And in that case, because maybe for whatever reason, an executive team can't be carving out, you know, several days to do development work, Courtney Kistler is at Starbucks now. I met her back when she was at Nordstrom. She had a phrase that has always really kind of stuck with me and resonated and it's honor their reality.

if you ask a handful of developers and engineering managers what it's like to work in systems. You don't even need a survey, right? Like if you want to send someone, if you want to chat with a handful of them, you need to honor that reality. You might not agree with it, but that doesn't mean that isn't their lived experience of doing work. And sometimes that's going to be the best proxy you can get. But getting it is super important and listening to it and being curious about it is super important. This episode was brought to you by Century. Buggy lines of code and long API calls are impossible to debug,

and random app crashes are things no software engineer is a fan of. This is why over 4 million developers use Sentry to fix errors in crashes and solve hidden or tricky performance issues. Centricos debugging time and half, no more soul-crushing lock sifting or vague user reports like, it broke, fix it. Get the context you need to know what happened, when it happened, and the impact, down to the device, browser, and even a replay of it.

what the user did before the error. Century will alert the right div on your team with the exact broken line of code so they can push a fix fast, or let AutoFix handle the repetitive fixes so your team can focus on the real problems. Central HelpMonday.com reduce their errors by 60% and spend up time to resolution for next door by 45 minutes per dev per issue. Get your whole team on Century and seconds by heading to Century.io slash pragmatic. that is s-en-t-r-y-o-slash pragmatic or use the code pragmatic on sign-up for three months on the team plan and 50,000 errors per month for free.

Now, most people listening who are software engineers or engineering managers who are pretty hands-on or stay close to the teams, they will agree with both of us that this is important, developer productivity is important, it's hard. And if you're above a certain size, at mid-sized or above, you kind of need some investment. It's nice to have dedicated folks who help out, may that be provisioning infrastructure, standardizing things or building systems. The number one question,

or not the number one, but a very frequent question I get from people, especially injury managers, senior managers, directors, is I see this as a problem. Like, I think we should invest more into developer productivity or internal tools that help us with developer productivity, platform teams. I need to get funding for this and I like to make a case. What is your suggestion of doing so? And I'm asking you as someone who's advised a lot of companies and seen them, what are ways you've seen engineering making the case that there should be an investment in platform teams that directly or indirectly address some of these topics? Because the business, the usual pushback will be engineering expensive enough. We gave you enough resources. you should just start it out as you are.

Yep. In general, I would say some of the best ways to communicate include both data and stories, right? If we can have some numbers, that can be incredibly helpful. It can drive at home. If you only have numbers, it becomes abstract. It's just another spreadsheet or a number on a document. Now, if we only have stories, it can be easily dismissed as a one-off. That's just that one thing.

Someone is just like a disgruntled engineer, right? Someone, some team just doesn't like, it was just that weak. And many times resourcing will come down to like, what are the tradeoffs? And you want to take a realistic look. Are your engineers having to, you know, rewrite, like, re-roll scripts every single time they deploy? How much time does that take? So go around to do just a rough estimate, like at Shepard. Like at Chef, we did this early on, and we just had a spreadsheet.

This does not have to be, you know, super intense or rigorous. We had a spreadsheet where at the end of the week, we asked engineering managers to, like, kind of chime in on, like, about how long certain things took. Because then you can do some kind of, like, rough back-of-the-napkin math to say, this is how much time we're losing in aggregate. Now, if you want us to repurpose engineers, we can, but this is the impact on the product. Because we can come with a recommendation, but it's going to be someone. else's decision. And the more information we can give them about the tradeoffs that exist in the

system, the more valuable it will be. And then pair it with a story. Now, it can be an actual story. It can be a quote of a comment or it can be a very true amalgamation, like almost a persona of what a day in the life of development looks like now and then what it could look like later. And then it comes back to being a decision around like, where are we going to invest a resources? We're always in a constrained environment. No one has unlimited resourcing, maybe a couple of AI companies. Right. We always have constraints. So what's the best way to do this?

I will say one additional thing to keep in mind for folks, kind of kicking this off or making decisions, you know, about whether they should be investing in this, is to acknowledge both to yourself and to leadership that there is a tipping point in this tradeoff where further investments, explicit, you know, dedicated investments, won't deliver that much more.

progress, which I think is important because there's, you know, I've talked to execs before who were like, I'm not going to take this meeting. It's just a black box. Everything's broken all the time. So being upfront about the fact and saying, I know that there are so many problems and we could like spend all of the company's resources fixing all of it, I know that's not what we want to do. I am trying to optimize our workflow so that we can deliver value to our customers. And I think this is going to be a very important leverage point. And once we get to a point, we'll continually reassess where it's good enough, we will retain these investments, right? We'll keep a platform team, but we won't need additional investment until it starts

negatively impacting the business. And that kind of opens up a door to have a real conversation around like, what are the constraints that we have, right? Don't just ask for headcount. Yeah, I love the both that you're saying, making the case, well, first of all, making about tradeoffs because I think that's a lot of you used to work with. The fact that mentioning that you can later repurpose teams. And I think just a reality right now in the market, it's all about efficiency. There's now a push with a lot of companies where they're hesitant to hire more software engineers. So I do think folks will need to get a lot more creative and also just accept the reality that we might have, you know, we've had a really good run of the last 10 years where we like so many

places got almost unlimited investment in these areas as well. And which meant that engineers didn't have to feel some of the pain. We might need to, you know, just decide. that some of this pain it is here. Let's solve it. Let's try to, you know, work around it. But let's solve the most important pain point. So I feel there might be a bit of a turning point. A lot of companies, not all of them were like what you've been used to, you know, getting resources to just building teams that helped it. Either help yourself or sort it or look at tools that can make it change up your workflow. So is this also a little bit what you might be seeing? Yeah. Yeah, absolutely.

So talking about highly productive companies, now in 2025, some of the most kind of productive companies that U.S. Air teams, engineering teams I would say, that just get stuff done. What are some of the common trace that you see across them? And are there some common practices that they use? May they be a startup or a company, sorry, a team within a larger company. Many times what I get instead from a company is how do I make my company high. performing, right? But it really is very contextual to teams, right? Because there are team technologies, their team onboarding, their team cultures, there are team norms. So it can be a group of teams, right? But I often see very different, I'll say like performance profiles, right,

across a company. Now, what makes them great? And you can be great anywhere. I've seen a great team working on mainframes. I've seen great teams working on, you know, SaaS software. I've seen bad teams working on SaaS software. I've bad performing teams working on. We've all seen it, no? We've all seen this, right? We might have even been that team. Maybe, maybe, yes.

That's a yes. The thing that I see in these teams is a few, you know, key characteristics and values. Psychological safety is always in there, right? I'm sure several of us have heard about that. You feel safe taking risks. You feel safe asking stupid questions. you feel safe, doing so many things. Another part of psychological safety is depending on your team

and also knowing that your work has purpose. And that can come back to asking stupid questions, right? Sometimes stupid. They're very important questions as like, how does the project that we're working on right now relate to the group or the business goals? And does it? I was at a company once where we rolled some of these things out and a few engineers quietly back channel came to me,

and they're like, I don't think this aligns to any of the key, you know, the exec okay, R's. What am I doing? And so we were able to have a discussion around it does. It's just worded differently. And you're not only impacting your line of business, you're impacting three others.

Or I don't see how it aligns at all. You should chat with your manager because this might be a good opportunity for prioritization. Yeah. So psychological safety is one. A very related one is getting curious, being very open to asking questions and learning more. And I especially see this in large companies, being open to the possibility and frankly the high likelihood that the way you do things now is not the best way to do things.

Just because something was difficult or impossible to solve five years ago doesn't mean it hasn't been solved. And so when you open yourself up to the possibility that there could be better ways to do things, there may be other teams doing better than you. Learning, then you can learn what that is. It offers an opportunity for improvement, right? Like, I've worked with teams kind of in larger, older orgs where I came in, this was a problem. And they're like, oh, well, but that's not one we're going to look at because, like, that just can't be solved.

Because the last time they really put their heads down to try to solve it, it was seven or eight years ago. And it's actually been solved now. Like, I can point you to conference talks, right? And so being willing to consider and ask several questions ends up being super important. And then tactically, like, onboarding ends up being a great indicator, right? How long does it take a new hire to commit code? And how can we improve onboarding practices and environments? Because it often puts a spotlight on things that are really, really difficult.

I was working with one really large org once where their onboarding time for a brand, new college hire engineer was the same as a senior developer who just switched divisions in the company because the tooling changes, the documentation changes, the context they needed to get was basically non-existent. Those two things shouldn't be the same. Just to be clear, it was a long time, right? We weren't talking a day. It was probably like... We were talking several months. Wow. Okay. That's, okay, that's a real unexpected. I mean, that's just terrible. Let's say it how it is. Some of the best teams were at 60 days. Wow. Which is not great, right? Not great. It's bad. It's bad. It's bad. And when we work with them on some general interventions, right, what are some of

the concrete things they could do to improve it? There were basic things like improving documentation, improving how easy it is to find documentation, creating a dummy exercise for a dummy pull request very, very early that has to be completed within the first week or two. Now, what we found is that even if you know, so some research on this has been done at Microsoft, my colleague Brian Howe did a bunch of this, which is really incredible. And shout out to the teams that, you know, partnered with us on this. Even something like a dummy pull request, right? Not dummy, but like fix a title, right?

ends up being hugely impactful. It increases productivity, some measures of easily automated measures of productivity through the whole rest of the year, right? There was like a 30 to 50% increase because you've gone through the system. Now, people push back. They're like, what if you're gaming the system?

Great. This is the best way to game the system because then someone has gone through every single step of the process. They are now more familiar with it. They've done it early on something that was fine, but kind of didn't matter. Like, they didn't have to have full context of the entire code base

to get familiar with the tool chain. And surface... This is just so interesting. It was great. And they could surface obvious problems that other people... When I joined Microsoft, I joked, because some people just knew where everybody was buried. They knew where every document was.

They knew where everything was. But it's because they'd been there for 14 years, and they had a list of bookmarks. Bookmarks are not transferable, right? Now, I mean, the same was true. at IBM. The same ends up being true at many large companies and even small companies, because if there's just a handful of you, you can ping somebody and ask. Yeah. But it slows things down.

When I used to work at Uber, where I had a dock of like the things to do. And I think most people had after a while because there's so many systems. But what you said is just so fascinating, especially about the onboarding, because I didn't think, you know, as we were talking to developer productivity, I thought about a lot of things. And I think a lot of people are like this. They don't think about how onboarding can predict. But as you said, it's not just the onboarding, but the internal transfer. And one super interesting thing that might tie back to this is this, a famous thing, a famous law that a lot of injury managers will quote is Brooks Law, which says for a late

project, if you add more people, it's going to be even more late. But this is true at most companies where onboarding takes a long time, you know, like the project is like, estimate is two months. If you add like three more people, they'll take a month to onboard and suck time away, etc. But if you're working at a company and, you know, like a company that's kind of known for super fast onboarding externally at least is meta. New hires will often, you'll push something, code on day one, day two. They have mono repo. If you move teams, you will probably work on the same code base. If you have a organization where changing teams means that the new, the person who

changed teams in a day, they will push code to a different part of the stack. Brooks law might not entirely apply, assuming you can pick up parallel tasks, which a lot of people don't realize, because I once had this experience at a company where this was actually at Uber, where we had a, people said, like, if you get more people on your team, can you get it done faster? And at first I was a Brooks law. But my director said, like, no, no, no. Like, those people, they're onboarded. Like, they know it's the mono repo, the mobile monoripo.

They can do productive on day one. And we had more people and we got it done faster. And later I was thinking, how did we, because this is 50 years ago, it was quite 50 years ago. But as you said, I think this is the part where you need to be open to things potentially changing around you for the better. And that's where the question kind of becomes, when someone's onboarding to a team, are they onboarding to the entire system and the entire environment in which they work? Or are they onboarding to a slightly new team with like maybe a slightly different culture and learning a different part of the code base? But everything else remains unchanged. If that's the case, you know, maybe not similar productivity on day one, but, you know, a handful of days to really dig into the new, new part of the code base, new functionality, but everything else is just done.

Whereas sometimes, even if you're within the same company, you're onboarding to an entirely new system. You have maybe the same tool chain, but it's configured entirely differently. So it's basically a different tool chain. You have different request processes. You have different documentation. If everything's changed, that's onboarding. that's new hire onboarding. They just already have their login.

Yeah. Related to this, culture turnarounds, a lot of companies that you will turn to experts like yourself and seek out different methodologies know that we've got a problem. We're not very productive. We look at our competitors. So other companies, they seem to be getting stuff a lot faster. We want to turn around the culture. We want to go from a, you know, like do whatever it needs to be done. Have you seen successful?

turn around. And if so, how did they happen and over what time frame? And, you know, let's talk about like companies that are like, you know, like not, not a tiny startup, but like company that like 100 plus people or even bigger. And I'm more interested in like kind of the generic learnings because what I see is, you know, people at the top, they think like, oh, yeah, we'll just hire some consultants, you know, we'll become a tech first company in no time. I've actually worked at a company like this a long time ago, a financial company that kind of advertise itself. And on the ground, developers, are you just rolling their eyes? Like, okay, yeah, we're getting, you know, another agile training or whatever that is.

And you feel not much is changing. And there's usually a disconnect between the two. And there are some companies that over time, by the way, like they do manage to become a lot more tech first companies, like old school companies who you'd be surprised, you know, like. A lot of banks are here. Exactly. Yeah, I've worked with a lot of banks that have kind of undergone that transformation successfully. Parts of the government.

For the ones that succeed, what do you usually see in terms of success, the time it takes, and what do engineers feel on the ground? You're right. I end up talking to a lot of folks about it. I think it's important to realize that if you're trying to change the tech culture, that's not going to be possible. And by that, I mean, you need to think about changing the culture of the company or the culture of the organization. You can do things to have it and help it influence the technology culture, but it's really difficult to try to just carve out the technology culture and change the way people think about software without changing the broader culture.

And I will say one, you know, I think notable cultural transformation is at Microsoft, right? Satina Della has really changed what Microsoft is like compared to, you know, the Balmer years or other. I used to work at the Balmer time. It was very different. Oh, my gosh. It's very, very different. Now, there are a handful of things you could do in general and then specific to tech. I'll probably focus on the specific to tech ones because I'm sure other people are familiar

with, you know, communication and commitment and those types of things. When we think about technology, I think we have additional opportunities because there we can look at some previous research that's been done, John Shook did some great work. We used to think that to change a culture, you would change. or to change the way people do work. You would change the culture. You would change the way you talk about it. And then that would just trickle down into the way people build tools and, you know,

kind of execute their work. What he found is that when you change the way people do their work, that bubbles up to cultural impacts. And I think that's where having an environment where these changes are allowed and supported is super important, right, from that broader environment. But, you know, as an example, people talk about NUMMI all the time, but I, the cart factory, but I love talking about it, kind of situating it in companies. If you're in a company or in an organization or a team or a situation where your tool chain

is just incredibly slow, it's very high friction. There's a lot of process to go through. You can talk about doing agile and it's going to help to a point, right? People can get their minds, wrap their minds around it. They can think about what it means. But actually changing the core culture isn't going to happen if you're working in systems that are incredibly slow. Now, if you introduce new tools and new ways of working that are actually much faster, that actually make it easier to communicate with others, that make it easier to surface and fix problems early, that make any prioritization decisions that you're talking about fairly easy to get done. Now you've changed people's lived experience.

You've changed what it feels like to do the work. And that's paired with larger communications, larger education, larger commitments to what is happening. Now, commitments is also important because are you committing to improving the difficult pieces of the technical stack and the workflow to get to the desired outcome? Are you investing in the communication that's required? You have to say things at least three times for it to kind of land. and it needs to come from several areas and several levels of the business. And so I think those two together can be incredibly powerful, but also needs to be explicit. And just being a little bit more specific, say I'm either a staff, senior plus engineer, senior,

principal engineer who joined a tech company or an engineering manager, a senior injury manager, not a C level, right? So I can't set the things. But in this sense, what is it that I could do? I've kind of onboarded. I took it in. I didn't rush to, you know, like copy, paste whatever I saw the previous place. You know, this is a frequent criticism of big tech employees joining startup bringing process.

But we didn't do that mistake. I took it all in. I realized there are things that are just really inefficient. I can see, you know, if I was a CTO, I could do stuff, but obviously I cannot do that. What are ways that I can improve the engineering culture, the efficiency? And is it going to be what you just said, which is around the lines of starting to change my team's lived experience, for example, and hope it bubbles up? Yeah, I think it's going to be a mix, right, just on a smaller scale. Right.

So what is within the scope of your control? And it'll start by, I love that you said, you know, it's when you've joined. You know, you've done your first 90 days. You've done a listening tour. You've been able to kind of identify what some of the challenges are. It can be super important to then, you know, write up a summary, run up. past one or two people to make sure, like, it kind of makes sense. Yeah, sanity check.

Yeah, sanity check. Meet with the teams. Say, this is what I'm observing. This is what I'm hearing from you. Does this resonate? Does this sound right? Is there anything I'm missing? And then end that without a big call to action, but say, I would like for us to look for

opportunities to improve this. And I would love for you to be involved if that's something that you're excited about. give people, you know, a little bit of time to kind of like marinate in that, kind of digest. And then go back and say, what are some ways that we can improve this? These one or two things really stood out to me. What does that, what does that mean for you? And then depending on, this is where some resourcing comes in, right? Because we want to change the way that people work.

And so that could be creating new education if it's just about, you know, changing a process. It can be changing. It can be improving the tech. And so we can have discussions about either, you know, kind of carving people off or maybe setting up a hack day. Sometimes a hack day can be great to just deal with paper cuts. Yeah. And that already, the thing that's great about that is you get this quick win. It's visible.

It's a quick win. It impacts people so they see it. And if it doesn't impact me personally, maybe it impacts my coworker. So I'm at least seeing progress. And then we have continued communication, right? Again, I also love what you said, which is, you know, talk with people and basically make allies because we're at least see progress. because when you start a new company, you have no real social capital beyond maybe a fancy title or, you know, if you have some oppressive credentials, but not there. And I think what people forget a lot of times because I've seen so many times where someone on paper, you know, at a high, comes in at a high level, tries to make changes.

There's a pushback because, you know, like, we just want to do our work. Like, right? Like, what is this person trying to do? And then they eventually they leave eventually disappointed that this company doesn't want to change. But what they miss doing is, first of all, not realizing it takes a longer time. You do want to build a team who supports you. You understand them. They understand you.

And now a team. And then later another team joins them and do this. So I feel like you've kind of conveyed this really important. And the fact that it's also not a sprint. In a sprint, I mean, you can do something, but it'll probably be washed away with like a sandcastle on a beach. Right. Yeah. And I will say I have found having aligned incentives ends up being important as well, right?

If you say something's important, acknowledge it and follow up with that actually being important. So if you're just on the team, sometimes that can be a little tough if you're just an engineer, like an IC you just started out of school, right? But you can thank people. You can highlight what they've done. You can send a thank you email on CC their manager. And as a manager, you can say, this is something that I find very important and we decided together is important. Any work that's done here, even if it's not feature work, is going to be valued.

And then reflect that in any performance reports or one-on-ones because it can be really discouraging to go along with like some transformation effort or fix effort or whatever. And then it just kind of, to your point, right, it just kind of washes away. It just goes away. So far, everything we talked about might as well been two years before, because before chat, GPD came out before any of the AI tools went mainstream. But these days, software engineers are using AI tools, coding ID assistance, and many others. It is changing software engineering as a whole.

How has it changed your thinking about developer productivity and how we should think of developer productivity and developer experience? With AI, I'm getting so many questions. now about how can we improve the way developers work? How can we make sure that we don't detract from their work? Does this impact space or dora metrics? So I'll quickly say, dora metrics in general won't change, right? Like their utility, I expect to remain about the same

because we're looking at how well that kind of outer loop is running. In terms of space, I think it's still incredibly applicable, right? Because we want to know how satisfied people are with the development experience. We want to know about performance outcomes like quality and security. We want to know about how many things they're able to get done the way they communicate. I think efficiency and flow is also super important because it's changing the way we do work, right? When we're writing code, we're writing code, but we're also now reviewing a lot more code than, you know, it's almost a review exercise instead of just purely writing.

There may be opportunities, I think, to expand space, right? Because trust ends up being a much larger factor. to everyone, right? So I studied trust with sysadmins and people doing, you know, high consequence, highly complex work and trusting a system and trusting the outcome of a command and a new tool was incredibly important. Developers didn't see that as much because, you know, we go through school, we use an IDE, we use a compiler, right? Does the compiler work? Like, are people, people are not asking, does a compiler work? And I understand, like, the underlying math is very different. But suddenly, you've introduced this question to an everyday, all-day workflow, is this the right output?

Can I trust what I'm seeing? So I think it's good to kind of capture that and watch. Now, what does that mean kind of more broadly? I think there, and we're seeing a lot of opportunities to kind of reinvent development and software engineering, right? We're seeing it in the IDE right now. We're seeing it in tests. We're asking questions around code quality.

There was a great study done with GitHub and Office of the Chief Economist here that found that for developers that had been seasoned developers, right? They have at least five years of experience. They were finding that, you know, they're saying that the code is easier to read. They have good approval ratings. They're seeing over a 50% increase in unit tests getting passed. So it can have some really good. And this is just from devs using, you know, tools like copilot or other AI coding. assistance. Yep, yep. It's just from that. Interesting. And these were the reviewers saying that,

you know, the PRs that are coming in just seem like nicer to work with. And, you know, you can have a PR review cycle that is done, you know, we're used to linters. Why not have an AI take a look and offer some suggestions, you know, that doesn't mean it's inevitable. It's approved. I think there's already like startups that are building it, but I cannot see this not being the case. It's just feedback loops, right? Right. Some folks, it's not quite on their radar, though, because we are so many times when we think about a developer, we think about someone like sitting there typing, right? It's like it's hackers or something, right? It's very IDE focused. But there's so much more to the development tool chain, right? One thing that my team and I are focused on that I'm like real kind of bullish about is how do we think about the rest of the software process? How do we think about roles like release engineering and deployment? Because historically, that's been very difficult. and something that a person makes a decision on. And it's basically a risk-based decision and they have to take data from the builds,

from the documentation, from the downstream system, from so many different places and make a decision. This is an opportunity for LLMs to really jump in, right? Because the person is still making the decision, but instead of spending days collating all of this data and trying to summarize it, you can instead get a quick summary with links out to what you want to look at

so you can confirm or see if there's any like, or edge case or something. And then you can make a more informed decision. So I'm personally excited about several ways that the AI can kind of help bolster and support and, you know, reinvent and disrupt, right, the way we do the full engineering life cycle. Earlier you told me that some of the most productive engineers that you've, or teams, but also engineers, you've seen. are curious, but also they don't hold on strongly to beliefs.

They're open to things changing. And you gave the example of like, you know, like a team tried an approach eight years ago and it failed. But now it's actually it was possible. Do I sense correctly that it's probably a smart approach for a lot of us engineers to just kind of put away a lot of our reservations because LMs are just fundamentally changing. A lot of things. They'll turn out to do some things wonderfully. Some things will just crash and burn. but unless we do this,

we'll probably be that team that kind of close itself down to how things are. Is this the kind of advice you give to engineers? Or what would tell people? Because there are some experienced engineers. They've been around the block. And there's some people who are skeptical. Like, okay, they're just a fancy auto-complete. We're giving them too much credit here.

It's not going to change what we do at the fundamental level. Or are there concerns that it will introduce errors or security bugs or other types? types of things that like I would never do or my team would never do. Right. Yeah, I would say if you're in an organization that is, you know, deploying or allowing tools like co-pilot and tab 9 and cursor, really take it for a span. See what works.

See what resonates. See what lands. If you don't like it, don't use it, right? If you only like it for specific types of use cases and specific types of tasks, leverage it for the things that it is good at. And don't use it for the things that don't fit into your workflow, that don't fit into your mental model. But give yourself a little bit of time to open up what that mental model is. Because it can shift and it can shift in ways that are very, very valuable, right?

It can open up time. Now you don't have to do like all of the setup and your code. You can just focus on really solving the problem, really looking externally, really trying to figure out where the block. is or what the longer term architectural components should be. Now, when I talk with developers and also what I'm building with AI, I see two types of usages. One is you have just specific talking about the coding. So, you know, I know what the problem is.

I know what I want to do and I now need to do the work. One of this is you have the inline AI assistance, which, you know, show automatic code completion, may this be co-pilot, a cursor or some other ones. And there's a, and you know, it kind of helps you move faster as you go. It continuously gives you that additional context, but it also kind of takes away your attention a little bit as soon as you type it, it shows up things. An interesting thing that I talk with a developer who's actually building a game. He uses AI very differently. He says he actually goes to one of these tools, chat GPT or Claude or one of the others.

And he goes there with like a skeleton code, pasted in with a to-do and it generates. And I asked like, hey, aren't you using it in your ID? And he said, like, I actually, I find that I can focus way more when I don't have this thing interrupting me all the time. But I'm like, no, here's, I need help. And I want to get your feedback. And I'm iterating with you versus going, I'm in my flow state. Are you aware of either some research or even just personal observations on how developers can preserve flow? Is it important?

Or can you actually have a flow state either way? Is it something that everyone needs to figure out for themselves? I just want to say that story you started with, I think, is incredible. Because someone played around with AI and they figured out the workflow that makes sense for them, that works for them, right? It doesn't have to be, you know, line completions inside the IDE. It could be writing up a chunk of code and then asking it for review, right? I will say anecdotally, I've seen folks sometimes adopt that type of workflow or they'll turn off autocomplete or auto suggestions as they're typing. I haven't seen any hard data on that, although I wouldn't be surprised if there's some coming out.

In terms of flow, there are a few things that we've discovered and noticed. So there's a paper that some colleagues of mine did here in MSR around. They used the Cups model. So it's how does coding change, right? You write something, you wait for feedback, you have to reassess it, you have to decide what's coming next. And some of that work and early work we did with the GitHub team even influenced how the timing of suggestions happens, right?

We ended up slowing some down because, as you pointed out, if you're getting like suggestion after suggestion after suggestion, it ends up being an interruption. When I think about your question around how do we think about flow, I think everyone will kind of need to play around with this and see what works for them and what that means, right? Because when you're coding in an IDE, I know sometimes I'll turn it off because I just want to code, right?

I did that this weekend because I wanted to play around with some code. And I wanted to be kind of driving and doing all of the writing. But that also is, I think one helpful way to think about it is imagine a world, we're here, right, where there's just a different way to do the work. And maybe that means, you know, you call it something else in your head, right? Maybe instead of coding, maybe I'm driving and reviewing or I'm guiding, right? Because if we can think about it differently, then we're not comparing it to something that's not the same thing. And we can find what that new workflow is.

I like it. I also like your initial suggestion of driving versus guiding. It is just very different. But I do think that a lot of us engineers just need to accept that is different. It will change. We'll look back this 10 years later. People will look back. People who are starting their career now.

They'll be like, oh, yeah, like a little bit like anyone who's a coder. It's like, yeah, stack overflow. It always existed. I mean, interesting enough, now we've kind of stopped using it. But there was a time where it was just a given. And there were people before and people after. So I think we're going to have that. It's a good and timely reminder.

And also, we're just in beginning of this, right? Like this whole thing, it's just wild how much it's gone in such a short time. I don't think there's been any technology and history of software engineering that has spread across a whole industry so quickly. And it's changing how people work, changing their productivity, what they're doing and still work in progress. Yeah, absolutely. I totally agree, right? When I think about, you know, big impactful technology shifts in tech, cloud is one for sure. But it didn't hit almost the entire industry within a year, right? Like we're seeing an AI. And specifically like LLMs, right? To the AI folks out there, AI has been around for years. Anytime you're doing any type of auto-complete in an email or, you know, in Telecode, this is all AI. I think LLMs. really changed. These foundation models really changed the way we can do work and the types of development

we can do. So closing up with some rapid questions, I'll just ask and you fire off whatever comes to mind. The first one is about coding full-time versus being a researcher. You were a software engineer, now you're a researcher. What is the one thing that you missed from coding full-time? I miss just like packing something up and being in the code. The thing I love about research is that it's about identifying and solving problems that are hard to solve where we don't know the answer.

Coding is very similar. And I love that. I kind of miss the thought process of figuring out the best way to write the code. What's the best way to structure the code? How do I want to do some of the work, right? Like, I'm a joke. Like, do the typey, right? I miss those parts of coding.

And luckily, I kind of shifted into something where it still kind of scratch. matches that itch of where are the big problems? What's the strategy I want to adopt? There's strategy in writing code. There's strategy in communications and product direction and research program. Plus, even if you're not coding, you're working with people who do code. Yes. And like I said, I still play around with it, but missing. I think a little bit also is like the rush of writing something for customers that I know is going to be a production system. I had folks at IBM a couple years ago, contact me and say that one of the systems

I wrote there as an intern was still being used. And I was like, awesome. First of all, sorry. Second of all, it's, there's just a different kind of excitement around writing code and seeing it deployed and pushed and shipped to customers, right? Which, luckily I get a little bit with some of my research. I'm realizing now, as I tell you this, like, I've found another proxy. I've found a way to kind of duplicate some of my coding and just another discipline.

Well, and you still code a little bit. Yeah. What is the next book you're working on? If you can tell us. Yeah. Right now, I'm working on a developer experience book. And it's kind of geared toward folks who want to think about, you know, improving developer experience.

They want to learn more about developer experience. It could be an IC or an engineering manager. It could be leadership. So we cover things like, I'm writing this with Abinoda. So we're covering things like what type of data? Should we be thinking about how should we communicate data? How should we communicate? like the ROI, right?

A lot of this is communication. How do we want to think about a long-term analysis and data collection kind of program? This is not a stats book, but, you know, getting people pointed in the right direction, right? Examples of survey questions, examples of types of data, examples of ways to operationalize because my kind of heuristic is once I've had like at least 100 people ask me a question

that ends up being in the same flavor of answer, right? It's very, very similar. something written, something like a book can be really helpful because then everyone can kind of cover basics and then think of the next level questions that they're going to get to eventually. But it's not a good use of time if like I'm a blocker if we're trying to do something over an hour. What's a practice or a tool that helps you personally be more productive on the day-to-day basis? Day-to-day. So day-to-day at work, I don't want to say a tool. I've got an incredible PM who has absolutely up-leveled all of the work that we're doing. doing. Jason Intimit, shut out. Having a very good PM-TPM could just be a game changer. I had an incredible one at Google too. It was just so, so good. But when I think about technology tools,

not in my work life because I kind of maintain this, I for sure maintain like a security boundary. Claude, claw is incredible. I'm able to use Claude to do many, many things. And that I would say has been the most notable impact. It is very interesting how the competition between LMs, it's just so great because you know, Open AI obviously started all of this and now there's entropic, Claude, Sonnet, in many ways, you know, like jumping ahead. But I feel it's just so great for the ecosystem. There's so much competition everywhere and they're all inspiring one another to become just

so much better. So I feel like as someone using all of these things, we're just in the middle of some of the best time in the sense of like getting all of this, these things for free because, you know, these teams want to like outclass one another and we're just benefiting from it. And, you know, like you said, it's so early that everyone is still learning. And I'll jump back and forth because some models are, you know, better for something else. But just even having someone to like. It can change every month, right? Yeah, it can change all the time. But having like a rubber duck that responds to.

to some of your questions. Sometimes I just want the rubber duck, but sometimes I really want to think through something. Like, what are my limitations? What am I missing? What are the holes? Okay, no, that's not actually a hole. What else would it be? It ends up being really useful, and it's super, super fast. And what are one or two books that you read and would recommend? It's tech-related, inspire by Marty Kagan is great. I'm still making my way through parts of it because it is not a small book. Yeah, it's right over there. It's really kind of solidified and been and articulated things that I've kind of known but like in very fuzzy way. And it's also like introducing some new things for me to think about, which I really appreciate.

Awesome. Another one that I loved, uh, that was just kind of like practical. I just like to dive into like random things. Um, the outlive by Peter Attia. It's just I like to nerd. I want to like nerd out and learn something, which, you know, my partner jokes is like, I've heard a lot of good things about Peter Attia's, from people working in tech. Well, Nicole. And then for a fun book, I always love Enders game. It's like an oldie but goody.

So if I just want to like escape, I'll just grab like a trashy beachy read or I'll grab like Ender's Game or Ready Player 1 or something because it's just a fun story that I can just race through it. There's such fun stories. I feel the movies were fine as well, but nothing beats the book. Yeah. And then, like, for news and articles, Tressie McMillan Cotton and Anne Helen Peterson are probably the most high value rates that I get outside of tech. Awesome. Well, Nicole, thank you so much for being on this podcast. This has been so fascinating, interesting, and I think just, like, motivating, but both on how much we've come with developer productivity and also just when we're getting closer to figuring things, all, things change again with AI and, you know, teams behaving differently.

It's exciting time. Yeah, it's really fun. This is a great time to be working, especially in tech. Thank you to Nicole for this deep dive into the ever-evolving field of developer productivity. You can read more from Nicole on her website or follow her on social media, both linked in the show notes below. In the Pragmatic Engine, we've done several deep dyes on developer productivity. Check these out linked in the show notes below or by searching developer productivity in the Pragmatic Engineering newsletter. If you enjoy this podcast, please do subscribe on your favorite podcast platform and on YouTube.

too. Thanks and see you in the next one.