ACM ByteCast - Nicole Forsgren - Episode 81

Episode Date: February 4, 2026

In this episode of ACM ByteCast, Rashmi Mohan hosts software development productivity expert Nicole Forsgren, Senior Director of Developer Intelligence at Google. Forsgren co-founded DevOps Research and Assessment (DORA), a Google Cloud team that utilizes opinion polling to improve software delivery and operations performance. Forsgren also serves on the ACM Queue Editorial Board. Previously, she led productivity efforts at Microsoft and GitHub, and was a tenure track professor at Utah State University and Pepperdine University. Forsgren co-authored the award-winning book Accelerate: The Science of Lean Software and DevOps and the recently published Frictionless: 7 Steps to Remove Barriers, Unlock Value, and Outpace Your Competition in the AI Era. In this interview, Forsgren shares her journey from psychology and family science to computer science and how she became interested in evidence-based arguments for software delivery methods. She discusses her role at Google utilizing emerging and agentic workflows to improve internal systems for developers. She reflects on her academic background, as the idea for DORA emerged from her PhD program, and her time at IBM. Forsgren also shares the relevance of the DORA metrics in a rapidly changing industry, and how she's adjusting her framework to adapt to new AI tools. 

This is ACM Bytecast, a podcast series from the Association for Computing Machinery, the world's largest education and scientific computing society. We talk to researchers, practitioners, and innovators who are at the intersection of computing research and practice. They share their experiences, the lessons they've learned, and their visions for the future of computing. I am your host, Rashmi Mohan. For most of us in the software development industry, the pace of the technology, the pace of Delivery of Services and products is of paramount importance. As we bandy about terms such as time to market, scalability and first-mover advantage, we don't think much about the force that is

powering all that innovation. Our next guest, however, has made it her life mission to measure and improve engineering teams, keeping developer satisfaction front and center. Dr. Nicole Forsgrin disrupted the industry by co-founding Dora, the largest and longest, and longest running research program dedicated to proving which capabilities actually drive software delivery and operations performance. Today, as she researches the impact of AI and the causes of organizational friction, her focus remains on using data to help build high-performing, sustainable, and enjoyable engineering cultures. She is currently the Senior Director of Developer Intelligence at Google, has helped multiple senior positions at Microsoft and GitHub software in the past

and has also been a tenure track professor at Utah State and Pepperdine Universities. She serves in the ACMQ board and brings impact to every role that she plays. Nicole, it is our absolute privilege to have you at ACM Bytecast. Thank you so much for having me. I'm very excited to be here. Lovely. I'd like to start with a simple question that I ask all my guests, Nicole. if you could please introduce yourself and talk about what you currently do, your current role, and give us some insight into what drew you into this field of work? Sure. So as you mentioned, right now I am at Google and I'm working on finding ways that we can

really improve our internal systems for the developers who make software for the company and for others, right? And so I lead a set of teams that find ways to measure this work, analyze the things that improve it, discover emerging workflows and agentic workflows, which we're saying is like really changing rapidly right now. It's really exciting. And then how can we kind of affect that change throughout an organization? And I think it closely relates to the work that I've done for so long. I think I originally got into this because I was a software engineer at IBM. And it was a wonderful learning experience. You know, I was able to work on some really incredible projects. And at one point, though, so this was maybe like pre-Devops days, but there

were like whispers of it, right? So I remember speaking with management and saying, well, what if there's a different way to do things, right? And they're like, no, no, no, that won't work here. And there were a couple of very large projects that they did at the same time. And it was really, really challenging. I'm sure there are engineers out there that are kind of smiling and nodding, thinking, you know, we scoped it out, we told leadership we could do one project or not two, and they told us to do two of them and half the time, right? So, you know, so many of us have have lived in this world and I saw how challenging it was, right? Because people really cared so deeply about doing their work. We all tried so hard. And at the back end of it, both projects were

fine, but like not a great success, but like they were success because like we got them done. But it also had such a large impact and blast radius on the engineers themselves, right? It's really hard to work under those kind of constraints. And, you know, I thought about it and I was like, there has to be a better way. And there were really. whispers that, you know, what we now call DevOps was maybe that way because if we could work more efficiently, it would lead to these business outcomes of moving faster and shipping value in better, more direct ways. And it would also be better for the engineers and the developers on the back end. And so that is kind of what kicked off a lot of this work that spans a couple

decades now. Wonderful. I'm so happy to hear about that. I mean, I have so many questions related to that. But maybe I'll step back even further. What got you interested in computing in the first please. Oh, this is such an interesting story. So I had a hard time picking majors when I first started college, right? I enjoyed English. I enjoyed math. I enjoyed psych. And so I just started college, and I think I was a psych and family science major. And that first year, my father was in a snowmobile accident and was in a coma for a while. And I'm the oldest child before. And it kind of hit me. And I was like, I might be a little on my own. And for some reason, I don't know why, but my brain decided, like, that meant that I needed to be able to make money and soon, right? Like, in case something

didn't work out, I'd need to be able to make money. And at the time, I was on the varsity volleyball team, and one of the girls on the practice squad mentioned that, like, she would be able to make $40,000 out of a two-year degree. And at the time, that was about as much as you could make with a master's degree as a high school teacher. And I was like, well, if I stay with psych and family science, it'll take me too long to make that much money. And so I just walked into the cancer's office and I changed my major. And I was like, I like math. I'll be able to figure it out. It's fine. And if I don't like it, then I'll just change, I'll figure something else out. And I ended up loving it. So my first computer class was my second year in college. That's incredible. I mean,

I love the way you kind of, kind of kept your mind open. I mean, obviously it came from a very personal space of like, hey, I need to do something about the situation that I'm in. But I feel oftentimes, you know, you have so many, I think computing is one of those fields that people enter from so many different sort of pathways. It's sort of the most open and welcoming in that way. So I'm so glad because, I mean, look at the kind of impact that you've had at the industry from the time that you've joined. But going back to the stories about, you know, software development and looking at like,

hey, there's got to be a better way. Most of us who work in the industry tend to be sort of bogged down. by what the organization offers. So we try to sort of innovate, but within the sort of limitations of the straight and narrow, if you will, right? Like, okay, here are the things that I can do. Maybe I'll experiment a little bit outside my organization. But you took a completely different approach. You decided to kind of say, hey, I'm going to sort of completely turn this on its head. So what was your next step when you discovered that, you know, well, there's got to be a better way. What did you do next? So my next step after that was saying, well, if I'm going to find a better way,

I want to know how to do that. And that's doing research. That's basically going to be a research degree. And so I went to get a PhD. One of my jokes, but it's not a joke is if I was going to be able to have these conversations and convince people, I wanted to, quote, win fight with numbers. If there was going to be a disagreement, I wanted to have some kind of evidence that this was a good risk to take. Because like you said, it can be much safer and much easier to kind of work within organizational norms and organizational boundaries. And so if I was going to be able to be able to convince people that there was another way to do it, or if I could convince myself first, that there was another way to do it that would be impactful. I wanted to have good, I said data

behind it, but like good methodology, good stories. I wanted to have asked enough people in the right way and have done the analyses the right way to kind of see what that would look like. Got it. And I've heard, I don't have a PhD myself, but I've heard that one of the most challenging parts of the PhD is actually trying to understand what you want to research, right, forming that initial question. In your head, you already had an idea when you sort of enrolled in the program, if you will. Were you always certain that this is exactly what you wanted to do? And sort of did Dora, which stands for DevOps research and assessment, did it sort of evolve from those early interactions that you had? Dora definitely evolved from that. But I'll say when I started, I had a vague idea

of what I wanted to do, but I wasn't sure. So my first year, I actually worked on projects in neural networks and deception detection and semantic analysis. And I thought, you know, maybe databases would be the thing because like that could be one angle to take a look at this. And then I want to say partway through my first year, I was back working with IBM and I did a user study where we wanted to see how system administrators would respond to a new interface. And my colleague and I, Todd Isid, did the study, and we wrote up the findings, and we kind of shared them back to IBM. And they came back and they said, well, this is interesting, but we followed UCD guidelines. You know, like, we did this following user-centered design guidelines. And to be fair, like,

this was really the way most people were developing software. It's like, there were users, and like, that's how they kind of did their work. And what I realized is that there wasn't a great understanding yet of in this case, specifically what it meant to be a sysadmin, what were the risks they had to weigh, what was the additional information they needed to see to assure them that using something like a GUI would be safe. Because when you're used to using a CLI and when you're used to just typing into the system, you can get a lot of information really quickly. You know, it'll echo your commands back to you. You can check on system state. And if that wasn't built into the GUI in ways that were super easy to find and fast to find and accessible, it would be hard to do that. And I think my next

kind of mental step was, well, for people building really complex systems, there is probably another way that we need to view this problem, like separate from the way we've been viewing it so far. Yeah, I love that you say that, Nicole, because what comes across in all of your conversations that I was able to cure some of them as well is that the persona of the developer is so central to, you any sort of solution that you're trying to propose. I mean, to your point, I think the CLI gives you a sense of power. Like, okay, I know I can get my information. I know what is happening.

The GUI kind of abstracts some of that. So I can see why there may be some concerns or fears when somebody is sort of first getting introduced to a new way, new paradigm of interacting with the system. Yeah, absolutely. Yeah, no, that's great. And so from there on out, I mean, you shared this data. And then what next?

did you complete your PhD? Do you have your thesis done? What was your next step? Yeah. So actually at that point was when I kind of settled on like this was the direction that I wanted to go. I switched advisors. I started working with some different folks and really started at that point focusing on some system administration. Early days of DevOps, it was actually called Agile System Administration. So this was like a really good spot to start digging into and investigating some of the work. And again, IBM plays such a good role in some of my early, maybe superhero or villain origin story, depending. But IBM Research Lab had a handful of engineers that were doing early work into Sysadmins, and they were the only ones doing any of this work. Haber, Candigan, Maglio, a bunch of these folks were looking into it.

And so I reached out to them to say, like, what are you seeing? They were doing incredible ethnographies. And I said, I wanted better understanding of how. sysadmins are doing this complex work. What are you saying? What are we doing? They ended up being reassigned to another project. So IBM let me come in to IBM research and do basically lead a bunch of that work for several months while I was finishing at my PhD. And so that became part of some of that thesis work in that dissertation is understanding how do sysadmins kind of think about and understand design and kind of execute this complex work. And so that became my dissertation.

and pieces of my postdoc. Got it. Yeah. And so you were talking about the fact that DevOps was still very nascent as an idea. I know in many of your talks, you also refer to like that flagship public moment was from the team at Flickr that you refer to and you say they claimed 10 deploys a day,

which was sort of like a, oh my gosh, mind-blowing moment. So you saw that shift in software delivery that really sort of moved into this more continuous delivery culture. What do you remember about that time that kind of blew you away. Oh, I remember, I mean, some of the conversations, right? So for folks who kind of go find the video of this talk online, it was really interesting because like, you can watch the talk and you're like, okay, 10 deploys a day. But when you were talking to people who were at the conference or, you know, the hallway track, there was kind of this split reaction where

some folks were like, 10 deploys a day, this is amazing, this is incredible, we have to learn more, we have to figure out how to do this. And then there was, was another set of folks that were like, this is chaos. This is absolute madness. There is no way this is sustainable. We can't do this. Like, this should not be a talk. Why are we trying to promote this? And so there was this just incredible dichotomy because it was like this how. And so I think that was probably what also kind of spurred me on just a little bit to again, you know, at that point, maybe branch further upstream. It's so funny now that we're in our AI moment. You know, everyone's really thinking about writing code and this interloop. And it's like, all I started was

admins and systems and worked my way up through outer loop into interloop because there were so many opportunities there to improve the systems in ways that weren't always top of mind for folks. So can you explain the concept of inner and outer loops a little bit more, Nicole, for our audience? Yeah, sure, of course. I will say everyone has their own definition of understanding of the boundaries, but very roughly, inner loop is everything you do yourself, right? So like here's the joke of like, well, it works on my laptop. I write code on my laptop. I do maybe early tests. I do a quick compile. If this works, that's my inner loop. Now, once I go to merge that into mainliner trunk, that's for many folks, kind of the handoff point where we have this code review, this peer review.

And then once we start hitting the rest where we have integration tests, integration builds, release, deployment, that's going to be your outer loop. Understood. Okay. So at that time, I think what you were trying to say is that, you know, as people were saying like, hey, I'm moving from whatever it is that my pace of delivery is to this 10, which sounds like a massive number, what did you see as the opportunity there in terms of saying, oh, okay, this is an area that I can spend more time on to not just help people adopt this sort of new paradigm of software delivery, but also help measure it. So I think that was where there was a huge opportunity, right? Is the inner loop, we can measure and we can watch and we can see, right? Even if you just go like sit next to someone with a stopwatch, like we can understand what that looks like. But the outer loop was basically a black box, right? Like you handed it off in like post code review, it just kind of existed. And in many, many cases, it existed with a bunch of manual steps and handoffs and transitions and transitions

between systems and different rules, right? Many teams had their very own processes for release and for deploy. And I realized that no one had good visibility into this, and I wanted to find a way to even measure it, right? Like, is there a significant difference between folks doing DevOps and folks not? I will say I started this project, the Dora project, kind of state of DevOps reports with Puppet, in particular in 2013.

And so this is still like earlyish days of DevOps. Alana Brown, so much credit to her for kicking off this vendor study and realizing it was like such an important trend and then allowing me to join as a researcher and kind of like help, you know, collaboratively rethink and reframe how we thought about this problem. But when we wanted to measure that end to end, it was basically impossible. There were no systems that had end-to-end telemetry at a single company, let alone several companies. Honestly, there are very few systems that have end-to-end telemetry today. And so I was like, this is a really interesting problem.

And we don't have a technological answer or solution to it. And so as I was chatting with some of my colleagues and as I was reviewing my earlier papers into some of the sysadmit work, I realized there might be an opportunity to leverage psychometric methods to measure this. So what do I mean by psychometric methods? Surveys, basically.

But very carefully, rigorously defined surveys, where we ask questions very specifically, and we ask for perceptions about various pieces about how you are running your pipelines or how you do your tests and how your team kind of accepts tests and move on. I was using something called latent and manifest variables, which is a way of saying, or answering,

you know, kind of responding to the challenge of, well, what if someone misunderstands the question? Are you sure you're asking the question you're asking? And so there's some statistical methods we could use basically ask several questions about behaviors that happen. So for automated testing or for, let me do CI. So for continuous integration, it needed to be green. The team needed to prioritize it being green. You needed to have feedback quickly in addition to a check-in automatically triggering tests and builds. And we found that that combination of things is actually what was predictive

of better outcomes. And so by asking various different pieces of what happens when you do your work, we could come up with an idea of, okay, well, these are the things that we mean when we say continuous integration when it's successful versus asking anyone else because everyone kind of had their own definition in their heads. So by doing that, we were able to kind of pull together this map of the core capabilities that were important and then statistically test to see if they had relationships to things like speed and stability. I will also say that's one of the things that we did first was instead of trying to go from investment in technology directly to profit and revenue, that was challenging, right? And so I kind of took a step back and I said, well, if it's important

or if it's impactful, if it's doing something for the company, that will be through the software that the company creates. And that software will, right, because like you've got to sell something. And so for technology companies, it was kind of leveraging software. And so that would present itself as your ability to update and ship new software relatively quickly and also with some good stability and reliability. And so once we introduce that in particular, that's when a lot of things kind of came together. And it helped us get a much better understanding of the practices that teams were using around the world. What are those practices, because technology practices, they're part tech, they're part sociotechnical, they're part cultural, which

ones were driving better outcomes and what some of those outcomes could look like. You know, I find it fascinating that you say. The one is, you know, you're working with individual teams because you're absolutely right. Understanding their processes in terms of what they value as, you know, everybody's processes are slightly different. And so to determine what your final metrics should be, you have to work very closely with them. What was that journey like or do you have any challenges at all in helping, say, the more senior leaders in the organization see that, you know, ultimate goal, of course, is more sales, more revenue, higher profitability. But the journey to that will mean that, hey, if I move these three metrics, that will eventually

result in those other ROI that you're trying to see. How did you make that connection? There were probably kind of a few steps here, right? So the first was pulling together the research and presenting it in a way that was very easy for people to consume and understand. Right. And so we did the State of DevOps report. It was laid out, and later the Accelerates Data DevOps report, it was laid out in a way that was very easy to understand. It was very easy to just steal a page and drop it into a slide deck somewhere. We tried to very clearly articulate these capabilities and practices, things like, you know, technological capabilities, cloud capabilities, cultural and process capabilities, they contribute to our ability to deliver software with speed

and stability. And once we have that speed and stability, that is how we can help kind of unlock some of this revenue or customer engagement and success or growth. And so really the first step, and I was still kind of coming at this as an academic, this was a side project or I was, working toward tenure. So it was research papers. So it's like, how can I make this as accessible as possible? So we did the report that way. I would go to industry conferences and I would read it out. I would continue working with a handful of teams I was working with. And when we tie, it to outcomes for organizations, this is where we could kind of work backwards with them. We'd say, okay, well, what are your sales goals and what revenue do you have associated with those sales

goals? Okay, for those sales goals, what software do you need to make that happen? Okay, now for the software that you need to make that happen, it needs to be happening under a time frame, right, relatively quickly, it needs to be stable. Okay, behind that, then the question is, well, if I need to improve, what should I start with? And so that's where folks could take a look at the questions. They could take a handful of questions and they could say, okay, well, there's 20, 30 things that contribute here. Is there anything that stands out at me as being a real sticking point? Like, I know automated tests are a challenge or I know this process that we have is a challenge. And so we tried to kind of help folks self-diagnose. After a few years, we started Dora,

the company, where we still did the research that was publicly available. So I think, you know, one thing that was also important to me is that a lot of this work was just free. Anyone could get access to the report. Anyone could read it. Anyone could self-diagnose. Anyone could get it. Now, if you wanted to engage with us, right? If you wanted to pay us, then you could have your own personal assessment. And algorithmically, we could help identify for you what your biggest constraints likely were. So then you could have kind of a slightly more focused or algorithmically backed approach. Now, other folks do things like value stream mapping, right? There are a few different ways to kind of back into this, but that was the one that we did.

ACM Bytecast is available on Apple Podcasts, Google Podcasts, Podbean, Spotify, Stitcher, and Tunein. If you're enjoying this episode, please subscribe and leave us a review on your favorite platform. What were the developers' reaction to this? Taking people along for the ride is also an important thing. I mean, one thing, obviously the survey that you defined, the surveys that you defined were probably very relevant and probably asked the right questions that kept people engaged because otherwise we know how hard it is to get people to respond to surveys. Yeah.

But what was your general reaction that you got were mostly where developers excited because they knew that the outcomes would be something that would directly improve their quality of life? Yeah, I will say it was largely positive, right? So at this point, we'd been hearing stories for, you know, a handful of years about how this was important or impactful or, you know, this new way of developing software would be better. But then they never had evidence. They never had proof. And so this gave them kind of a research backed.

Here's some evidence that doing a handful of things, you know, improving the way we make software improves outcomes. And we helped in this report, we tried to help give them the language around that, the business language. And also, I think, was validating because it helped highlight or share that some, of the challenges that they had were also shared by others. And when others had improved kind of these sticking points, they had seen impacts in the way they did their own work. I will say, occasionally, still, I got folks pushing back and saying, well, how can you know this is right? It's based on surveys. It's not based on system data. And for that, I would say there are a couple

different answers to that. One is it can be really prohibitively expensive and it could take a ton of time to fully instrument a system. But for the places, where that I've worked with that have a lot of instrumentation, I will say for the most part, actually I would say every time I am aware of when the system data contradicted the survey data, if the surveys were written appropriately, the surveys were always correct. The system telemetry, they had missed something. They were missing an aspect or a tool that they weren't aware of or the tool had been reconfigured and the instrumentation and the logs didn't reflect that. And so that's one answer, right? I think is that surveys can still give us a lot of really useful information and a

nice mirror and reflection back into the systems, right? Developers are working with these systems quite often. So, for example, if the system telemetry tells us that a release takes a week, then there are a couple options, right? So either maybe the developers tell us it takes about a week, cool, or maybe developers tell us that it takes three weeks. So that could be, that's probably something missing in the telemetry. Or they could tell us that it takes, you know, four or five days, and it's unbelievably hard. It's full of manual labor. It's full of toil. It's full of risky handoffs. We heard that quite a bit, right? So it can still give really good insights. I think the other part of the answer is when we ask questions, I made it very, very easy to answer correctly. So that's in

the ordering of the questions. That's in the phrasing of the questions. It's also in the scaling of the answers. So for some of the questions, it was, what is your lead time for changes, right? How long does it take to go from code committed to co-running and production? You know, things like that. And the options there were often something like on demand, right? Anytime the business needs, we can do it, or a week, or a month, or six months. When the responses are kind of phrased that way, you don't have to be precise, right? I can hunch it. I can say, oh, it takes about a week. Or it definitely takes six months, right? We only release software every six months. At that point, it doesn't matter if I'm off by a few hours or even a few days, if I'm comparing something

that's a week or two with six months. And so it really helped people at least get started and pointed in a direction because it's just so discouraging to realize you don't have the data that you need to make a decision and feel like you have to instrument all the data first. Because first, one, that can take years. And two, you might not be. instrumenting the right thing. So it really kind of helped unblock. Right. And it's fascinating also to uncover these things exactly what you said, which is sometimes the data gives you, I mean, it's accurate data, but it's just not complete. And I think the second part also is in many ways, you know, the time to actually see some early wins is also sort of very, you know, reaffirming in that,

yes, we're moving in the right direction. So what was some of the metrics that you kind of highlighted at that time, Nicole, and do those metrics? Is there a framework that you define that still holds good today? Because obviously the world is changing rapidly today. So I will say there were a handful of things that we pointed out, right? There were things like of technical capabilities, which ones improve, right? And so we found that there were things like continuous delivery, have a flexible infrastructure, have good database change management, have deployment automation,

There were things around process, things like streamlining your change approval, having clear and predictable change management processes, working in small batches, having whip limits. We had cultural pieces, right? Do you have good psychological safety? I used a model developed by Ron Westram. Do you have a climate from learning, right? Because when you're moving quickly like this, the thing that's most important is that you feel free to take smart risks and you learn really quickly from them when you move on, right?

you don't just double down or you don't ever try anything new. So those were the types of things that we studied and investigated. And I would say, in many cases, they hold true. And I think this is where we kind of come to some of my next work around space, how we measure productivity and frictionless, how we kind of think about developing systems and kind of evaluating systems now is, for example, one that will stay relatively true is streamlining change approval. That has been true for the last decade or so, that is true today, and I suspect that will remain true. Now, there will be differences in how much of these processes are streamlined and executed by humans and how many are streamlined and executed by tech, and including AI agents. But thematically,

it stays quite the same. When we think about removing friction and using friction as a lens to improve our system, there are certain types of friction that might reduce. You know, we might start seeing people orchestrating agents and running what we now think of as the outer loop, but realizing that friction, whether it impacts humans directly or it impacts agents because of the complexity of our systems or the complexity of the handoffs, that is still a really good lens to identify opportunities for improvement. Understood. So the space framework itself, you know, I heard a little bit about that stands for satisfaction, performance,

activity, communication, and efficiency, I think, I wrote that down. So if you think about it and you had to sort of reimagine it in today's world, which, of course, with AI, for an AI native world, if you will, and the ability, I mean, like for agents, for example, you have to rely on systems data, right?

You're not getting a survey result back. How would you rethink this in an AI native world? What would you add or what would you remove? So this is such a great question. I'm getting it quite a bit, right? So I think I would say, in general, space still holds true. So when we use the space framework, this stems from the question that people used to come to me with quite often, which was, okay, well, now I know I need to measure or understand or improve this thing, right? Whether that's the N10 process or peer review or something, right? They would say, how do I pick my data, what metrics should I use? And my answer was always, first, it depends, and it depends on your system and the data that you have available and all those things. And then you should always select metrics that represent different dimensions and aspects of what you're looking for, and they should be intention. That's where we get space. Space says satisfaction, right? We should

ask people, we should know if they're satisfied with the system or the tool or the process that we're talking about. Performance is an outcome, right? So what does that? outcome of the process? You know, is it like failed tests? Is it a pass rate? Is it regressions? Activities account? Most of us are very familiar with counts, right? So this is going to be number of lines of code, which is bad, or number of pull requests, which is only slightly better, or number of something. Communication and collaboration is, how can we talk to people? Or how do we need to talk to people, right? Do we have to coordinate through dozens of teams to get a change done or can our systems talk to each other? Do we have good API usage and definitions?

And E is efficiency and flow. And so it's how long does it take to kind of get things through a system? What's the time component there? And so as long as we pick a metric from at least three or four of these dimensions, that will give us a much better look at what's happening. And it'll kind of like help build guardrails into the metric. So as we move into a, AI now, I think, you know, we're going to hit a point where, again, what are the right metrics? It depends, right? It will depend on what it is we're trying to measure. It will depend on what it looks like, right? Agentic workflows are probably going to look very different in six months than they do today, but we can still ask very similar questions. Is the developer satisfied

with how they spin up or understand or orchestrate an agent or something like that? You know, performance. Is the agent well-performance? How often is it right? How often is it wrong? Activities, right? This is where we could also pull in something like number of CLs. And are they human generated or are they AI generated? Communication, right? It's easy to understand and communicate, efficiency and flow. How long does it take? So we can still stick to space. Occasionally, I think there are some opportunities to add here. Trust strikes me as one, which is interesting because when I was working with this admins, trust was top of their list, right? They really needed to be able to understand and verify and trust. what their systems were doing while they were doing it. Now, many time developers don't need that, or we didn't need it as explicitly, right? It's like if we compile our code and it compiles, then it's compiled and it's fine, right? But now we're seeing that as we're doing much more with agents that are non-deterministic, now we're taking time to, like, is this the correct information, has it hallucinated?

So their opportunities kind of add to space, but I don't think it's taken space away. Understood, yeah. No, that makes a lot of sense, the way you've described it. And I mean, I want to talk a little bit about this concept of trust, right? I mean, we're in a world where there's so much AI generated code as developers, you know, you want to encourage developers to leverage these tools to help expedite maybe tasks that we're doing or to basically, you know, take away the mundane repetitive tasks and so that we actually can use our time for higher order projects.

But how do you work with developers to say, you know, there is. a certain amount of trust, you also have to be, like, to your point about guardrails, like, how do you prevent the overtrusting of AI? And what does that mean in terms of, you know, developer satisfaction as well? It's like, you're spending more time reviewing code than writing it. Do you think that that has an impact on that number? I do. I think it does. There were a lot of questions in there. I think, you know, satisfaction is going to be important. That last bit that you mentioned, I think, is a really interesting open question. And that is, in many cases, developers are now reviewing code more than they write it. Even if by reviewing we mean I have

generated some prompts and AI has handed me back a significant amount of code or like a very strong scaffolding. And so now I need to read and review my own in ways that were quite different from before. There's a satisfaction component that I think is relevant. But I think there's also a really important question for us as a field to say or to redefine what does it mean to be a developer and what is valuable and what is worthwhile. But I think that's one part, right? Because for many folks, many times I've heard people say, well, if I'm not writing the code,

then I'm not doing development. I'm not engineering. I'm like, well, architecture isn't writing code. There are a lot of things that aren't writing code that are valuable, but I don't think it's necessarily really thought of as doing good prompting and reviewing can be generating value. It just looks different. And I think another question that is still,

open is when we shift the work from primarily like a lot of generation and then reviewing later to super fast generation that's generated somewhere else and then a lot of reviewing, how does that impact our ability to understand the work in front of us? Like how does it affect cognitive load and our ability to focus? How does it affect our mental model? Right? I remember when I was building systems, I got a really good fill for the system by doing a lot of development. How can we support building up that mental model in this new paradigm. And I'm sorry, I didn't answer your question.

I gave you more questions. No, it's good. I mean, I like the open end and back and forth because this is all, I mean, new to most of us, right? And it's wonderful to kind of hear your perspectives on it. You know, one of the things that we often talk about is also, when you talk about developer satisfaction, it is, oh, I hear that a lot,

right, uninterrupted time so that I can focus, I can be in the state of flow. And I'm just wondering when, you know, with using AI tools, we're constantly prompting, reviewing, rewriting. It's a different way of sort of, you know, interacting or being in that flow state. Have you heard more about what constituted deep work earlier, has that changed? And are developers kind of adopting to this and still feeling that sense of accomplishment? So there are some folks that I'm aware of that are looking into this.

I'm sure there are more. It does look like, and I haven't done this work myself, I'm just aware of some of it. It does look like that flow looks and feels different because it's very different from basically being uninterrupted and writing, writing code, writing papers, right, like whatever just you want to write, to turning it into an interaction instead, where there's some input, some response, some input, some response. Now, in terms of folks, are they enjoying it, are they liking it? That, it really depends. I know some engineers who love it and they think like this is just, it's a superpower.

They can do things in a matter of weeks that would have taken them months before and they're incredibly fulfilled and they're super excited about it. And I know other developers who are real upset and understandably so because the thing that they love and the thing that they've done, the thing that they have really fine-tuned

into being something they're incredibly good at, has now changed a lot. You know, without warning, right? Someone just came in and said, this is the way things kind of are now. And so for them, it's, you know, I think this is similar to the change, frankly, that we saw with mainframes. There were a lot of folks that did incredible work and they had expertise that no one else had and suddenly cloud comes along. Like something new is here. And it's like, it's just a different way to look at building systems.

And the expertise that you have is still important and relevant. And in some ways, I think it's going to serve folks better if they've coded what is now maybe the old-fashioned way. because we have a really good feel for the systems. But it also means we have a huge opportunity to really kind of embrace and pioneer what these new methods of development are going to be. Yeah. And the premise is that the ability to develop software faster, more accurately with, you know, of course,

human in the loop and reviews, etc., is going to mean all of those things that we spoke about earlier, right, time to market, you know, the advantages that you have to be able to generate more revenue you, et cetera. Do you think that's a real change that's already happening? What are you hearing? Absolutely. Yeah. Absolutely. Because if we look at, I mean, if we just kind of look back over the last few years, we have seen significant model changes in six months or less quite often, right? And just looking at that, historically, we would not have seen this rapid rate of change this quickly. And so for companies,

who want to build new products, capture new market share, understand their customers better, they also need to move really, really quickly. And I think this is where we have really an incredible opportunity with AI with agenetic workflows, because when we can develop much faster, I think here's one thing that we should also consider and keep in mind here is developing super, super fast doesn't mean everything hits a customer. What it does mean, no, is that we can prototype really quickly. We can run through so many ideas in an afternoon that almost anyone can do now. Before we ever write our PRD, right, we can prototype. And then we can turn that prototype into just a slightly better prototype enough to maybe get it in front of a handful of users.

Even if we don't do a full A-B test, we can now show something to someone and see how it lands. And then we can decide, let's say we run 10 quick experiments, which one of those is the best. instead of doing 10 full features or parts of a product and then investing all of that resource, money, effort, compute, and then taking a guess, which almost all of our products are a guess, right? So it's like, how many shots on gold can we get so that we can improve our ability to guess? And like how quickly and cheaply can we do that, right? And by cheaply, I mean, like, for human investment too. Yeah, no, absolutely. I mean, I think that early customer validation will actually end up being great signals for us to be more efficient in kind of what we put out. Like you said,

not everything will hit the customer. I think that is a very, very insightful comment that you made. As we sort of come to a close, Nicole, one thing I would want to know is for organizations that, because this obviously, as we think about like, hey, look at my systems, what kind of things should I be measuring? How do I design my surveys? If we don't have access to you, what can we do? Like, what are our options? There are actually a lot of really good. resources. I will say the AI models are pretty good partners on this if we interact with them the right way. So here we can say things like, this is the question I have. I had these survey questions in mind or can you suggest some survey questions and then turn that into a conversation.

Because I will say as a researcher, right? So when my product training hat comes on, I think one way, when my research hat comes on, then I'll really look at a survey question and I'll say something like, is this a leading question, right? Like, have I suggested the answer that I want? Have I actually asked for two or three things in one question by mistake? Am I using a term of art that someone won't know? Or am I using really clear language? And most of the time, when I partner with folks at other companies and I take a look at their survey questions, like if I'm just doing a favor, we just jump on a call, those are usually the biggest opportunities for me to nudge or kind of help them edit and update their questions so that they can get better insights out of it. Well,

Now, honestly, we can ask Gemini, we can ask Cloud, we can ask ChatGPT, right? Here are the, of the questions you generated or of the questions I gave you, are these good questions? What else should I consider? Are there any gaps? Is there opportunity to misunderstand? Have I asked what things should I look for? And even if you don't know what questions to ask, they're quite good at giving you feedback and iterating there.

Amazing. Thank you so much. That was a very, very good advice. And more importantly, thank you for your heart-breaking work and taking the time to speak to us at ACM Bytecast. Oh, well, thank you so much. It was a pleasure. ACM Bytecast is a production of the Association for Computing Machineries Practitioners Board. To learn more about ACM and its activities, visit acm.org. For more information about this and other episodes, please visit our website at learning.acm.org.

or G slash B-Y-T-E-C-A-S-T. That's learning.acm.org slash bite cost. This podcast was edited by resonate recordings.