PurePerformance - Back to Basics: Increase DevEx in the Age of AI with Laura Tacho

Episode Date: November 10, 2025

Don't get stuck using AI to build faster horses. Instead, find the opportunities and rethink your software delivery processes! That, and only that, will help you increase Developer Experience and Efficiency!This episode is all about how to measure and improve DevEx in the age of Artificial Intelligence. And with Laura Tacho, CTO at DX, we think we found a perfect guest!Laura has been working in the dev tooling space for the past 15 years. In her current role at DX she is working on the evolution of DORA and SPACE into DX Core 4 and the DXI Measurement Framework.In our episode we learn about those frameworks but also how tech leaders need to rethink where and how to apply AI to improve overall efficiency, quality and effectiveness! The key takeaways from this conversation areDevEx is all about the identifying and reducing friction in the end-2-end development processTech Leaders need to become better in articulating technical change requirements to businessAs of today only 22% of code in git is really AI generated. Don't get fooled into believing AI is already betterBack to Basics makes companies successful with AI. That is: proper CI/CD, testing, documentation, observability!Here the links we discussedLaura's LinkedIn: https://www.linkedin.com/in/lauratacho/DX: https://getdx.com/Cloud Native Days Austria Talk: https://www.youtube.com/watch?v=kZ1F0-XS1l4Engineering Leadership Community: https://www.engineeringleaders.io/

It's time for Pure Performance. Get your stopwatches ready. It's time for Pure Performance with Andy Grabner and Brian Wilson. Hello everybody and welcome to another episode of Pure Performance. my name is Brian Wilson and as always I have with me my energetic co-host Andy Grabner how are you doing today Andy pretty good pretty good actually we had a little bit of sunshine today here in Linz Austria even though most of the day it's been gray like the last last week I would say yeah yeah sun is always good for energy if you say energetic maybe that's the reason oh there you

go see I'm getting my energy because I just had my double espresso and we have cold and gray skies out yesterday and today. So it's been a little bit of a downer. But we're heading in. So people listening, I'd say, you know, it's going to be quite an experience today because if you listen to this episode and the next one, we're actually recording these back-to-back, which I don't think we've ever done before. We've done like day after day. We've never done back-to-back episodes recording. So it'll be interesting to see how much steam we have second time or if we're just so stir-crazy that it just becomes wild. We have no idea.

You'll have to tune into the next episode to see what kind of experience you have. See, Andy, I try to do a poor man's, Andy. Just a reminder, Andy is the king of transitions. So, or segways, if you will, right? He's always great at taking our dumb banter

here and leading it directly into the topic. And I tried that. The thing is, if we keep talking like this, we probably just run over and we won't have a chance to actually get our guests on the line. So that's why I think let's make sure she has a great experience as a first time guest on this show. Let's do this. Laura Taco, thank you so much for being here and being our guest at the Pew Performance Podcast.

You are the CTO at the DX, the developer intelligence platform. I had the pleasure to welcome you on stage at Cloud Native Days, Austria. It was just like three weeks ago. And when I listened to your talk, I thought I would like to have. have Laura on the podcast because you've been talking about a very important topic that I think we have covered a little bit over the last couple of episodes, but hearing it from you, it's just great. To get started, Laura, maybe a quick introduction, who you are, what you do at DX, besides obviously being the CTO, what I just said, but what is driving you, what is

motivating you, and then I have a couple of questions for you. Yeah, my pleasure to be here with both of you. I'm Laura Taco, CTO at DX. I would say, actually, I have been working in developer tools for the last 15 years. And so my role at DX, we can kind of see as like a capstone. If you're listening to this and you've been involved in Docker and cloud native for a long time, you probably know me already because I've been working with Docker, started working with Docker in the very, very early days. I've always been kind of obsessed with developer tooling, cloud tooling, and trying to find ways to make the day-to-day work of developers just easier so that we can focus more on what's important.

Of course, that varies company to company, individual to individual. At DX, what we do is we are an intelligence platform. So we're taking data from your systems. We're taking data also from developers themselves through self-reported surveys, marrying that together, and then giving organizations basically a map to understand where the friction is, where the pain is, so that they can improve the experience for developers, remove the friction, and get to better outcomes. I lead a lot of research at DX.

We're very research-backed. So we work with a team of researchers. We also have a tremendous amount of data, as you can imagine from all of our customers that we can aggregate and look for patterns and trends, which is a lot of what informed my talk that I gave back Cloud Native Austria because AI is, of course, a big hot topic. and there's lots of claims in the headlines and then having a front row seat

to how, you know, 450 companies are actually doing it on the ground. There's a little bit of a discrepancy, and I think we'll talk a little bit about that today as well. But kind of the bottom line is I'm here to make developers lives easier, take friction out, make sure that leaders are equipped with the right data and the right storytelling around

why it's important to invest in things like CISD, performance monitoring, all of the tools that make life back. better and lead to the right business outcomes. Cool. And an interesting fact is, if I remember correctly, you're originally from Wisconsin, but now you live in Austria? I am.

Yeah, we're like Austria two-on-one here. Yeah. To Austria, two, to one in America. But, yeah, I was born in Wisconsin, strangely, or maybe not strangely, but it's very common to have German ancestry. So my great-grandparents and great-great were born in Germany. They came over to Wisconsin. Then I went, turned around right back to move to Germany about 10 years ago, and I live in Austria.

Well, I hope you enjoy your new home. We definitely enjoy it. As I said, your keynote at Cloud Native Day's Austria. Laura, in preparation for this podcast, I went through your talk again. And folks, by the way, if you're listening, we will also add links to the recording of the talk because Cloud Native Austria already published all of the videos. really check out the talk and Laura I also know you've done obviously this is not your first podcast I just listened to a podcast that you recorded in September and also took a lot of notes so we will add as many reference links that we discuss today to the podcast summary one of the

things that I want to start with when I give presentations around platform engineering which I do quite frequently I typically start with two metrics that I took from the state of DevOps report, which I think two years ago, a year ago now, the Platform Engineering Edition and they talked about that only 40% of engineering time is spent in productive activities and that I think 36% of developers are leaving an organization because of bad developer experience. And I always bring this up as two numbers, which is for me, first of all, the productivity is already shocking enough, but also as a leader, in the leadership role, where I want to retain the talent that I have in my organization

and I see 36% of engineers are they want to leave because of bad developer experience. This is very alarming, yeah? And I wanted to ask you because in the podcast and also I think in your presentation that I've opened here, you made a very interesting and nice statement. When we talk about developer experience, developer experience is not just ping pong tables and beer taps.

So I would like to get, maybe a definition from you to get started. What is developer experience? What are the things that organizations need to look for to know whether they're doing a good or not a good job? How can they maybe measure it? I know you have a framework. But let's start with what is developer experience from your perspective

and what are the things that are indicators that things need to change. Yeah. I think the question of like what is developer experience is an interesting one, because sort of at its heart is, you know, how do developers think and feel and perceive the work that they do day to day? What is their experience doing work in day today? I think the tricky thing about developer experience is that oftentimes we're not searching for the existence of something. We're looking for evidence of absence, which is really

difficult. Developer experience is the absence of friction in the system. And if we think about, I mean, we can like, let's think very kind of big picture and tie this. You know, you have green tests that verifies that the cases that you thought to test about were, you know, they're green. It doesn't tell you that your software works. It's very, very difficult to prove the absence of something. It's very difficult to prove that the quality isn't there. It's easier to prove that, you know, tests are passing or we have an incident or something. The measurement around this, therefore is just really difficult. We want to prove the absence of something.

So we can look for ways to detect then where is the friction in the system. And so when I'm talking about developer experience, I talk about, you know, a developer getting work done, whatever part of the SDLC they're touching, how much of their time is being wasted due to inefficient processes, lack of tooling, how much time do they spend looking for information they can't seem to find and that they need that? How much cognitive load do they have? Are they getting feedback quickly enough? Do they have enough time to focus on things?

So it's a kind of a mix of all of those things together. I think very importantly, two things I would say developer experience, it's the absence of friction in the system, and it must be measured with multiple dimensions. And there is no one metric that can tell you if you have developer experience or not. I really like the, thank you for that definition. I think this will become maybe our,

our snippet on social media to promote this session well it's hard to say because we still have at least half an hour to go so maybe it got a half hour still exactly but you came out swinging strong

so yeah yeah so I do know that you know if looking at your website and looking what DX is doing you've obviously

you've defined your own framework of metrics and as you just say there's more than one metric multiple dimensions Many of our listeners are probably familiar with the Dora metrics because they've been around for a while. They might be familiar with the space framework

because we've also had sessions about it. What does DX now do? What do you define? What's the next evolution of these frameworks? It's a great way to put it as the next evolution because we work with the Dora team. We work with the researchers that did space. DevX framework was the next evolution after space,

also with Nicole Forsgrin and Dr. Margaret N. Story, who worked on space and worked on Dora. If you've read Accelerate, those names might kind of stick out to you. The DECS framework is all about cognitive load, feedback loops, and flow state, so having enough time to get into the workflow. Those three things together, Dora metrics, which talk about software delivery capabilities. Then we have the space framework, which isn't a set of metrics. It's not a list of metrics, but it's a framework for how to think about developer productivity. all the way from satisfaction to performance, activity, communication, and then efficiency.

Then we have DebX, which is about cognitive low feedback loops and focus. Those things together, it's a lot of ground to cover. And organizations, even though we had all this research from over a decade, organizations were like, what? Okay, but what do I put on my dashboard? You know, what's the number or what's the metric that I should bring to my board or my exec team when they want to talk about how is my organization performing. So we worked with all of the researchers from DORA space and devX. We put together that body of research and we made

a recommendation, which is the DX core four. This framework takes all of the research from the last decade that we have on developer productivity, but turns it into an actionable list, short list of metrics that's easy to deploy and then also benchmarkable. So if you're trying to get started with metrics, there's the DX core four framework, is absolutely the thing I would recommend. I co-authored it. So, of course, I'm going to say that, but I really believe it's the right thing. I co-authored it. It's just, it's there to help organizations get started faster. Cool. Can you enlighten us quickly? What are these four core metrics? Yes. So we are talking about speed, which we're measuring in terms of merge frequency, PR, throughput.

aggregated, never at the individual level, but at an organization team level. Developer experience, and we're measuring that with, you know, when you unpack that, there's like quite a few other dimensions to measure it. Quality, which is our good friend, change failure rate coming from Dora, and then impacts, which is the innovation ratio, how much time developers are actually spent doing new work, innovative work versus maintenance, quality issues, all those other things. So looking across speed, effectiveness, quality, and business impact. This gives a common language to speak about developer productivity

and engineering work performance across the organization. But then also we have just those, you know, those four numbers they need to be looked at together. And they tell a story about quality, innovation, developer experience. And we see that organizations that focus on that developer experience piece, they see the other three numbers rise because developer experience is the core of everything. And that is the place to focus. that's the lever to pull on for organizations that want to improve. That's cool. I was just going to say one thing I noticed,

and I guess this is where the humanity meets the numbers, right, is those are all metrics that are going to be numerically based. There's going to be quantitative, yes, quantitative metrics to see what these numbers are. And I suppose the idea then is when you see, efficiency down or any of these metrics being down, the idea then is not just like,

all right, let me go yell at developers to get more efficient or something. It's let's look at why. That's when maybe the anecdotes come in. That's when you're talking about discussing with the developer. So it's not just a matter of, you know, if you think of a production line, right?

We didn't produce enough cans this time. It's not about let's make people work faster and it's that's fine where the inefficiencies are maybe those inefficiencies or the developers aren't happy with something right whatever it is but that's where like the human then steps in to investigate what's creating the friction there to help figure out a way to remediate that yeah you've you've got that absolutely right dx core four is the only kind of developer productivity framework that has the developer experience as a core pillar those are self-reported qualitative metrics coming from engineers, we have to look at that. We have to look at that part of

the story because what I see so often is, let's say, for example, change failure rate, we see that rate go up, meaning there's more escape defects, quality is going down. We can look in the developer experience measurements and see that developers are already, they've been saying, I don't feel confident when I release changes. I think it's going to break production or, you know, our build and test process has way too many flaky tests, just not reliable. Those are leading indicators for these lagging indicators that are, you know, that the business tends to pay a little bit more attention to. And so, you know, when we want to focus on improvement, it's going back to the, what's the leading indicator? Where's the bottleneck? Where's the friction? Can we remediate that problem

and then see the results downstream, just like in your can factory example? Do you provide, I guess people can obviously go to your website and go through all of the material, but do you provide guidance on how? how people can start measuring these things themselves? Is there anything where people can get started? Does it always need a professional service engagement or tooling? Or can you quickly give us an update on that? Yeah, there's a couple articles, and we'll link them in the show notes.

But all of the research that we do at DX is open source, for lack of a better word. We're not taking contributions because it's like a team of researchers that we have our methodology. But it's open for anyone to use, and you can do it by yourself. that was very important to us, especially with the core four, that any team with access to Google forums or Microsoft forms or whatever can collect all of these metrics self-reported without needing very sophisticated ETL pipelines, and they can get a baseline very quickly, compare it to benchmarks, benchmark against themselves, and then start going. So I have a template for a survey to use to collect all of the core four metrics, including the breakdown of

the developer experience drivers. And that's just in a Google sheet that we can link here. And then I did a pretty long form article on exactly how to use it, when to use it, if you want to collect metadata, all these really descriptive things in Lenny's newsletter, which I'll also send you. We can link. It is behind a paywall if you're not a subscriber to Lenny's newsletter, but it doesn't start until like 80% down the article.

So you still can get pretty much a lot of it without having to be a subscriber, although it's a great newsletter. Yeah. Well, folks, definitely give you email, subscribe to the newsletter and get continuous updates on that topic. Laura, there's obviously a lot of information already about this topic out there. And as we said in the beginning, we want to make sure that we give people, our listeners, a quick overview. But then I also want to dive into a topic and kind of putting the AI elephant in the room. the AI is a big topic and also your talk at the cloud native days it was developer experience in the era of AI and for me the biggest aha moment and I think I also posted this on

LinkedIn later on you had one slide up and I have it here now where you said you did another it did research right and when we think about AI for developers at least the first thing that comes to mind is cogeneration right makes a lot of sense But then the question is, if code generation can make the developer 20, 30, 40 percent more efficient in writing code, what does this mean for the whole developer experience and developer efficiency in productivity? Because you then had a slide up that says, where is most of the development time actually spent or where the developer spent most of the time? And there's only very little in actually coding. There's so many other tasks that are, and excuse my language, time suckers, that are taking away the time, that are making developers improductive.

And I thought this was an aha moment for me. We're always talking about using the latest model to generate more and better code, yet we would only impact 10, maybe 15 percent of the developer time because that's how much time on average they're spending in coding, but they're spending much more time somewhere else. So my question to you now, where do organizations, where do developers waste their time? And what do organizations need to look at? And how do you measure this? What I can say, let's start with a little bit of data about how much time savings developers in general are getting back from AI,

which is only about three hours, 45 minutes, a little bit less, actually. It's strangely gone down, not significantly, but it's gone down a little bit since last quarter. So we're looking at like three to four hours per week. of time savings. So this is not 50%, this is not a huge amount of time. This is maybe 10% of time. Because, Andes, as you said,

we are, as a developer, not spending 100% of our time coding. And so when we look at the surface area of optimization, if we're only using AI coding assistance for code generation, the surface area is already only 25% of the time

that developers are actually coding. And then when we think about how much time within that coding time is appropriate for AI to augment, we're down to even a smaller chunk. And so what happens is that we might get some local optimization in this task acceleration, but those gains are lost in the system because code review is still a bottleneck, information seeking is still a bottleneck. And in fact, we have had AI tools now in organizations for about three years. If I look back to data three years ago,

the two biggest points of friction and two biggest opportunities for time savings were reducing the number of meetings and reducing the number of interruptions. Those two things have remained steady. We're three years into this AI transformation. We just pulled the data actually

for our next Q4 AI report. Sure enough, those two things are still, they are still the highest points of friction for developer teams overall, you know, for a very large sample size. This is due to, you know, it's information seeking. I can't get the information that I need. I'm waiting on a decision from someone. My feedback loops are too long.

Documentation is poor. I mean, there's a million reasons. There's a million reasons where that coordination cost is just high. I'm not suggesting that every organization should stop having meetings. But what if we used AI to solve those problems instead of code generation, when organizations start to do that, that's when they're seeing much more rapid acceleration toward whatever goal they've set with AI. Yeah, so maybe use AI to make the meetings more efficient, right? Maybe using AI to figure out who should really be in that particular meeting and don't invite many people because you think they can contribute. maybe use AI to write better, you know, agendas for meetings and like prepare for meetings.

So the meetings just keep a better outcome. I would even take it five, you know, five steps forward and think about why do we have so many meetings anyway? Why do we have so much? There's so much back and forth. Can we structure our teams and work in a different way now that we have AI? And perhaps there was a lot of handoff between design and PM and engineering so that we could design things in back. validate them in the early stages and then move into execution. But what if that doesn't have to be the case anymore

because your designer and your PM with Claude and some very good quality checks and things can make some prototypes and validate things easier. And now instead of a project needing that four weeks of planning and back and forth in validation, we can get that same thing done in a week. I've seen companies do that before. Takes a lot of effort, but that's quite a, you know,

it's not a, I think what organizations are getting trapped in right now is, you know, faster horses. If you know that's saying about, you know, if we would have good people what they wanted, they would have asked for faster horses when we really need to reimagine altogether. And maybe that's an automobile, maybe it's a rocket ship. And the companies that are just rethinking things, the whole value stream, are seeing a lot of acceleration because the system is the problem. It's not the individual task. That's the problem. Because the typing speed has never been the bottleneck in development. It's always something else.

Yeah, I would take it one step further and say, just send all AI chatbots to the meetings and then see if there's any difference. You know, you hear horror stories. We can laugh about it, but like people do that. People do that. Or it's like, you know, you make a chatbot to apply for a job and then it gets trapped in a conversation with the chatbot of the recruiting tool of the company. and it's just like, can we just, this is a waste of time,

can we just have two humans talk to each other? Yeah. Well, thank you so much. I really like the, I think it was Henry Ford, who has been attributed to that quote with the faster horses versus a car, right? I think we need to rethink engineering in the era of AI, maybe as AI native companies,

it means we need to rethink on how we engineer and what the process is to engineer a product from design phase to first implementation, to test validation. So there's a lot of opportunity to change the way we build software. And with the goal, obviously, to reduce these tests that are currently consuming a lot of time because we're doing back and forth and interruptions and all that. There was another interesting metric. There was, I think it was number four, the CI wait time.

So developers waiting for feedback from their continuous integration. So does my code actually build, do my tests, all run? It strikes me because we've been talking about how important it is to optimize CI, to provide faster feedback loops to engineers. And yet it still seems, based on your research, that this is still, it's in the top four. How is it still possible? I think one thing that a lot of engineering leaders lack the skill of is building business case because we it's like historically not really been a skill that we've had to exercise a lot.

You know, I even think back to my time as hands-on keyboard engineer and then my early days in leadership. It was sort of like, well, we need this because we say it's important and trust us. And it was always sort of like a, you don't know what we're doing because you don't understand code. And so you have to trust us on this one. And this is the thing that we need, which sometimes it worked, but sometimes it didn't. And organizations, you know, the budget holders sometimes don't, they need a little bit more to go off of. And I think it's been difficult as we have moved toward more accountability for how engineering is allocated in terms of like total spend. But also as we move toward, you know, if we have the option to optimize our CICD pipeline versus build a new customer facing thing, how can I, as an engineering leader, articulate the business value of this investment in plumbing against a concrete new feature that we know has X revenue potential?

And it's a lot easier to draw it. It's not apples to apples, you know, but we can quantify the impact. We can think about how many developers is this impacting. How much weight time is being lost? We can talk about it in terms of salary recovery. We can talk about it in terms of capacity creation. We can link it back to research like the DevX framework that shows us that feedback loops is one of the three major components that are research-backed tied to better organizational outcomes. There's a lot of things that we can do.

But because engineering leaders, we haven't been trained in it, it's really a new skill for a lot of engineering leaders to practice to try to talk. about investments in very kind of peer engineering things in that way. And so a lot of the time it's just been neglected or passed over because, oh, we're going to go chase the new customer facing thing. One of the things that I care a lot about is giving engineering leaders the phrases and the words and the mental models to articulate the value of why things like CI wait time are so harmful to your system and actually hurt organizational performance. and how we can change the conversation to not being engineering versus product,

but actually thinking about improving the CICD pipeline is a product investment because it's going to enable us to capitalize on all future customer-facing features much faster, and they're going to be higher quality. And what's that worth to our business? Quite a lot. When I go to a senior leader asking for budget and I say, hey, I have a way that we can execute 10% faster on every single project going forward, and we're going to reduce maintenance costs and quality overhead by 25%.

Any engineering leader or budget holders can be like, tell me, we can we start yesterday? But when I go and say, hey, you know what? Our developers are complaining about CI wait time, I need three weeks to finish it, and we're going to have to delay this project. That doesn't really fly. And so we need to change the way that we talk about things in order to get the results that we want. Engineering is there to support the business. You know, I know earlier, if I think back to earlier in my career, I'm like, oh, this is cool.

I get to build cool things that I'm interested in all day. And I, you know, as I grew up and kind of went up in leadership, I realized that importance of tying what I'm doing back to how it benefits the business. Because at the end of the day, this should be enjoyable, but it's also not a club. It's not a coding club. Yeah. I just had a I think we had a similar conversation a couple of months ago

I was in Prague at the engineering leadership conference they have a great community so Laura Prague is just around the corner from you where you live right now I want to give a quick shout out to Marian Kaminista. He is actually

leading this community and I was invited to speak there about platform engineering and then we had him on a podcast and I think he made the same claim. He said, technical leaders need to understand how to speak the language of business in order to really have an impact right. And so definitely spot on and also what I hear. And maybe this would also be good for Laura for you.

If you want to make a trip to Prague, I'm pretty sure. There will be a great community for you also to chat with. And I assume you have been because I know we also had Tucson. on one of our podcasts a while ago, who was also a technical leader. Dusan Katona is his name. He works at Atta Kama in Prague, or in Bruno, actually, but I met him in Bragg. And he mentioned that is also implementing the DXQA44 metrics right now. Yeah.

Yeah, it's a great, lovely city. We had a team offsite there one year. That's very memorable. I only have good memories of Prague. I would like to, so we talked about, if I do a little bit of a recap, we talked about the metrics, the four-core metrics, speed devX,

quality and impact. We talked about that just using and thinking about AI as a code generator and helping developers to code is not really leveraging as its full potential because there's most time spent in other tasks. But you brought up a good point instead of just thinking, how can we use AI to reduce the amount of meetings we have for time?

Why not rethink the way we engineer, rethink the way we do things? And if you want to make these drastic changes, then tech leaders that need to drive these changes need to understand how to communicate the benefits to the business because it doesn't help you if you say the CI needs to be 10% faster. You know what's interesting with that, Andy? It triggered a thought in me

where we're trying to retrofit AI into the developer experience, right? But if you think about the transformations recently, a few years ago we were talking about cloud native. Recently, we've been talking about AI native. And I think instead of trying to back, you know, retrofit AI into the current process, it might be beneficial.

And Laura, I don't know if you've seen any of this where people have taken small teams and started experimenting to say, okay, we have all these different uses of AI. Let's imagine fresh what a new developer workflow would look like with these AI tools and try building it out

instead of just where can we plug AI into the existing system. Right? So developer native, if you will, maybe. Yeah. There's some orgs that have talked about what they're doing. So Travolopia at the Enterprise Tech Leadership Summit was a great presentation. And they talked about taking 10-person teams and turning them into four-person teams.

What they're doing is having designers literally sketch out things like wireframes on paper. They take a picture of it and then have that generated as a prototype in the browser, going from screenshot to browser. We can do that now. Once it's there, we can validate much. more quickly. We can put it in front of a customer and have conversations about it. We can think about load testing. We can think about all these things much earlier in the process. We can bring engineering in. And then once the design and the spec is really settled on, the engineer can

coordinate agents to do some of the work while also staying in the loop and making sure the quality checks pass. And they've had a tremendous success with this. And I've seen lots of other organizations have tremendous success with moving, just really fundamentally changing the team composition and the workflows. When are things handed off? Who's participating in which section? Which personas or job roles get access to the coding tools and which personas are now coding because you know, designers usually didn't, you have to have a GitHub account. Like, now they do. Now they might be doing prototypes. And just that prototyping and validation is a huge unlock because we can talk about, you know, saving incremental time three hours, four hours a week of speeding

up coding. But where most businesses are losing the most time is that they build the wrong thing. A whole team spends two weeks on something that they decide, oh, actually, that was the wrong strategy. We're building the wrong thing. We need to go do it. Or they build a feature that doesn't have any business impact. That's where the real inefficiencies are as well. And so I'm very excited about using AI to empower that kind of classical maybe product trio. I think that probably needs to change. But the designer, product manager, someone who's on the business side, an engineering lead to validate things and just keep those feedback loops way tighter and much closer to each other. And so that really that team owns everything from, you know, from police to thank you

throughout the whole process. It's exciting. We'll see what happens. Yeah. Because as you said, right, I think the, was it, who was it? I think it was Guaranka, who we had on in the very early days of our podcast. She worked with Facebook and she was giving us some insights on how they're doing feature testing and feature evaluation at Facebook, right? And they basically said, you know, we have an idea of a new feature. It doesn't have to be perfectly implemented, but get it out quickly, get feedback on how many people are actually adopting it ahead, then to specify a success criteria. So if not, within the first month, 20% of the Facebook user base is using that feature at

least once a week. If you don't meet that bar, it doesn't make sense to actually invest and finalize that feature and optimize it. And for me, what you're saying is we, AI now is able to revolutionize the way we would have built this because it would have needed many different people to build this feature and get it out there. But now with AI, you can quickly create a prototype, give it to your test customer base, get feedback, figure out, is this, you know, a feature that we should keep on up?

Because I think we know with statistics, at least I'm sure there's many reports around, that only maybe, you know, 20% or 30% of features actually really take off. And so how can we get faster to understand which 30% we need to keep and invest and the 70% we will need to throw away? And if AI helps us there, that's a huge improvement. Yeah. Yeah, that validation step, you know, most companies are of a certain size or more and plus one and bigger are using something like product board.

they have a ton of user feedback that they have in a centralized system. And what I've seen some companies do is take all of that user feedback. They have longitudinally going back in the past, however many years. And when they create a prototype, they use that data to create certain personas. And then they actually use AI to test their feature against these like AI generated personas and use AI to coordinate the validation or even user acceptance testing of the features before it even goes out to like an actual real human being to point out flaws or point out under-solved problems and things.

I think there's a huge, you know, there's a huge amount of potential for AI in this part of the SDLC. Also on the architecture side as well doing, you know, even like generating huge data sets or generating, it's, AI can be such a powerful tool in early validation. before we go chase it

and build the wrong thing for two weeks before realizing like, oh, we actually miss this thing we need to go back. If I recap the last couple of minutes, basically, it's hopefully an eye-opener for those that haven't seen this already. AI in developer experience

is not limited to just generating code and being an assistance to code generation. It really impacts the whole SDLC. and stuff that you just brought up is interesting to hear like these use cases on taking a screen, taking a photo from a sketch on a piece of paper converting that into a working prototype or using AI to even validate if that design actually makes sense, if there's no flaws in there. Then he mentioned data generation. One of the talks at Cloud Native Austria was also around chaos engineering with AI, where they also talked about using AI for a good

data set generation, right, for your tests, and then also for executing some, some chaos experiments. There's so many use cases out there. I want to now a little bit play devil's advocate in the last couple of minutes that we have. Please. Yeah. So now we have this brand new world where with only half of the people we create 50% more features because everything is optimized and automated and now

things run in production. How much of that code is really generated that runs in production generated by AI versus how much is really a human engineer still involved in creating new code? And

if something happens in production who is fixing it and who can fix it? Brilliant questions. So there's a lot of marketing claims out there from vendors. I understand why they do it, but right, we have, oh, up to 90% of code is going to be generated by AI. There's lots of bold claims about it.

The truth is there's a big up in telemetry when it comes to understanding how much AI authored code gets to production without large human intervention, I would say. DX is working on a tool. We've had a tool in Alpha that's doing this. It's not using AI to determine whether the code was AI generated, which I've seen a couple tools that do that that aren't very accurate.

We're looking on the actual file system level. It's agnostic across any AI tool, across any IDE, and then we can track it on the commit level and then track that commit to production really understand line by line, what was generated by AI, what was copy-pasted, and what was actually typed in by a human being, which is, I think, we just don't have that information right now. What we do have is self-reported data right now, which, you know, we don't have an alternative. So self-reported data is what we're going to go with right now.

I was just actually looking at, because I'm putting together a research report. On this, very things I can tell you the exact number. AI authored code. So looking across a sample of over 43,000 developers at over 265 companies, the average number of, like the average amount of self-reported code that gets, that is AI generated without large human intervention that gets merged upstream and put in a customer-facing environment is 22% looking at the last trailing three months. This is very different than 30%.

It's very different than 50% and it's definitely a lot different than 90%. So that's kind of the, you know, that's what we're seeing on the ground right now. And this is across all sorts of software projects, I guess. Yeah, we're looking at companies all the way from the Fortune 50, very large enterprises and all the way to startups. I mean, any sector is probably represented, and I made sure the sample is pretty well distributed across organization size as well, because that has a lot to do with adoption of AI tools, usage of AI tools.

Interestingly enough, though, that when I broke down, so this was like 40, what did I say just now, 43, 43,000 plus developers, when I broke them down into usage categories and looked at, you know, the people who are actually, and this is system data, this isn't their, like, self-reported data, this is system data. evidence from their AI tools, a usage event. And I'm looking at, okay, those that are using it daily or more, what is their percentage compared to someone who's using it just like monthly? And there's not a huge amount of difference. It's only a couple percent. It's like 21 versus 24 percent. We seem to like be hovering around the 20 percent range, 22 percent range pretty steadily for the last couple months. I'm really curious about how this changes as companies adopt more like agentic workflows, as people get more confident as we see some investments in specifically like SRE tooling, quality testing, become a bit more robust because

people have realized, oh, what's holding us back from using AI more effectively is not the latest model or whether we're using cursor or clod, it's that our testing is not sufficient. And like, we really need to rethink how we're going to get code that was not authored by a human safely to production. Companies are making investments in making that more robust. And, like, you know, I think the next six to 12 months are going to be very telling in terms of what happens. So that tells me back to basics.

Absolutely back to basics. Yeah. Mm-hmm. Because Laura, for you to understand Brian and I, we both have a big background and how we started in quality engineering. And that's why if we are trying, if we're getting into a world where AI generates more code, obviously we need to make sure that the safety nets are there and the best safety net is a good test. Hoping that the test was not generated by a body of that AI,

meaning another AI body that kind of created flaky tests. maybe as a parting thought I think back to basics is a great headline for what makes company successful with AI when we look at the things that make agentic workflows we'll just say agentic workflows really well it's like clearly defined services really robust testing the ability to detect anomalies in the system and fix them quickly production debugging observability all of these things are core components to develop our experience they're all the basics we've known them for decades now. And the interesting moment that we're at is that Agent X, which is what I'll call this, it's the same as developer experience, but companies are more willing to open their wallet

for Agent X. That's just where we are in time and space right now is like people are willing to invest in AI. So actually, we have a great opportunity to capitalize on this and make a lot of the investments that we've been wanting and advocating for, but couldn't, you know, maybe couldn't secure budget for it. Now we can frame it as an AI. investment and have access to bigger piles of money. But it is very much back to basics. AI is not a magic silver bullet. We need to approach it just like we approach every other tool and engineering fundamentals.

The physics of great software haven't gotten away just because AI is here. So does that mean I can look for a new job and paste some chips and stuff on me and say, look, I'm AI, hire me? You could try. Or maybe the people that really know the basics well, right, that write solid test frameworks that really build a foundation that invest in speeding up their CICD

where more tests can be done, faster feedback. This is a trade that will definitely not go away but is more important than ever. It'll be interesting to see if AI gets chipped up on the N-plus-1 problem that's when that just keeps

following every innovation around is N plus one. Hey Laura, thank you so much time flies and there are so many additional things that I would have liked to ask you I wanted to talk about

Egentic versus assisted versus workflow. I wanted to talk about the EI measurement framework that you've also mentioned in previous talks but I think we will leave this to hopefully another episode if you agree to come come back to us.

Yeah, I would be happy to and I am posting very frequently on LinkedIn research snippets, whatever I'm thinking about that day observations that I have so if folks want more information they should find me on LinkedIn

I'm pretty sure I'm the only Laura Tacho out in the world and we will obviously have the link to your LinkedIn and your talks and everything in the description of this podcast. It would be very interesting to see where this

all leads in the next five years or so. I think we're at an inflection point for sure, but all remains to be seen where we need that out. But thank you again, Laura, for taking the time and be with us. We really, really appreciate it. I was more quiet today because I was just listening and learning, which sometimes is my thing. So fantastic stuff. Thank you very much for being on. We look forward to having you on the next time. Bye, everybody. My pleasure. Thank you. You know what I'm going to be.