Everyday AI Podcast – An AI and ChatGPT Podcast - Ep 725: Measuring AI ROI: Why you’re doing it wrong and the 7 Steps to fix it (Start Here Series Vol 11)

Starting point is 00:00:00 This is the Everyday AI Show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business, and everyday life. Meet Firefly AI Assistant, now live in Adobe Firefly, the all-in-one creative AI studio. Just describe what you want to create and the assistant handles the rest, orchestrating multi-step workflows across Photoshop, Premiere Express, and more in one conversational interface. You direct the outcome. The assistant accelerates execution. Can you actually prove AI is working at your company?

Starting point is 00:00:50 Sure, everything feels faster and hopefully outputs have improved, and maybe more work is getting done. But is there ROI? And don't just give me vibes. Give me receipts. What's the actual return on your investment for AI at your company? chances are you can't answer that. And the reasons are actually very simple.

Starting point is 00:01:16 It's because businesses are using the same digital transformation playbook. They've always used when it comes to AI. And well, that playbook is useless in 2026. So there's a 99% chance you're not actually measuring ROI. But I'm going to simplify it for you on today's show and tell you why the ROI on a AI debate is absolute nonsense. And I'm going to give you seven simple steps on how your company can fix it today. All right.

Starting point is 00:01:49 Let's get measuring. So here's what you'll learn on today's show. You're going to learn the basics of measuring Gen A.I. R.O.I. And why most companies never actually do it. You're going to learn why a piece of MIT marketing disguised as a study misled markets in what five larger studies actually. show. You're going to learn how to use a four metric scorecard covering quality, cost,

Starting point is 00:02:15 reliability, and risk, and how to run a seven-step evaluation sprint that actually solves your ROI. So what the heck is this? Well, this is everyday AI, number one, but this is our start here series. I didn't have a good answer when people first subscribe to the podcast and they're like, Jordan, there's 700 episodes. Where do I start? I'm like, I don't know. Well, now I do know you start here with the Start Here series. This is the essential podcast series to both learn the AI basics and to double down on your AI knowledge. So make sure to go to start here series.com.

Starting point is 00:02:53 That's going to give you free access to our inner circle community. And then you can go in the Start Here series space and listen to now all 11 episodes of this series. So last episode, which was a banger, FYI, we, We talked about from AI chatbots to autonomous workers and how consumer AI has changed and what's next. So make sure you go check that out. It's episode 723, but volume 10 of the Start Here series.

Starting point is 00:03:24 And today we're going to be measuring ROI and tell you why your company is doing it wrong and these seven steps to fix it. Let's start here. Talked about this once or twice, but let's talk about this benchmark from OpenAI. It's called GDP bow. So this benchmark tests AI on real world deliverables across 44 jobs in nine different GDP sectors. Essentially, this is a unbiased benchmark that pits AI models against expert humans, experts in their fields, right? They each have the same task, which is generally to produce something of value, right?

Starting point is 00:04:04 That's why it's called GDP Bell to produce something of economic value. And then there's expert judges, expert human judges. And they don't know which output came from the AI. They don't know which output came from the expert human. And right now, top AI models either tie or win 70% of the time in these blind evaluations against professionals that have an average of 14 years of experience. I mean, my gosh, we should just end the episode now, right? There's your ROI on AI, right?

Starting point is 00:04:35 Done in under four minutes. There's a lot of issues here. Number one, even what I just said, right, do you know how to use today's models to, you know, one shot personalized, synthesize information, create, you know, spreadsheets, PowerPoint decks and use your own company's information without you having to do anything and verify outputs? Well, probably not because companies aren't educating their people. And it's hard to keep up with today's capabilities of AI models.

Starting point is 00:05:02 But what this did show us is that the whole debate on our. ROI is absolute rubbish. Yes, there we go. For listeners across the pond, look at me. I'm multilingual here. It's garbage. This whole discussion on does artificial intelligence give you a return on your investment? It's honestly a very dumb question, if I'm being honest, right?

Starting point is 00:05:30 Go read the GDP valve study. I keep saying, oh, I'm going to do a show soon. I will actually do a show soon because it's worth it. So not only did it show that, well, today's AI models are exponentially better than expert humans when judged by other expert humans, right? But it said that in many tasks, they do it a hundred times faster. So again, I am not a mathematician. But if the AI model is the same or better 70% of the time and it's 100 times faster, that's the math. So why are we still even debating this concept of is there return on investment?

Starting point is 00:06:11 Well, that's because of an infamous piece of marketing from MIT. So, and yes, I'm calling it a piece of marketing or study in very heavy quotes, right? Like the, what is that, the Austin Powers, you know, a billion dollars. Yeah, huge quotes on the word study. Right. So MIT claimed in their wildly viral study that 95% of enterprise AI pilots delivered zero ROI. And this was in August of 2025. Right.

Starting point is 00:06:45 And I mean, the markets moved, right? Ambitya stock tanked three and a half points. Palantir went down nearly 9%. There's actually billions of dollars of losses across the, not just the tech sector, just the business sector because, right, AI has essentially propped up in otherwise struggling U.S. and at times global economy. And then, you know, this study came out and said, oh, AI's, you know, there's no ROI. Well, most people, right, as a former journalist, I understand this, right? There is one study that came out and then essentially

Starting point is 00:07:20 every other journalist hit the copy and paste, you know, more or less, hey, that headline sells, you know, 95% of AI pilots don't produce any ROI, right? That's great. Let's go ahead and run that. But I don't think any journalists actually read the story or very few actually did. Right. That's because this quote unquote study was based on 52 qualitative interviews. There's no quantitative piece to that 95%. And it was directionally, right? This was a directional study. So it was more or less, this was a vibe. This was a vibe study. I'm going to talk to 52 people, you know, nothing quantitative. I'm just going to give my vibe on this. But the stat came from 52 qualitative interviews that measured P&S and impact in under six months, which there's no such thing, right? When you talk about digital transformation, go back and look at the internet, right? You couldn't have shown P&L impact in under six months on using the internet, right? Because if you could have, you could apply that exact same methodology to AI.

Starting point is 00:08:21 Anyways, well, why this is a piece of marketing and not actually just a bad study. I would call it a bad study, but it wasn't. It was marketing because in the end, they were selling their own product. They said, hey, right, all these AI pilots fail because you're not using our agentic AI product. And they were, you know, selling access to it. Yeah, I'll stop there. If you really want the download on that one, make sure to go listen to episode 597, where I, like a human read it, like a human with a brain, read the study multiple times and it's, you know, laughable.

Starting point is 00:08:54 Anyways, if you look at the real data, right, companies that talk to thousands and got quantitative data, obviously the ROI for AI is there, right? The IDC, the International Data Corporation, found a $3.70. Return for every dollar invested in AI. Wharton, a three-year study found that 74% of enterprises report positive ROI on Gen. Google Clouds, 2025 annual AI study found 74% of executives reported achieving ROI just within the first year of generative. of AI deployment. Also, Deloitte's 2025 AI survey found 84% of those investing in AI said that they were gaining

Starting point is 00:09:40 ROI. This, I think that this debate is still going on because humans are innately lazy, right? That's the reason why. We don't want to actually sit in measure, right? Again, my background, I have a little bit of marketing and advertising background. So I understand the importance of measuring something before and after to see if there's an impact, right? Running millions of dollars of ads for clients over the years, there's this thing called return on ads spent, right? You see how much, you know, how much money was spent on ads.

Starting point is 00:10:21 You have to be able to attribute and track the revenue that it brought in. And then you do some simple math, right? Return on ads. But, well, there is also a simple equation that I'm going to give you guys here in a little bit the same thing. Return on AI. So here's the dirty little secret. Adobe just introduced an entirely new way to create, bringing the power and precision of its creative suite into one conversational experience. Meet Firefly A.I. Assistant now live in the Adobe Firefly app, the all-in-one creative AI studio.

Starting point is 00:10:59 Powered by Adobe's creative agent, Firefly AI Assistant lets you start with your vision, Just describe what you want and shape the outcome as it takes form with the assistant. The assistant orchestrates multi-step workflows drawing on 60 plus pro-grade tools across Adobe Creative Cloud apps, including Photoshop, Illustrator, Premiere, Lightroom Express, and more to help bring your ideas to life. You can also get started with creative skills, a growing library of pre-built workflows for common creative tasks, like batch editing photos, creating mood boards, portrait retouching, and creating social variations. Every step the assistant takes is visible

Starting point is 00:11:38 so you can refine, redirect, or take over at any time. You stay in the driver's seat as the creative director. Adobe Firefly AI assistant now in public beta. See it today at firefly.adobie.com. Because you're probably thinking, okay, Jordan, well, if every company's using AI, well, why isn't everyone just seeing exponential growth? well, I think you would see that a little bit more if only 5% of businesses were using AI, right?

Starting point is 00:12:16 But most studies say more than 95 or 97% of enterprises. So the playing field has just gone up, right? The minimum entry is no longer like, oh, you're using AI. That's great. That's such a differentiator. No, no, it's you're using AI. Okay. I've been saying since 2023, it's the same thing as, oh, our company's,

Starting point is 00:12:38 using the internet. So, right? But here's what's happening. Workers are pocketing it. That's where the ROI is going, right? Because true productivity and true ROI, well, it requires results-driven metrics, not time-based management. Most companies haven't, you know, gotten this figured out yet. And I think a big portion of this is that remote and hybrid workers are using AI, whether it's approved AI or, you know, whether it's, you know, AI sprawl, you know, dark AI, whatever you want to call it. And they're just pocketing it. It's this invisible productivity, right? I've literally talked to countless people. I won't say hundreds because I don't know if it's actual 200 or more, but I've easily talked to more than 100 people over the past few years.

Starting point is 00:13:32 And very successful, right, even one of my, you know, what are my good friends, told me this like maybe one or two years ago, pretty high up at a company that was public company when he worked there. He said he automated about 95% of his job with AI, right? No one knew because he's at home. I'm like, what are you doing with all this time? He's like, oh, you know, chilling golfing a lot, right? That's the reality.

Starting point is 00:14:00 You know, so many companies, so many employees I think are just pocketing this time savings, right as we've gone to this uh you know remote hybrid workforce and i think that's one of the reasons why last year so many companies have this you know big return to work push uh and right now according to work day 89 percent of organizations haven't even updated their roles to reflect AI right so you're still using uh you know 2026 tools uh inside of job structures made 10 years ago right so why does that matter well again hiring and roles in today's enterprise are still pre-a-i so what that generally means well expectations or output is measured in a pre-a-i way which isn't necessarily the right thing right and in the same way that i've encouraged um you all for for years to not upskill or

Starting point is 00:15:04 reskill. That's a waste of time when it comes to AI, right? You need to unlearn and rebuild. Been saying that for a long time. Departments and companies need to do the exact same thing, right? You can't just, oh, let's add a little bit of AI to this job role or, you know, hey, let's, let's try to think a little more AI natively here at this company. No, you got to tear it apart. In the same way, that job roles haven't changed, right? That means, outputs aren't going to change either. But if you're using AI, if you've trained your people on AI, if they have the right AI, they're just banking that productivity. So here's how finding ROI is done. It's simple. ROI can mean a lot of different things. A lot of people just think,

Starting point is 00:15:53 oh, that means you're bringing in more revenue. No, I think mainly it's time saved. Right. But I think ROI also means just cost reduced, revenue increased, or risk avoided. It's not just prompts sent, right? I think that's what a lot of people, they're looking at the number of hours, your utilization rate. If you're using, you know, an enterprise tool like chat GPT, right? Everyone's like, oh, you know, we need to increase our utilization from, you know, 8% to 20%. And then companies, you know, hire us to, you know, train hundreds or thousands of employees. And it's like, okay, utilization is not the metric that pushes it. It's, are you saving time? Are you increasing revenue in avoiding risk.

Starting point is 00:16:36 The formula, well, simple math. It's the time saved or increased revenue. But let's look at time saved. So it's just time saved multiplied by fully loaded hourly rates minus any monthly AI subscription costs. Simple formula. And right now boards and CFOs just care about cost per task, cost per task throughput in error rates, right?

Starting point is 00:17:03 not usage dashboards, not utilization. But we're seeing the same reasons for failure, still years later, right? Through my own conversations and conversations with others, I can say when it comes to measuring or not measuring ROI, I still think that there's five patterns or five main reasons why either individuals and departments aren't able to do this. So number one, not having a pre-AI baseline. Number two, you know, doing these slow year-long pilots. That's a recipe for disaster.

Starting point is 00:17:43 Number three, celebrating one lucky run and making that your AI strategy. Number four, when vanity metrics become the focus. And number five, shiny object syndrome pulling focus away before real company-wide implementation can actually start, right? You get a little ground, you show ROI, and then you're like, oh, what about this? model. What about this? What about this? What about this? Right? Before you actually, you know, implement it company Y. And these aren't technology problems, right?

Starting point is 00:18:12 This isn't standard digital transformation. These are we as working human beings as knowledge workers, we don't have a playbook to follow. Right. So all we've been doing is following the same playbook that we've always done. Right. Those slow pilots, you know, you get one thing right and you're like, oh, this is what we do now. It's the wrong way to do this. The other big reason, well, training. The training actually leads to those five other failure, so it's more of an underlying foundational issue. Right. But right now, 49% of enterprise leaders cite recurring, sorry, site recruiting AI talent as their single biggest challenge.

Starting point is 00:19:01 all right AI you need documented workflow steps and company data and that process documentation is part of the ROI investment right education

Starting point is 00:19:15 training ongoing learning it is required right so many I think when when AI when an AI strategy or actual AI implementation looks like you know pushing

Starting point is 00:19:31 top down, you know, either C-suite or board pushing, you know, a certain tool top-down and not training or educating. That's when it's so easy to fail, right? I use AI all day, every day. And I'll tell you this, if I take two weeks off, I'm going to fail. That's the pace, right? People are always like, okay, well, Jordan, you clearly don't know what you're saying here. If AI is a hundred times faster, well, shouldn't we just cut all these jobs? No, I think jobs are going to look very different. I've been on record as saying this since the very first episode. AI is going to, I think, quote unquote, take away more jobs than it will create,

Starting point is 00:20:15 but it will create many roles that we don't even understand. I've always said, most enterprise organizations, they need a lot of people like me, that all they do all day is they're. keep up with AI and they apply it to that business, right? Very few businesses have that because they're like, okay, that's a crazy role. We're not going to let, you know, Bill over in IT or, you know, Deborah over in marketing, just spend all day, you know, tinkering with AI and seeing how to apply it.

Starting point is 00:20:45 No, right? No, but yes, that's exactly what you should be doing. And here's, all right, this is usually something I think we only give away to companies that pay us a couple dollars. But let me just go ahead and give you some of the secrets here. All right. Here's an acronym, base. You need a base.

Starting point is 00:21:04 This is the baseline rule. So this is base stands for baseline assessment of standard execution, pre-AI. All right. So before AI touches any workflow, you got to get that base, right? That's your time multiple employees completing that exact task without it. you need to time it. You need to record the average time, the error rate, the rework cycles and cost per completed task as the baseline. You can't go back and collect this retroactively, right?

Starting point is 00:21:37 Because usually what happens, well, if you've already, you know, sprinkled AI in the process and you've been doing it for a year, right? Let's say it's a 10-step process and, hey, number two and step two and step six, we, you know, we molded those a little bit around AI, right? It's too late, right? You need to do it before you implement AI. You need to measure how long it takes humans to go through and do these certain projects or do these certain tasks, right? Think of it like an internal GDP valve, right? You are going to go through and have a human with no AI go through and do this task, document every single step of the process, painstakingly, Because you need to see, okay, what other humans need to be involved in this? Are there other meetings?

Starting point is 00:22:33 What about the person checking the work? What about the communication going back and forth? You need to measure it all and document it all every single step. And then what's the completion rate? What's the error rate? What does quality look like? You need to establish those baselines. And then redo the entire process, right?

Starting point is 00:22:53 Blow it up. Measure it first. that's your base right baseline assessment of standard execution that's your base blow it up do the same thing with AI right not your first iteration iterate on it and then you measure in that number right there you need that because that is the base quite literally for how you're going to ultimately measure your ROI so now let's give you that seven step guy All right. I'm delivering. Here we go. Well, let me just bring my notes up here because, you know, in typical live stream unedited, unscripted fashion, I didn't put my slide up here with my seven steps. But don't worry, I got my notes. All right, there we go. Notes on the screen. So here's the seven step blueprint, right? Step one, we already talked about it. That's the, well, actually comes a little bit before the base. So step one, you have to define. You have to define. You have to. define what the heck it is you're doing.

Starting point is 00:24:01 Uh, right? And you have to be very rigid. You can't be flexible because you need to get, um, an accurate before and after. So that's defining the rubric, uh, defining the success criteria and KPI's before you even begin testing. And then step two, like I talked about, that's the base. You need to measure the human baseline.

Starting point is 00:24:21 And then you're going to, uh, you know, time multiple employees doing that and then record the averages. Uh, step three, get messy. Right. You have to build 20 to 40 real messy work examples, including drift cases, right? These aren't easy things for humans or for AI. All right. And then step four, you need to configure the exact production workspace.

Starting point is 00:24:47 It's the same plan, the same model, and the same permissions. So you're not flip-flopping, you know, between different people, different accounts. No, right? It has to be controlled like any experiments. and it has to be repeatable and scalable, the exact same criteria when you're taking this to a production run. Step five, you're going to run every three times. Okay.

Starting point is 00:25:12 So, sorry, run each test three times with memory off and also require proof artifacts. So here's what I mean by that. We're doing this on the front end, okay? If you listen to this podcast at all, you know, I'm a big believer in, bringing as many of your processes over to front-end large language models. We actually did a dedicated episode on that in the start here series, talking about an AI operating system. Yeah, this is all going to be doing things on as an example, chatchbt.com,

Starting point is 00:25:45 claw.com, jemini.com, right? But doing it with memory turned off in doing it usually in a temporary chat, if your, you know, AI operating system of choice lends itself to that. All right. So essentially there, you have steps three through six is that's your internal GDP val, right? You have to get the correct use cases. And then from there, you grade blind, right? So same thing. You have the series of people do it three times those 20 to 40 use cases using the same AI. model. And then you have humans do those same exact things as well. Multiple humans in the same way. I would suggest, you know, three different times running the same case and then three different sets of humans. And then you grade blind. You standardize the output format. You have to agree on, you know, what's a pass, what's a fail, what's the grading scale, etc. But then at that point,

Starting point is 00:26:54 well, you have the input and the output, right? At that point, after still, Step six, when you can have your grading criteria, you know, you run through and you do the test three times, you know, those 20 to 40 use cases three times in a large language model, memory off, temporary chat. You have your grading rubric. You have your humans do it, right? The same 20 to 40 tasks, three different humans. You have it right there. You have the time, right? You multiply the time that it takes the humans to do it on the AI side.

Starting point is 00:27:29 side, all right, minus out. You know, so take, let's say, it's 100 human hours, take their hourly rate, times it, minus the cost of whatever AI tools that you're using. All right. There's your augmented cost. And then you compare it to your human only cost, right? And obviously the human only cost is going to be much higher. Right.

Starting point is 00:27:53 The same thing. If you are using any, you know, paid non-AI tools for the human only cost, right? Maybe there's some, I don't know if you're using the Bloomberg terminal or right, whatever. You know, a certain SaaS and maybe you don't need to use that SaaS application. If you're doing it, AI Native, the same thing. You need to look at the total human and software costs. And there you go. That's your return on investment.

Starting point is 00:28:16 And then step seven, you need to retest this monthly after every model update. All right. And then track eight three month rolling average. Costs are going to go up and down. right you think that oh well they're just always going to go down well no sometimes uh you know the frontier AI labs will actually roll out an update that's under the radar it's not like going from a gpt 5-2 to a 5-3 right a lot of times there might be five 10 different versions of a gpt 52 until there is a gbt 53 and you know whether it's open a i anthropic google microsoft etc

Starting point is 00:28:52 sometimes you know one of those under the hood updates actually might make things worse all right so that's why you you need to retest monthly or after any major model update and then keep a three-month rolling average. There you go. All right. I'm going to go through it quickly now with no commentary. Step one, define the rubric and success criteria. Step two, measure the human baseline. Step three, build a 20 to 40 real messy work cases.

Starting point is 00:29:20 Step four, configure the exact production workspace. Step five, run with three times AI models. in three sets of humans. Step six is grade blindly and standardize the output format and criteria. Calculate your ROI. And then step tab, step seven is retest monthly. So there you go. That's how you measure return on investment in AI.

Starting point is 00:29:50 But let me just tell you this. And I'm going to be very direct when I say this. You don't need to do this. you absolutely should, right? You absolutely should. You don't need to. This is gravity, right? AI is gravity at this point.

Starting point is 00:30:15 It is the all-encompassing force, and there's no denying it, right? Real studies with quantitative data that asked thousands of business leaders all over overwhelmingly show. ROI in AI is real. $3.70 for every dollar invested. The GDP val, right? AI models without training, right? Without iteration in a single, in a simple shot, blind test,

Starting point is 00:31:01 the AI model is the same or better than the expert. 70% of the time and 100 times faster. So, I'm going to leave you with this. Yes, you need to configure and figure your ROI on AI.

Starting point is 00:31:24 But the new ROI question is not, did AI work? It's how much did we lose by not educating and measuring and measuring sooner. That's the reality. And that is the challenge to you today, dear listener,

Starting point is 00:31:46 stop looking at R.O.I. On AI, like it's something tricky. Like it's something we can't all achieve. It's as simple as being meticulous in your measurement. And that's it. When you are meticulous in your measurement, you will undoubtedly see an insanely high return on investment of your AI. So no longer a question mark, no longer does AI work. The question is, in the way you need to change, is how much are we losing by not measuring sooner and educating our teams better?

Starting point is 00:32:36 All right, that's a wrap, y'all. That is volume 11 of the Start Here series. I hope this was helpful. And if it was helpful, well, number one, please like and follow the podcast. I'd appreciate that. But then when you're done, go to starthereseries.com. That's going to give you free access to our inner circle community. And then you can go listen to the entire Start Here series all right there.

Starting point is 00:33:01 And connect with others who are trying to grow their company and their careers with Generative AI. And hey, while I have you, make sure if you haven't already, go listen to the 2026 AI prediction and roadmap series. That's episodes 712 and 713. So thank you for tuning in. Hope to see you back tomorrow and every day for more everyday AI. Thanks y'all. Meet Firefly AI Assistant. Now live in Adobe Firefly, the Allman One Creative AI Studio.

Starting point is 00:33:32 Just describe what you want to create in your own words and the assistant handles the rest. Orchestrating multi-step workflows across Adobe Creative Cloud apps, including Photoshop, Premiere, and more in one conversational interface. You direct the outcome while the assistant accelerates execution. Stand control with the ability to step in and refine at any time. See it today at firefly.adobie.com. And that's a wrap for today's edition of Everyday AI. Thanks for joining us.

Starting point is 00:34:05 If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more AI magic, visit Your EverydayAI.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.

Everyday AI Podcast – An AI and ChatGPT Podcast - Ep 725: Measuring AI ROI: Why you’re doing it wrong and the 7 Steps to fix it (Start Here Series Vol 11)

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.