The AI Daily Brief: Artificial Intelligence News and Analysis - Knowledge Workers Produce 40% Better Results When Using AI - Study

Episode Date: September 18, 2023

A new study undertaken by Boston Consulting Group and prominent AI researchers shows that consultants who used generative AI did more work, more quickly, and better when they were using AI. NLW discusses the nuance and implications. Before that on the Brief: a new study reveals that many large companies are hesitant to fully embrace AI due to concerns around bias; Gen Z is the generation most likely to trust AI and about a quarter of senior finance professionals are worried AI will replace them. Today's Sponsor Netsuite | The leading business management software | Get no interest and no payments for 6 months https://netsuite.com/breakdown TAKE OUR SURVEY ON EDUCATIONAL AND LEARNING RESOURCE CONTENT: https://bit.ly/aibreakdownsurvey ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

Today on the AI breakdown, we're looking at a study that shows that consultants using AI are significantly more effective and faster than their peers who are not. Before that, other brief, some interesting survey results about professionals and AI. The AI breakdown is a daily podcast and video about the most important news and discussions in AI. Go to Breakdown Not Network for more information about our Discord, our newsletter, and our YouTube channel. Welcome back to the AI breakdown brief, all the AI headline news you need in around five minutes. Well, as you will see today from our main episode, which is all about a new study about how consultants using AI outperform consultants, not using AI. The big theme for today seems to be the professional use of AI and how it is finding its way or not into businesses. Given that, we kick off

with a study that shows that these ethics concerns are not just the conversation for podcasters and YouTubers, but are actually having serious impacts on how businesses are adopting AI. Workforce Solutions in Marketing Operations Consultancy, Algo Marketing, has just released the results of a survey of over 300 global marketing leaders that all work in firms that have 10,000 or more staff. There are a lot of interesting things from this study. The headliner is that a significant number of these firms have delayed their implementations of AI because they have concerns around AI ethics, specifically concerns around bias and fairness. 77.5% of the firms that were surveyed said that these issues have forced them to delay implementations, and 32.7% of those said that the delays have been significant. Now, another big problem when it comes to implementation of AI appears to be around talent and resources. 44.4% said that they have had issues finding people that have the right skills in both marketing and AI, and only 23.2%

of these companies said that they have adequate internal resources to take advantage of these new technologies. 52% said that they're opting for a blended skills approach using AI implementation in-house alongside external vendors and consultants. Another major challenge is cost, with 51.3% saying that the price tag of AI services is a barrier to adoption, but at the same time, the companies that are actually investing in AI are seeing pretty significant returns. 23.2% of those firms said that they're seeing a return on investment of more than 75% in marketing spend, while the majority, 68% report an ROI between 50 and Again, this is just for the marketing use of AI, but still pretty promising results. The geographic breakdown was also really interesting as well. In the U.S., only 26.7% of the firm surveyed

said that they've been using AI tools in the last three years, compared to 45.3% in the UK and 54.5% in Australia. Now, one of the big questions that I am left with after reading these results is the extent to which these ethical concerns are even more pronounced because we're talking about marketing, which is ultimately going to lead to a public-facing use case. Specifically, if you are tapping into something like chat, GBT, to help you write marketing copy, problems of bias are going to be even more pronounced and visible. Over in the world of finance, another survey says that 23% of senior finance professionals are worried that AI might put them out of a job.

This study comes from accounts IQ and was conducted on over 500 financial professionals and accountants from the UK. All the finance professionals surveyed work in companies between 40 and 500 employees, and they were evenly split between people who are 31 and up and have more than 11 years of experience, and those who are between 18 and 35 who have three years or less experience. Now, interestingly, in spite of the fact that they are worried about AI replacing them ultimately, that hasn't stopped these companies from experimenting with the technology. 24% of finance functions among those surveyed were underway with onboarding of AI, and 51%

planned to integrate AI in the next 12 months. Now, it probably follows from that that these professionals do believe that AI is going to be useful inside the enterprise. 41% see AI transforming their ability to add value to their business, while 34% say that it will help them save time and increase efficiency. Among the young professionals, there was even more optimism. 82% of the younger cohort surveyed said that tools like ChatGPT will impact finance teams within the next five years, and that a lot of what that younger group is looking forward to is automation of manual tasks that take up a significant portion of their time. 75% said that

they spend up to a quarter of their time on manual tasks such as data collection, and more than 60% said that they still rely on time-consuming spreadsheets as their primary tools. Now, different contexts, but a similar study, a new study commissioned by user testing and conducted by one poll, found that Gen Z was the most trusting generation when it came to AI with 67% saying they trusted it, while Baby Boomers were the least trusting with only 29% saying they trusted AI. Now, maybe not surprisingly, it appeared that where people trusted AI was in lower priority tasks, where the cost of getting things wrong, was simply lower. For example, making reservations, choosing clothes, auto-ordering household items when they were running low, but things that involved complicated financial issues like taxes, anything having to do

with their children. On those areas, people were much less trusting. The study also found that 72% were concerned around not knowing what was being done with their personal data. One thing that really stood out from this is the extent to which there was a dichotomy between how much people said that they didn't trust AI or were concerned with how their personal data was being used, but then who still reported being willing to give that data away if it got them better deals. Now, staying on the theme of AI automation and employment, Bernie Sanders waded into the AI conversation, saying that he believed that if AI makes us more productive, that benefit should show up in a shorter workweek. It seems to me that if new technology is going to make us a more productive society, he said, the benefits should go to the workers.

It would be an extraordinary thing to see people have more time to be able to spend with their kids, with their families, to be able to do more. Cultural activities, get a better education. So the idea of making sure artificial intelligence and robotics benefits us all, not just the people at the top, is something absolutely we need to be discussing. These comments came around the United Auto Workers' strike, which includes a part a debate around a potential four-day workweek. I actually think this is a hugely important conversation and one we're not having nearly enough. The market economy will fill in and use every hour basically of work that people are willing to give. But that doesn't mean that as a society we can't renegotiate the social contract based on changes in what technology

allows us to do. This has been a theoretical conversation for quite a while, but is now becoming much more applied. And while obviously there's going to be lots of sides to the debate, I'm glad that people with the standing of Bernie are actually bringing it up. And with that, we will wrap this slightly shorter edition of the AIA breakdown brief. If you're interested in these questions of employment and work and AI and where it all fits, make sure to comment tune back in for the main episode, which will be out shortly. Before we get to the main episode, I want to tell you about today's sponsor, NetSuite. I know many of you guys are entrepreneurs, executives, managers, business leaders who are trying to figure out how technology is changing the world and how it can change your

business. Given that, I am thrilled to have NetSuite as a sponsor of the AI breakdown. NetSuite gives you the visibility and control you need to make better decisions faster. It is the software superpower behind so many of the world's most successful companies. And for the first time in NetSuite's 25 years as the number one cloud financial system, you can defer payments. of a full NetSuite implementation for six months. That's no payment and no interest for six months and you can take advantage of this special financing offer today. Listen, NetSuite is number one because they give your business everything you need in real time all in one place to reduce manual processes, boost efficiency, build forecasts, and increase productivity across every department.

I know that if you are listening to the AI breakdown, you understand intuitively and deeply just how much data matters to any modern business. Having all of your information in one place can be the difference between making the right decision and making the wrong one. I think it's awesome that NetSuite has this new offer designed to really make their suite of tools available for all the businesses that need it. So if you have been sizing up NetSuite to make the switch, then you know that this deal is unprecedented. No interest, no payments, take advantage of this special financing offer at netsuite.com slash breakdown. Go to netsuite.com slash breakdown to get visibility and control you need to weather any storm.

That's net suite.com slash breakdown. And with that, let's get to the show. Welcome back to the AI breakdown. One of the biggest questions surrounding artificial intelligence, especially when you get outside of the questions of whether it's going to extinct us entirely, is how it is going to impact the professional world. Now, within that, there are multiple sub-questions, specifically what jobs and industries will AI almost entirely automate away

versus what types of roles will AI change the balance of what people spend their time on? Of course, underlying all of that is an assumption that artificial intelligence is going to singularly improve how we work. In other words, that it actually has the capacity to be transformative. Certainly, one of the things that has markets most excited about AI is the potential boon and productivity that it represents. For example, back in May, legendary hedge funder Paul Tudor Jones said AI could bring the type of long-term productivity boon that we just don't see very often. And yet, by and large, so far, when it comes to AI's impact on our work, the evidence has been large. largely anecdotal. In other words, when one uses ChatGBTBT to change the speed with which they accomplish a task, or to expand the breadth of ideas they bring to a particular situation,

we have a sense that this is a transformative technology. But that's different than actually studying it with some sort of parameters that allow us to quantify what that change actually actually looks like. Wharton Professor Ethan Mollock back in February, captured this sentiment a little bit when he started playing around with Bing AI, which, as we would later learn, was actually testing GPT4. He asked it things like, could you conduct a SWAT analysis of AI use in agriculture? And I want to compare AI use in agriculture across different regions, and pointed out that every consultant in the world was probably shocked when they saw this type of results. He said, I'm sure that some of the data is not correct, but this feels like magic. Well, today we are back with Ethan Mollick,

and we're talking about a study in which he and a number of colleagues actually tried to put some numbers around just how powerful that magic actually is. In a blog post, he writes, a lot of people have been asking if AI is really a big deal for the future of work. We have a new paper that strongly suggests the answer is yes. So basically what has been happening is that Ethan has been working with a group of other social scientists and the Boston Consulting Group, turning their offices as he puts it into the largest pre-registered experiment on the future of professional work in our AI-haunted age. He says lots of important and useful nuance in the paper, but here's the headline. For 18 different tasks, selected to be realistic samples of the kinds of work done

at an elite consulting company, consultants using chat GPT4 outperformed those who did not, by a lot. On every dimension, every way we measured performance. Now, first to understand this study, we have to understand how they were attempting to construct the test itself. Even in his colleagues refer to something they call the jagged frontier. And the idea of the jagged frontier is that when it comes to what AI is actually good at, it's not necessarily always super clear. The example that he points out is when you give AI two tasks that might seem really similar. To use his example writing a sonnet or an exactly 50-word poem, they're actually on different sides of this frontier. It's very easy for AI to write the sonnet, but it's very hard for it to write the exactly 50-word

poem because AI doesn't think about the world in terms of words. It thinks about it in terms of tokens. So in order to test the impact of AI on knowledge work like that done by consultants, What they did was create a variety of work for a fictional shoe company, a set of work that they had worked with the BCG team to design to accurately represent what their consultants might be asked to do in the course of a normal engagement. There were four different categories of tasks. Creative tasks, analytical tasks, writing and marketing tasks, and persuasiveness tasks. So creative tasks were things like proposing ideas for a new shoe targeting an underserved market or sport. Analytical tasks were things like segmenting the footwear industry market based on

users, writing and marketing tasks were things like drafting press releases, and persuasiveness tasks were things like, quote, pen an inspirational memo to employees detailing why your product would outshine competitors. Now, from there, they took hundreds of BCG consultants and randomized whether they were allowed to use AI or not. This wasn't any sort of fancy model, just GPT4, the same thing that anyone who's willing to pay $20 a month to open AI can get. From there, they measure performance in a number of different ways. They measured how long it took to complete certain tasks. They measured the number of tasks completed overall given an overall time limit, and they measured the quality of the outputs which was rated both using human and AI graders, which, as an aside,

they found interestingly that human and AI graders agreed with each other in most cases. Now, this gets us to the splashy results, which are going to show up, I would imagine, in lots of headlines in tweets. The TLDR, as Ethan said, is that consultants using AI had market improvements from consultants that weren't using AI. They finished 12.2% more tasks on average in their allotted time. The tasks that they completed, they completed 25.1% more quickly. And based on that human and AI quality grading, the consultants using AI produced 40% higher quality results than those who didn't use AI. So conclusion one clearly seems to be that having access to AI tools does improve both the quality of the work and the speed with which it gets done. One interesting

piece of nuance within that was that AI didn't serve to expand the range to the highest performing to the lowest performing consultants, in other words, it didn't make the best better than it made the worst. Instead, AI acted like a skills leveler. The team assessed all the consultants at the beginning of the experiment. And those who were in the bottom half of skilled participants saw a significantly bigger jump than those who were in the top half of participants. The bottom half of skilled participants saw a 43% increase in performance when they got to use AI. Meanwhile, the consultants who are in the top half based on that initial assessment still saw a jump, a meaningful jump of 17%. but obviously that's a lot less than 43%. Now, Professor Malik barely has time to get into that part of the

results, but obviously that alone has huge implications for things like the cost structure of consulting, the competitive landscape of consulting, and just in general performance when it comes to knowledge work. Now, what about negative sides? One of the things that they tried to uncover as part of the study was whether there were types of work where humans without AI would actually outperform humans with AI. Ethan writes, we identified a task that used the blind spots of AI to ensure it would give a wrong but convincing answer to a problem that humans would be able to solve. Human consultants got the problem right 84% of the time without AI help, but when consultants use the AI, they did worse, only getting it right 60 to 70% of the time. Part of the team's speculation is that over-reliance on AI can indeed

backfire. In a separate experiment from one of the researchers who was also involved in this paper, he had uncovered that recruiters who were using high-quality AI became, quote, lazy, careless, and less skilled in their own judgment. Quote, when the AI is very good, humans have no reason to work hard and pay attention. They let the AI take over instead of using it as a tool. He called this falling asleep at the wheel and it can hurt human learning, skill development, and productivity. In our experiment, we also found that the consultants fell asleep at the wheel. Those using AI actually had less accurate answers than those who were not allowed to use AI, but they still did a better job writing up the results than consultants who did not use AI. I think what actually matters about

this study is the combination of the clear definitive results that these consultants using AI did outperform their peers who weren't allowed to use AI, but combined with the propensity for those using AI to become over-reliant on it. A lot of the output of this type of study is going to be manifest in terms of how companies think about bringing AI into their organization, how they prepare their workers to use it, how they figure out training to minimize the downside and this propensity for falling asleep at the wheel. And of course, this is also going to filter down into educational structures as well. Now, what's very clear is that the consulting industry is going to be an absolute sandbox for watching how early adopters change in industry. Just over the last few months, we've seen

basically all of the big consulting firms announce major AI initiatives. Last week, EY unveiled their $1.4 billion AI platform that they're training all 400,000 of their workers on. In August, McKinsey debuted Lilly, which they call, quote, our generative AI tool that's a researcher, a time saver, and an inspiration. Now, I've spoken before about how tools like Lilly, which bring together information from across a big consulting organizations siloed projects and put it into a place where that knowledge can be shared just makes a ton of sense in terms of institutional learning being able to influence new projects. Lilly, for example, is trained on more than 100,000 documents and interview transcripts, including a ton of proprietary information that wouldn't be available to some outside model.

In June, Accenture announced that it was going to be spending $3 billion on AI, including doubling the size of its AI-focused staff to $80,000. And if nothing else, there is a clear business logic to these big announcements, given that, as the Wall Street Journal put it in June, quote, consultants emerge as early winners in generative AI boom. Lacking in-house know-how, companies are turning to outside experts for help putting chat GPT-like tools to work. In the same way that enterprises are turning to the software vendors that they already know, their cloud providers, for example, to help them deploy and develop AI models. They're turning to the consulting firms that they already trust to help them figure out the

business processes around using these new AI tools, which is creating the financial feedback loop for these consulting firms who are spending big bucks, re-skilling their entire workforces on AI. Now, of course, one interesting subpart of this story is that some have wondered if AI is actually a threat to management consultant jobs. And sifted from FT in August, Tim Smith asked, will AI kill the management consultant? Management consultants are some of the highest paid, highest charging professionals in the world. Could AI be coming for their jobs? The article discusses how even before the rise of AI, there had been downward pressure on the expensive price tags for these consulting engagements, given structural changes and things like remote work. And this to me

is where we get back to this skills leveling question. A lot of what those big firms charge for is the premium associated with their brand. Does that premium change in a world where AI is leveling the playing field? Or are brands still going to be shorthand for quality because firms will still assume that McKinsey using AI is better than some other random firm that they've never heard of using AI? And of course, this gets back to the tools question. Can companies like McKinsey keep the premium on their services because they're using their own proprietary AI trained on consulting engagements of the like that enterprises are hiring them to do in the first place. Anyways, I think it's a fascinating microcosm of these larger questions of how AI will impact work in general, but white collar

work specifically. And I'm really glad that we're starting to get more actual studies around these topics, so it's not just podcasters speculating. That is going to do it, however, for today's episode. I hope you found this interesting or useful. If you did chuck me a like or subscribe, and until next time, peace. Thank you.