ColdFusion - AI Fails at 96% of Jobs (New Study)

Episode Date: July 5, 2026

Artificial intelligence has been hailed as one of the most transformative technologies of the century. That may be so, but just not yet. In this episode, we take a look at a study that pits humans directly against AI for paid work. The results were surprising. Study here: https://www.remotelabor.ai/paper.pdfWebsite: https://www.remotelabor.aiWatch or listen to ColdFusion on Spotify: https://open.spotify.com/show/1YEwCKoRz8fEDqheXB6UJ1ColdFusion Music: http://burnwater.bandcamp.com https://www.youtube.com/@ColdFusionmusicColdFusion Socials: https://discord.gg/coldfusionhttps://facebook.com/ColdFusionTV https://twitter.com/ColdFusion_TV https://instagram.com/coldfusiontvCreated by: Dagogo AltraideProducers: Tawsif Akkas, Dagogo Altraide Learn more about your ad choices. Visit megaphone.fm/adchoices

This episode is brought to you by Accenture. When your advertising operations fall out of sync, everything else follows. Spotify and Accenture are working together to reinvent the rhythm of ad sales, using automation, analytics, and smarter workflows to simplify campaign delivery and access better data across the business. The result? Less time spent on operations, more time connecting brands with the moments and fandoms that matter most. Learn more at Accenture.com slash Spotify.

Hey y'all, it's Kelly Clarkson with Wayfair. Ever order furniture online and wonder, what if? Like, what if it doesn't hold up? That sofa was four days old. You should have ordered from Wayfair. With Wayfair, there's no what if. Just style you love and quality you can trust. Visit Wayfair.ca.

Wayfair, every style, every home. In the absence of AI and robotics, we're actually totally screwed. We are working to build tools that one day could help us make new discoveries and address some of humanity's biggest challenges, like climate change and curing cancer. Hi, welcome to another episode of Cold Fusion. Here's a question, how can AI be disrupting the job market but also be losing billions of dollars at the same time?

Well, this video will answer that. The truth is, while AI helps make some jobs easier, when compared to a human it performs worse a whopping 96.25% of the time, which basically means, given AI, 10 tasks, and it will perform at least nine of them worse than when compared to a human. That's at least according to a new study. It's such an interesting finding and begs the question, why has no one systematically compared how well AI does versus a human who's done exactly the same job? All previous benchmarks have been simulated human work, not real generalized work.

The results from the team of researchers who did the study makes one think, maybe the true value of consumer AI isn't hundreds of billions of dollars, but orders of magnitude less. I'm not saying that all AI sucks. This study is just a general reminder that AI is a time-saving tool and not a replacement. Just maybe the economy is valuing it too highly when it comes to near-term capabilities. In this episode, we'll take a look at the study in detail and discuss what it all means. So the synopsis of the study was straightforward enough. Give paid jobs already completed by real people to AI models and then see how well the results compare.

Once the AI completes the tasks, humans evaluate the results. The researchers call this method the remote labour index, or RLI. It's so simple, most of us use a computer to do modern work, right? So why not just directly compare how well AIs compete on a professional computer-based job? The jobs to be completed were real ones from the freelancer site Upwork, a site where you pay remote workers to complete any given task. The jobs were varied from video creation, computer-aided design, graphic design, game development, audio work, architecture, and more.

Both humans and AI were given the same brief and any attached files that were necessary for the job. For example, an Excel spreadsheet of data or instructional images. The AI models were tested on 240 jobs, each paying $630 on average. So, how did they perform? The performance was abysmal. The best AI was Claude Opus 4.5, with a 3.75% success rate, when it came to producing work of an acceptable quality. You heard that right, a 96.25% failure rate was the best performer.

Interestingly, Gemini was the loser with a 1.25% success rate. Now, Claude Opus 4.6 might score 5% better, but that's still a 91% failure rate. When these scores get to 35%... In Toronto, every arrival is a statement, and nothing says it better than this. Cadillac Optic was the number one selling luxury EV in Canada for 2025. Find your rhythm across a seamless 30%. 3-inch display and an immersive 19 speaker AKG surround audio system. This city demands agility and optic delivers with precision to make every drive extraordinary.

Let's take the Cadillac. Find out more at Cadillac Canada.ca. Luxury sales claim based on S&P Global Mobility Canadian New Vehicle Total Registrations for calendar year 2025 for the Cadillac definition of luxury. This episode is brought to you by L'Oreal Group. Beauty is a powerful force that moves us. That's why L'Oreal Group has built a business that is inclusive at its heart. with 100% of its brands championing diversity.

With 25,000 professional opportunities for people under 30 worldwide and 54% of leading positions held by women, diversity is a strength that helps L'Oreal Group create the best beauty products for all people. Visit L'Oreal.com to learn more. Percent of 40%, then we can talk. So a couple of things to note, the original paper used AI models that were six months or so old, but their website has up-to-date results, which are the scores that that I'm referring to in this episode.

I'll leave a link for the website below. So where exactly did the AI systems fail? Well, first we need to define exactly what failure means. Failure counts as not performing a task at, or better than a human level. This is specifically in the context of a freelancing environment, an environment where people actually pay money directly for the work. With that in mind, the paper lists four main failure points

for AI systems. Number one, sometimes the AI would produce, quote, corrupt or empty files or deliver work in incorrect or unusable formats. Number two, AI, quote, frequently submitted incomplete work characterized by missing components, truncated videos, or absent source assets. For example, a video of eight seconds when an eight-minute video was required. Number three, another one was quality issues. Quote, even when agents produce a complete deliverable, the quality of work is frequently

poor and does not meet professional standards. end quote. And finally, number four, inconsistencies with AI generated work. This includes a house's appearance changing across different 3D views or digital floor plans that don't match the supplied sketches. It's all very interesting. So for years now, we've been told that AI is going to replace humans everywhere. But the truth is, we are nowhere near that point, at least not yet anyway.

So then, where did the AI succeed? Success would mean that the AI does the same work at the same quality or better quality than human output. They note that AI was proficient in creative ideas, like audio and image-related work, along with writing, data retrieval or web scraping, and that kind of checks out. The success of OpenClaw attest to the latitude, and AI images and audio are already good enough to fool a lot of people. Advertiment and logo creation was another successful area. It's also no surprise that AI was good at report writing. and generating simple code for an interactive data visualization.

Competent video generation is coming very shortly. Just take a look at CDance 2.0. Main takeaway is AI is pretty good at some things, but horrendous for general work. But what else do we learn? This paper exposes a lot, much of it negative, but it does show that the RLI format is a very useful measure of AI performance in the real world.

Reason being, current-day benchmarks aren't reflective of real-world performance. of real-world performance. As the paper puts it, quote, while AI systems have saturated many existing benchmarks, we find that the state-of-the-art AI agents perform near the floor on RLI, end quote. I found the study to be very robust, by the way, so I'll leave a link to it below.

According to this study, AI may impact jobs with lots of language requirements, audio, simple advertising, or data retrieval, but human oversight is still needed. A PWC report found that the majority of CEOs, see no financial returns from AI. Upper management and CEOs just command workers to use AI and expect it to all work. For AI to work within a corporation, there needs to be a planned and skilled implementation of the technology with the knowledge of its shortcomings. And that doesn't happen

a lot of the time. Gartner predicts that by next year, half of the companies that fired workers for AI are going to hire them back. Also, nine months ago, Microsoft proudly proclaimed that 30% of their code was written by AI. And since then, We've seen seen some of the worst software issues at the company in its history. Now it's obvious that AI is disruptive and some jobs will be lost to the technology. For example, diffusion models are proficient in the visual arts as you saw earlier, but as for LLMs and the general workforce, this study indicates that job losses could be a lot less.

The AI space does move fast, so I could be wrong, but that's how things are looking today in early 2026. To sum up the job prognosis in one line, if you're a software engineer, set up a business that fix us vibe-coded apps, and you'll make a lot of money. I think the thing is, artificial intelligence really is going to transform the world, like in ways we can't even imagine. But it's not going to do it now, not with this technology. My favorite example of this is one trains them on the whole internet,

so they get access to a lot of written rules of chess and lots of games of chess, and they still make illegal moves. They never really abstract the model of how chess works. That's just so damning. You would not be able to learn chess. after seeing a million games, reading the rules in Wikipedia and chess.com, just making it bigger is not going to solve these problems. We need to do foundational research.

That's what I was saying for the last five years. What is intelligence, the problem is to understand your world. And reinforcement learning is about understanding your world, whereas large language models are about mimicking people, doing what people say you should do. They're not about figuring out what to do. Just to mimic what people say is not really doing. to build a mile of the world at all, I don't think. So I'm not saying that AI will never work,

or it's not genuinely useful. This episode is brought to you by Activia. You might already be eating yogurt, but not all yogurts are created equal. Activia contains over one billion probiotics per serving to survive and reach the gut alive. When it comes to gut health, Activia is the number one family doctor-recommended probiotic yogurt brand.

Choose Activia. Feel good from the inside out. Visit Activia.ca for more details. When you're a mid-sized business, you need every competitive advantage you can get. Like an AI solution that works for you, not against you. SAP Grow is built with AI embedded at its core, working across every system. And it's ready to go from day one so you can hit the ground running. Bring it with SAP Grow, AI Cloud ERP for any size business.

Ready, there will be some narrow AI products that work really well. I'm just warning that there's a significant financial risk in the current AI space. The investment ethos and the rollout of AI everywhere might be misallocating hundreds of billions of dollars. Even in the medical field, Reuters just reported that the FDA has received 100 reports of AI malfunctions, botched surgeries and misidentified body parts. In a few cases, a lawsuit alleges that the AI misinformed the surgeons on the locations of their instruments, causing one to mistakenly puncture the base of a patient's skull, and causing

strokes from the damage to a major artery in two others. We don't need to put AI in every field. It's just not ready yet. Again, in some fields like coding, high maths, and writing, AI is pretty good and can make jobs a lot easier, but we can't pretend like it's going to replace everyone perfectly right now. Now I was going to stop the video here, but just a couple of personal thoughts. Back in 2016 when I started covering AI, it was fun and fascinating to see how these things worked. But ever since the big money started coming in, the hype has just gone off the charts. CNBC just reported that companies like Anthropic, Google and Microsoft have paid individual content creators 400,000 to half a million dollars each to promote their AI models. Now brand deals are fine, but if the current generation of AI was as revolutionary as being advertised, they wouldn't need to spend so much money to convince us.

It's a jarring disconnect. One last thing. We're fooled into thinking those machines are intelligent because they can manipulate language. And we're used to the fact that people who can manipulate language very well are implicitly smart. But we're being fooled. Now, they're useful. There's no question. There are great tools like computers have been for the last five decades.

But let me make an interesting historical point. And this is maybe due to my age. There's been generation after generation of AI scientists since the 1950s. claiming that the technique that they just discovered was going to be the ticket for human-level intelligence. You see declarations of Marvin Minsky, Newell and Simon, you know, Frank Rosenblatt, who invented the Perseptron, the first learning machine in 1950, saying like within 10 years we'll have machines that are as smart as humans. They were all wrong. This generation with L&M is also wrong.

I've seen three of those generations in my life. time okay so you know it's it's just another example of being fooled that's Yan Lee Kuhn the creator of convolutional neural networks he's been outspoken in saying that the current AI architecture is reaching its peak he thinks that throwing more data and power at the problem isn't going to solve it and I think that's what the early data is showing us it's called the scaling problem and it's a large part of my upcoming video about how open AI is in big trouble when it's complete I'll leave a link for that episode below

So be sure to check it out after this. Anyway, that's about it from me. You've been watching Cold Fusion? Let me know your thoughts. I'm sure the comments section will be very, very full of very good discussions. Anyway, that's it. My name's DeGogo, and I'll see you again soon for the next episode. Cheers, guys. Have a good one.

It's new thinking. Are you one of those media strategy people clicking through slides, scrolling spreadsheets? Yes? Good. This is for you. Because on Spotify, there's an audience that's different. in, loyal, invested. They're called fans.

Fans don't just listen to music. They feel seen by it, like it belongs to them. So when your brand shows up on Spotify, that's who you're talking to. And you're right next to artists like me, Lizzo. So, are you ready to talk to fans? Spotify Advertising. You're among fans.