Better Offline - Yep, We're At Peak AI

Episode Date: December 4, 2024

In this episode, Ed Zitron discusses big tech's discovery of the diminishing returns in training generative AI models - and how we may have finally have reached peak AI. --- LINKS: https://www.tinyurl.com/betterofflinelinks Newsletter: https://www.wheresyoured.at/ Reddit: https://www.reddit.com/r/BetterOffline/  Discord: chat.wheresyoured.at Ed's Socials: https://twitter.com/edzitron https://www.instagram.com/edzitron https://bsky.app/profile/edzitron.com https://www.threads.net/@edzitronSee omnystudio.com/listener for privacy information.

This is an IHeart podcast. Guaranteed Human. Run a business and not thinking about podcasting. Think again. More Americans listen to podcasts than adds supported streaming music from Spotify and Pandora. And as the number one podcaster, IHearts twice as large as the next two combined. Learn how podcasting can help your business. Call 844-844-I-Hart.

Another podcast from some SNL late-night comedy guy, not quite. Unhumor me with Robert Smygel and friends. Me and hilarious guests from Bob Odenkirk to David Letterman help make you. you funnier. This week, my guest, S&L's Mikey Day and headwriter, Streeter Seidel, help an a cappella band with their between songs banter. Where does your group perform? We do some retirement homes. Those people are starving for banter. Listen to humor me with Robert Smigel and friends on the I-Heart Radio app, Apple Podcasts, or wherever you get your podcasts. Life is full of hurdles, so how do you keep going? On Hurtle with Emily Abadi, we're talking with

the most inspiring women in sports and wellness from professional athletes, coaches, and Olympic champions about the challenges that shape them and the mindset that keeps them moving forward. At our level, at this scale, being able to fail in front of the entire world, like, I can do anything. I can do anything. Listen to Hurtle with Emily Abadi on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. Presented by Capital One, founding partner of IHeart Women's Sports. I'm Michelle McPhee, and I've been unraveling the strangest criminal alliance I've ever reported on. a Mormon polygamist and an Armenian businessman.

Multi-million dollar house, Ferraris and Lamborghinis, private jets, a billion dollar fraud. But how long can this alliance last? Tell me what you know. Is somebody coming after me? Listen to Kingdom of Fraud on the IHart Radio app, Apple Podcasts, or wherever you get your podcasts.

Imagine an Olympics where doping is not only legal but encouraged. It's the enhanced games. Some call it grotesque. Others say it's unleashing human potential. Either way, the podcast's Superhuman documented it all, embedded in the games and with the athletes for a full year. Within probably 10 days, I'd put on 10 pounds. I was having trouble stopping the muscle growth.

Listen to Superhuman on the IHard Radio app, Apple Podcasts, or wherever you get your podcasts. AllZone Media. Hello and welcome to Better Offline. I'm your surly yet lovable host, Ed Zitz. Today I'm going to kick off by reading something I wrote in March 2024 and talked about on the episode PKK. What if what we're seeing today isn't a glimpse of the future, but the new terms of the present? What if artificial intelligence isn't actually capable of doing much more

than what we're seeing today? And what if there's no clear timeline when it will be able to do more? What if this entire hype cycle has been built on hot air, goosed by a compliant media, ready and willing to take career embellishes at their word? Reading that back, Well, I think I might have been right, and that's kind of what I'm going to get at today. I don't want to scream mustard. I'm not going to get smug about it. But this is what we're getting into, today and in the next episode that will come out on Friday. Now, I'll be linking to some articles, so check the episode notes if you want to read them, but I'm going to get a lot into the spoken word. So I warned you in February that Generative AI has no killer apps and had no way of justifying its valuations.

I also warned you in March that Generative AI had already peaked, and I pleaded with the tech industry in April to consider. an eventuality where the jump between GPT4, which is the most current model, or GPT40, to GPT5, was not significant, in part due to a lack of training data, one of the more obvious things. I shared more concerns in July that the transformer-based architecture underpinning generative AI, things like Chad GPT, was a dead end, and that there were really not many ways we'd progressed past the products we'd already seen back then, in part due to the limits of training data, like I mentioned, and the limits of the models that use said training data. I summarized the pale horses of the AI apocalypse, events, many that have now come to past,

I'm afraid, that would signify the end, well, being nigh, though it's not quite here yet and it's not obvious when it will be, but this can't last forever. But I also added that GPT5 would not change the game enough to matter, let alone add a new architecture to build future and more capable models or products of any kind. Now, throughout the things I've written and the things have spoken, have repeatedly made the point that, separate to any core value proposition, training data drought or unsustainable economics that I've gone over quite a lot. Generative AI is a dead end due to the limitations of a probabilistic model that hallucinates. Now, just to be clear with what that means, it's guessing what the next thing might be.

And it's quite good at it, but quite good is actually kind of shit. And hallucinations of, of course, are whether they authoritatively state things that aren't true. Like when chat GPT tells you something like, I don't know, there are two R's in strawberry. The hallucination problem is one that is nowhere closer to being solved. You may remember a few months ago when you had every tech executive, you had Tim Cook saying it's Satchin Adela Sondapeshai, we'll deal with the hallucination problem. It'll be all right. But I want to be clear, they have not solved it. They have not really mitigate it. And there's no fixing it, at least with the current technology. It's not going anywhere. And it makes all of this stuff kind of a non-starter for many business tasks. I have since March expressed great dismay about the credulousness of the media about this. And their weird, of this inevitable way in which generative AI will change society, despite the fact there's not really a meaningful product that might justify any of this bullshit, this environmentally destructive nonsense, led by a company that burns more than $5 billion a year in big tech firms that are

spending $200 billion on data centers for products that people don't want or even potentially use. And you're going to need context for everything I'm saying today. So it's worth going over how these models work and how they're trained. And I must be clear. the reason I'm repeating myself on so many levels here, that it's just really important for you to know how obvious the problems of generative AI have been since the beginning. It's really important. Let's go over how they work real quick. A transformer-based generative AI models such as GPT, which is the technology behind chat GPT, generates answers using inference, which means it draws conclusions based off of its training, which requires feeding it masses of training data, mostly text and images scrape from the

internet. And both of these processes require you to use high-end GPUs, graphics processing units, and lots of them, tens, hundreds of thousands of them, well, over 100,000. I'll get to that next episode. Now, the theory was and might still be, that the more training data and compute you throw at these models, the better they get. And I've hypothesized for a while that we'd have diminishing returns, both from running out of training data and based on the limitations of transformer-based models. And wouldn't you know it? I was bloody right. I'm not going to do many of these, but this one really, this one I'm right on. A few weeks ago, Bloomberg reported that OpenAI, Google and Anthropic are struggling to

build more advanced AI, and the Open AI's Orion model, otherwise known as GPT5, did not hit the company's desired performance, and that, and I quote again, Orion is so far not considered to be a bigger step up as it was from GPD3.5 to GPD4, its current model. You will be shocked to hear that the reason is that it's become increasingly difficult to find new untapped sources of high-quality human-made training data that can be used to build more advanced AI systems. Something that I said in March, I said it would happen in March and pissed off that people said I was a pessimist. Well, who's a pessimist now? Me, I guess. I don't know. But they also added one other thing, which is that they believe, and I quote, that the AGI bubble is bursting

a little bit, which is something I said in July. AgeI isn't coming out of this shit. Let's just be honest. And I also want to stop and stare really hard at one particular point, and I quote again from Bloomberg. These issues challenge the gospel that has taken hold in Silicon Valley in recent years, particularly since OpenAI released ChatGPT two years ago. Much of the tech industry is bet on so-called scaling laws that say more computing power, data, and larger models will inevitably pave the way for greater leaps forward in the power of AI. The only people taking this as gospel have been members of the media unwilling to ask the tough questions, and AI founders that don't know what the fuck they're talking about or they intend to mislead you.

Generative AI's products have effectively been trapped in amber for over a year. It's been blatantly obvious if you fucking use them. I'm pissed off, I shouldn't swear so much. There have been no meaningful industry-defining products out of this because, and I quote Darren S. Mowgli, the economist at MIT, back in May, more powerful models do not unlock new features or really change the experience, nor what you can build with transform-based models is really a worthwhile product. or, put another way, a slightly better white elephant is still a white elephant.

Despite the billions of dollars burned and thousands of glossy headlines, it's difficult to point to any truly important generative AI product. Even Apple intelligence, the only thing that Apple really had to add to the latest iPhone, it sucks. It's not useful. I can make a special emoji now. I now get summaries of my texts that are completely or vaguely incorrect or just summarize a giant meaningful paragraph into a, a blob of a sentence. It's so stupid. And just as a side question, what the hell is Apple going to put

in the next iPhone? I buy one of these every year. I'm a little pig, oink, coin, coin. But still, I don't even know why I'd upgrade again. The camera is already about as good as it's going to get. Anyway, there are people that use chat GPT, 200 million for them a week, allegedly, losing the company money with every prompt, by the way. But there's little to suggest that there's widespread adoption of actual generative AI software. The information reported in September that between 0.1% and 1% at the 440 million of Microsoft's business customers were willing to pay for its AI-powered copilot, and in late October, Microsoft claimed that it was on pace to make AI a $10 billion a year business, which sounds really good until you think about it for roughly 10 seconds.

First of all, Microsoft does not have an AI business unit, which means that this annual $10 billion or $2.5 billion a quarter revenue figure is split across providing cloud compute services on Azure, selling co-pilot to dumb people with Microsoft 365 subscriptions, selling GitHub copilot, and basically anything else with AI on it. Microsoft is cherry picking a number based on non-specific criteria and claiming it's a big deal when it's actually pretty pathetic considering that Microsoft's capital expenditures will likely hit over $60 billion in 2024, with no sign they're going to slow down. Note that sticky word revenue, not profit. Those are two very different things. How much is Microsoft

spending to make $10 billion a year? OpenAI currently spends $2.35 to make a dollar, and Microsoft CFO Amyhood said that Open AI would cut into Microsoft's profits in their last earnings call, losing it a remarkable $1.5 billion, mainly because of the expected loss from a company that has only ever lost money. Now, a year ago in October 2023, the Wall Street Journal reported that Microsoft was losing an average of $20 per user per month on GitHub copilot, a product with over a million users. If this is true, by the way, this suggests losses of at least $200 million a year. They have 1.8 million users allegedly.

This is based on documents I've reviewed. It's not great either way. $200 million is a lot of money to lose. I would personally like to make $200 million rather than lose it. Don't ask me, though. Don't run Microsoft. Now, Microsoft is still yet to break out exactly how much generative AI is increasing revenue in the specific business units they have.

Generally, if a company's doing well at something, they take great pains to make that clear. Instead, Microsoft chose in August to revamp its reporting structure to give better visibility into cloud consumption revenue, which is something you do if you say, anticipate you're going to have your worst day of trading in years after your next earnings, as Microsoft did in October. It's all very good. It's all going well. Now, I must be clear that every single one of these investments, you know,

investments and products has been hyped with the whisper that they would get exponentially better over time, and that eventually the $200 billion in capital expenditures would spit out this remarkable productivity improvement, this crazy new product that would change our lives, fascinating new things that consumers and enterprise would buy in droves and talk about how much they loved. Instead, big tech has found itself peddling increasingly more expensive iterations of near identical large language models and shitty products attached to them, a direct result of all of them having to use the same training data, which they're now running out of. But if you're running out of stuff and you can't find stuff to buy, I really recommend the following advertisement. I'm sure it will

totally gel with my beliefs, the things I'm talking about right now, won't be embarrassing at all. Another podcast from some SNL late-night comedy guide? Not quite. Unhumor me with Robert Smigel and friends. Me and hilarious guests from Jim Gaffigan to Bob Odenkirk to David Letterman help make you funnier. This week, my guest, SNL's Mikey Day and head writer, Streeter Seidel, help an acapella band with their between songs banter.

There's the worst singer in the group. The worst? Yeah. Me. Is there anything to the idea that because you're from Harvard, you only got in because your parents made a huge donation. The group. The yard birds, right?

That's the name. The Harvard yard, but they're open to change. Do you have a name suggestion? We're open. Since you guys are middle. A one erection. Listen to humor me with Robert Smigel and Friends on the IHeart Radio app, Apple Podcasts, or wherever you get your podcast. Humor me.

I need some jokes to make me seem funny. Run a business and not thinking about podcasting, think again. More Americans listen to podcasts than ads supported streaming music from Spotify and Pandora. And as the number one podcaster, IHearts twice as large as the next two. combined. So whatever your customers listen to, they'll hear your message. Plus, only IHeart can extend your message to audiences across broadcast radio. Think podcasting can help your business. Think IHeart. Streaming, radio, and podcasting. Let us show you at IHeartadvertising.com. That's iHeartadvertising.com. Life throws hurdles big and small. The question is, how do you conquer them?

On Hurtle with Emily Abadi, we sit down with the most inspiring women in sports and wellness. professional athletes, coaches, and Olympic champions to talk about the challenges that shaped them and the mindset that keeps them going. From the WNBA standout Kate Martin and rising hockey star Layla Edwards. If a boy can do it, I don't see why a girl can't. Like, I've never understood that. Like, it didn't make sense in my brain. It's hard to be in spaces that no one looks like you, but don't ever feel like you don't

belong. Don't let that be the reason you don't do it. An Olympic champs Gabby Thomas and Katie Ledecki. The ability to show a gold medal to someone and have their face light. up and smile. That means the world to me. And that's what motivates me to win more gold medals. At our level, at this scale, like being able to fail in front of the entire world. Like, I can do anything. I can, like, I can do anything. Because resilience isn't just about winning. It's about showing up, even when it's hard. Listen to Hurtle with Emily Abadi on the IHeartRadio app, Apple

podcasts, or wherever you get your podcasts. Presented by Capital One, founding partner of IHeart Women's Sports. Imagine an Olympics where doping is not only legal but encouraged. It's the enhanced games. Some call it grotesque. Others say it's unleashing human potential. Either way, the podcast's Superhuman documented it all, embedded in the games and with the athletes for a full year.

Within probably 10 days, I'd put on 10 pounds. I was having trouble stopping the muscle growth. Listen to Superhuman on the I-Hard Radio app, Apple Podcasts, or wherever you get your podcasts. is a win. A win is a win. I don't care what you're saying. Yep, that's me, Clifford Taylor the 4th. You might have seen the skits, the reactions, my journey from basketball to college football, or my career in sports media. Well, somewhere along the way, this platform became bigger than I ever imagined. And now I'm bringing all of that excitement to my brand new podcast,

The Clifford Show. This is a place for raw, unfiltered conversations with some of your favorite athletes, creators, and voices that not only deserve to be heard, but celebrated. One week, I'll take you behind the scenes of the biggest moments in sports and entertainment, and the next we'll talk about life, mental health, purpose, and even music. The Clifford Show isn't just a podcast, it's a space for honest conversations, stories that don't always get told, and for people who are chasing something bigger. So if you've ever supported me, or you're just chasing down a dream, this is right where you need to be. Listen to the Clifford show on the IHeart Radio app, Apple podcast, or wherever you get your podcast. And for more behind the scenes, follow at Clifford

and a TikTok podcast network on TikTok. And we're back. Now, there's another assumption that people have about these so-called scaling laws that's been that by simply building bigger data centers with even bigger, more powerful GPUs, the expensive power-hungry graphics processing units that use to both train and run these models,

and throwing as much training data at them as possible, they would simply start doing new things. They'd have new capabilities, despite there being little proof that they would do so in any way, shape, or form. Microsoft, meta, Amazon and Google have all burn billions on the assumption that doing so would create something, you know, a thing, a good thing, like a human-level artificial general intelligence or a product that made more money than it cost, the people liked. It's become kind of obvious that that isn't going to happen. As we speak, members of the media who should know

better are already desperately trying to prove that this is not a problem. The information, in the similar story to Bloomberg's, attempted to put lipstick on the pig of generative AI, framing the lack of meaningful progress of GPT5ers, fine, because Open AI can now combine its GPT5 model with its O1 reasoning model, which is the one that can't count the number of ours and the strawberry, by the way, which will then do something, something good, something's going to happen. Like, Sam Altman said it could write a lot more very difficult code. You know, Sam Altman, the career liar who intimated the GPT-5 may function like a virtual brain in May. Like, these people are liars, they're liars, they're lying to you, they were lying then,

they're lying now. Now, I couldn't possibly leave out Chief Valley cheerleader Casey Newton, who wrote on Platformer a few weeks ago that diminishing returns in training models may not matter as much as you would guess, with his evidence being that Anthropic, who he also claims has not been prone to hyperbole, do not think the scaling laws are ending. Now, the original scaling laws paper, partly written by Daria Amadaisi, of Anthropic, important to know. And to be clear, in a 14,000 word op-ed

that Casey Newton for no reason wrote two pieces about, Anthropic CEO, Dario, he said that, and I quote, AI-accelerated neuroscience is likely to vastly improve treatments for, or even cure most mental illness, which is the kind of hyperbole that should have you tarred and feathered and put in a jail.

I'm not seriously saying you put him in jail, but why are we trusting these people? Why are we listening to them? Why are we treating them as if they're telling the truth, or even that they know what's going on? But let's summarize. The main technology behind the entire, and I say this in quotation marks, by the way, artificial intelligence boom is generative AI, transform-based models like OpenAI's GPT4 and soon GPT-5, and said technology has peaked, with diminishing returns from the only ways of making them better, feeding them training data

and throwing tons of computer them, suggesting that we may have, as I said before, reached PGT. AI. Generative AI is incredibly unprofitable. Open AI, the biggest player in the industry, is on course to lose more than $5 billion this year, with competitor Anthropic, which also makes its own transformer-based model, Claude, on course to lose more than $2.7 billion this year. They just raised another $4 billion. Every single big tech company has thrown billions of dollars, as much as $75 billion in Amazon's case in 2024 alone, are building the data centers and acquiring the GPUs to populate said data centers specifically so they can train their models and people's models, or serve customers that would integrate generative AI into their businesses,

something that does not appear to be happening at scale. And these investments could theoretically be used for other products, but these days aren't as a heavily focused on generative AI. Business Insider reports that Microsoft intends to amass 1.8 million GPUs by the end of this year, costing it tens of billions of dollars. Worse still, many of these companies integrating generative AI do so by connecting to models made by either OpenAI or Anthropic, both of whom are running unprofitable businesses, and likely charging nowhere near enough to cover their costs.

As I've said before in my article, the subprime AI crisis, in the event that these companies start charging what they actually need to, their real costs, I hypothesize that it will multiply the cost of their customers to the point that they can't afford to run their businesses, or at the very least we'll have to remove or scale back generative AI functionality in their products. It's just, it's such a waste. The entire tech industry has become oriented around this dead-end technology that requires burning billions and billions of dollars

to provide inessential products that cost them more money to serve than anybody ever would pay. Their big strategy has been to throw more money at the problem until one of these transformer-based models created something useful, despite the fact that every iteration of GPT and other models has been well iterative? And it's weird, you think at some point that goes, shit, do we actually have the ability to build products with this?

what are the products? Maybe we should work out the products first before we throw all the capex at it. But wait, no, over yonder, I couldn't possibly not do this because the other big tech company that also has no ideas, they're doing this. And if I don't do this, my investors are going to be angry at me. And then what will I do? Oh no. Oh no. What could I possibly do if the investors? I don't fucking know. That's your problem. Why waste this much money? it's just there's never been any proof other than these benchmarks that are really easy to game and also only show just this vague power of these models. It's been obvious that GPT or other models wouldn't become conscious,

that they're not going to do more than they do today or three months ago or even a year ago. Hesitate to give Gary Marcus credit, but in 2023 he was saying this, if not earlier. Many people have those well. And it's just really, really, really, really frustrating. Better Offline isn't even a year old, but when we put out our PKI episode, I got so much flak. I got so much shit for being a hater that I didn't really understand things, that my fly was opened in my Instagram picture, that I didn't get it, and that in mere months I would be proven wrong. Well, here we are, how wrong am I now? What happens next exactly? Where do all these hundreds of billions

of dollars go? What happens to Open AI when it collapses? What does Microsoft do with all of these GPUs? Because you can't just move them into other shit, you know? From what I hear, they don't really have a plan. And that's the scariest thing. Because what happens to a stock market that's dependent on big tech companies for growth when the big tech companies can't work out a way to grow anymore? And in fact, their big path to trying to grow more was to burn a shit ton of money on things that people hate that destroy our environment.

I know, I know I'm angry. I know. I should calm down. I should. But as I said in the ROT Society, this money could go elsewhere. more things could be done, it would enter a fallow period of tech, but we don't just have to burn all this money. We don't have to do that. Why not make the products you have already better?

Because stapling generative AI on them, I think it makes them worse. But there are more problems ahead. There are problems around the infrastructure. And in the next episode, I'm going to break down these worrying problems. And I'm going to kind of tell you what happens next as best I can. I really appreciate your faith in me and there are many people who also contacted me and said No, you're bang on

keep going. I'm glad they did. I'm very grateful for you audience. I love you all, much like he said in the menu. Thank you for listening to Better Offline. The editor and composer of the Better Offline theme song is Mattersowski. You can check out more of his music and audio projects at Mattisowski.com. M-A-T-O-S-K-I-com.

You can email me at E-S-E-S-E-S-K-I-com. at betteroffline.com or visit betteroffline.com to find more podcast links and of course my newsletter. I also really recommend you go to chat. Where's Your Ed?at to visit the Discord and go to R slash Better Offline to check out our Reddit. Thank you so much for listening. Better Offline is a production of Cool Zone Media. For more from Cool Zone Media, visit our website, coolzonemedia.com or check us out on the IHeartRadio app, Apple Podcasts, or wherever you get your podcast. Another podcast from some SNL late-night comedy guy, not quite.

Unhumor me with Robert Smygel and Friends. Me and hilarious guests from Bob Odenkirk to David Letterman help make you funnier. This week, my guest, SNL's Mikey Day and head writer Streeter Seidel help an a cappella band with their between songs banter. Where does your group perform? We do some retirement homes. Those people are starving for banter. Listen to humor me with Robert Smigel and Friends on the I-Heart Radio app,

Apple Podcasts, or wherever you get your performance. Podcasts. Life is full of hurdles. So how do you keep going? On Hurtle with Emily Abadi, we're talking with the most inspiring women in sports and wellness from professional athletes, coaches, and Olympic champions about the challenges that shape them and the mindset that keeps them moving forward. At our level, at this scale, being able to fail in front of the entire world, like, I can do

anything. I can do anything. Listen to Hurtle with Emily Abadi on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. Presented by Capital One, founding partner of IHeart Women's Sports. I'm Michelle McPhee, and I've been unraveling the strangest criminal alliance I've ever reported on. A Mormon polygamist and an Armenian businessman. Multi-million dollar house, Ferraris and Lamborghinis, private jets, a billion dollar fraud.

But how long can this alliance last? Tell me what you know. Is somebody coming after me? Listen to Kingdom of Fraud on the IHeart Radio app, Apple Podcasts, or wherever you get your podcasts. Imagine an Olympics where doping is not only legal but encouraged. It's the enhanced games. Some call it grotesque. Others say it's unleashing human potential.

Either way, the podcast's Superhuman documented it all, embedded in the games and with the athletes for a full year. Within probably 10 days, I'd put on 10 pounds. I was having trouble stopping the muscle growth. Listen to Superhuman on the I-Hard radio app, Apple Podcasts, or wherever you get your podcasts. A win is a win. A win is a win. I don't care what I'm saying.

Yep, that's me, Cliver Taylor the 4th. You might have seen the skits, my basketball and college football journey, or my career in sports media. Well, now I'm bringing all of that excitement to my brand new podcast, The Cliver Show. This is a place for raw, unfiltered conversations with athletes, creators and voices that not only deserve to be heard, but celebrated. So let's get to it. Listen to The Clifford Show on the IHeart Radio app, Apple Podcasts, or wherever you get your

podcast. And for more behind the scenes, follow at Clifford and at TikTok podcast network on TikTok. This is an IHeart podcast. Guaranteed human.