Instant Genius - The hidden forces driving the AI bubble

Episode Date: November 14, 2025

This episode was recorded on November 6th 2025. Artificial intelligence has been the movement of the moment in recent years. Since it burst to prominence in 2022, hundreds of millions of people have started using AI systems every day – for everything from writing essays to coding software, generating art and planning their lives. But with soaring valuations, constant hype, and growing concerns about how much these systems really understand, some experts and investors are starting to ask whether we’re heading for an AI bubble. One of them is Gary Marcus, a scientist, author, entrepreneur, and one of the AI industry’s most outspoken critics. Over the years, Gary has become a leading voice in debates about artificial intelligence, and many of his predictions about where the technology is heading have proven remarkably prescient. In this episode, Gary cuts through the noise and digs into hype vs reality, what current systems still can’t, and may never be able to do, and what a more reliable path forward might look like. Learn more about your ad choices. Visit podcastchoices.com/adchoices

This podcast is sponsored by name, audio and focal. Streaming has made music more accessible than ever, but true listening is about more than ease. It's about quality. British audio experts name audio, alongside French acoustic specialist focal, combine handcrafted tradition with cutting-edge innovation and high-end materials,

delivering digital precision with analogue warmth. So you can experience exceptional sound at home. Music just as the artist intended. Visit name audio.com. to learn more. Hello, and welcome to Instant Genius, the Bitesize Masterclass in podcast form. Each week you'll hear from world-leading scientists and experts

talking about the most fascinating ideas in science and technology today. I'm Tom Howard, trends editor at BBC Science Focus. Artificial intelligence has been the movement of the moment in recent years. Since it burst to prominence in 2022, hundreds of millions of people have started using AI systems every day, for everything from writing essays to coding social, software, generating art and planning their lives. But with soaring valuations, constant hype, and growing concerns about how much these systems really understand, some experts and investors

are starting to ask whether we're heading for an AI bubble. One of them is Gary Marcus, a scientist, author, entrepreneur, and one of the AI industry's most outspoken critics. Over the years, Gary has become a leading voice in debates about artificial intelligence. Many of his predictions about where the technology is heading have proven remarkably prescient. In this episode, Gary cuts through the noise and digs into the hype versus reality, what current systems still can't and may never be able to do, and what a more reliable path forward might look like. So Gary, welcome to Instant Genius.

Happy to be here. So by this point, most of our listeners will be familiar with systems like chat GPT, Claude, GM and I, but I imagine that most of them won't really understand what's going on under the hood. So maybe if we could start there, what are these systems actually doing to produce the outputs that all of us have become so familiar with? Well, first, I should say that nobody understands them completely. We know how to build them, but we don't know how to predict exactly what they'll do. And with that said, the fundamental thing that they're doing is basically predicting next words in a sentence. They're sort of like autocomplete in your phone.

You type something, it guesses what's going to come next. Basically, these are really sophisticated devices for making that prediction. looking at context, not just from the last few words like your phone might use, or maybe used to use, but using everything maybe in all the conversations that you've had going back some distance. So they're trying to say, what would a person say in this context? Now, they're not actually doing that intentionally or consciously. They're really just big statistical machines. They use a lot of particular kind of math we call matrix arithmetic.

They're sort of vaguely brain-like. people call them brain inspired. They're not really like brains except the fact that they do all of the statistics in parallel in the way that you have a bunch of neurons doing things in parallel. But what they are doing is predicting what might come next in the context. You can also do variations on that theme. So what pixels might come next in a video? So because of that way that they work, what does this make them good at doing? And what does it make them bad at doing? That's also a surprisingly complicated question. So it obviously makes makes them good at auto-complete.

And a lot of their strengths come from that. So probably the best application of these systems is for coding. And a lot of coding is sort of boilerplate. Kind of write the same things over and over again. Depends on what coding you're doing. But there's often a strong element of that. And so programmers use them essentially as a form of auto-complete, predicting, given this context, what is the thing that I should say?

What is the thing that I should write? They're pretty good at things like brainstorming. So you say, give me 10 ideas for this and that, and it might give you three or four good ideas. There you have humans in the loop, in both the coding context and the brainstorming context. The thing about these systems is they're really just mimics. They don't have a deep understanding of what they're talking about.

So if you have a human in the loop and they say something stupid, the human can just filter it out. The place that they're really bad at is doing things that are automated with no human in the loop and things that are unusual, because what they're doing is matching patterns that they've seen before. If something is different from those patterns, they might break down. And they're really bad where stakes are high.

So, for example, medical situations, sometimes they get it and sometimes they don't, and there can be a high cost of error. And so this gives rise to things that people call hallucinations. Could you maybe explain a little bit what we mean when we're talking about AI systems hallucinating? So it's not inherent in an AI system in general that it should hallucinate. But in these neural network systems, the way that they're built basically gives rise to overgeneralization errors that a lot of us call hallucinations. And what that means is a system

says something that just has no basis in the data that it's seen before. I wrote a whole article about one example that I'll walk you through, which is my friend Harry Shearer is an actor, a voiceover actor. He does voices from Mr. Burns on The Simpsons and some other characters. He was the bass player in spinal tap. So he's reasonably well known. And it's very easy to look up the fact that he was born in Los Angeles. You can find that in Wikipedia. because he's a movie star, you can find it in IMDB and Rotten Tomatoes, and he's done lots of interviews. It's easily found.

But one day, someone sent him a biography of him that he sent to me, claiming that he was British. Well, he's not British. It's just that there are some other people in the category that he belongs to, voiceover actors, comedians, et cetera, that happened to be British. And what these systems do is instead of keeping track of individual facts, the way that a classical AI system might do, and we can talk more about what that might mean, they break everything into little bits of information, and they learn the correlations

between those bits of information, and they tend to get lost. I actually pointed out this problem, not using the word hallucination, but overgenitalization in 2001. And every year since then, often many times a year, I've heard people say, well, if you just add more data, it will solve that problem. And the fact is, adding data has not solved that problem. People have been talking about the scaling hypothesis, that you kind of predict how well the systems do.

as a function of how much data there is. And that's not really working that well anymore. It has not solved hallucinations or more broadly, the kind of stupid errors that I think most of us have seen these systems do, right? Sometimes they're brilliant and sometimes they're stupid, incredibly stupid, as if they don't know what's going on. It's because they don't know what's going on.

Sometimes they get it right because the statistics guide them in the right place, and sometimes they don't. It's very unpredictable. If you say write a biography of me, it probably gets some of it right and some of it wrong. you don't really know which part in advance until you actually see it. And what's the point of using a system if you don't know if it's going to be right or wrong? It's kind of crazy.

People used to have calculators. That's a form of artificial intelligence, depending on how you define it. Very narrow piece of artificial intelligence. Calculators, we trust. Nobody would use a calculator that's right 80% of the time. But we're in this era where we're using these large language models that are often 80% correct in any given domain, and essentially never reliable. Yeah, so you mentioned scale.

there, and that has delivered what we have seen in terms of the growth in the abilities of these systems in recent years. What do we mean by scaling? How has it delivered those improvements? And I guess why is that system of scaling kind of flawed? So scaling basically means if you add more data to these systems, they'll get better. And there was a hypothesis that we could predict exactly how much better they would get. And for a little while, that was actually true. But we're reaching a point of diminishing return. So there are many curves that you can draw that start out really great and then they fade off

or start out very impressive and fade away. And that's what really we're seeing with scaling. So, you know, if I told you that, you know, my baby was nine pounds at birth and 18 months a month later, that doesn't mean it's going to keep doubling and becoming a trillion pound baby by the time it goes off to college, right? So, you know, not everything follows the same curve for all time that it did initially. But almost this entire industry was based on the guess that it would go perpetually. In fact, in 2022, I said, you probably shouldn't count on that.

You're probably going to reach a point of diminishing returns. And people were very mad at me. Like Elon Musk took a crack at me. Sam Altman took a crack at me, Jan Lacoon, all these people. The reality is we actually have, for that technique that we were using then, seen diminishing return. So things did improve for a while. You know, every time you add something new to the database on which they're trying, they'll at least do better on that particular example.

But there's another problem that I would call the outlier problem, which is if something's not in the database at all, these systems don't do very well. And adding more data has not really solved that problem. There's this kind of infinite periphery around the center of things that the systems haven't been exposed to. And so it turns out that just adding more data doesn't. doesn't help you infinitely. There are other problems too. So it was a really neat trick to go from a hundredth of a percent of the internet to one percent of the internet to basically like 98% of the internet. Systems kept getting better and getting impressively better. But now everybody's

been using essentially 100% of the internet for the last couple years and they're not getting the same gains anymore. There isn't like 10 more internets to draw on. This is why these companies are doing crazy things like transcribing YouTube videos in a desperate need to try to get more data. And there may be just not enough data to pull off what they want. And then I think fundamentally, the whole idea of using a black box where you just pour data in, like you would pour cranberries into a grinder and expect cognition to come out of it, I think, is just a bad idea to start with. And we're starting to see the limits on that in terms of the things that are not improving

even as we add more data. Why do you think people were so dogmatic about these scaling laws, you know, because it seems like, yeah, people really believed it. Yeah, I mean, there's two theories about that. One is that some people probably really did believe it. They were naive. They just looked at the curves on paper and didn't really think very deeply about it. So, you know, people who don't know a lot of biology or psychology just sort of look at the curve can make naive extrapolations. And there was some of that.

And some of it was willful. There's a lot of money at stake. And what you have to understand is that if you can tell a story that you can convince your investors make some sense and you're a venture capitalist, the bigger the story you tell, the more money you make because you make 2% of whatever you invest. The scaling stuff licensed people to go to the limited partners, things like pension funds, and say, we have this plausible way where we're going to make you a whole bunch of money. Can you invest X dollars?

And when it's, you know, a million dollars, the venture capitalist doesn't make that much with 2% of a million dollars. But when it's a billion dollars, the venture capitalist starts to make a lot of money. And so venture capitalists love this story and kept telling it and telling it and promoting it and using their power and influence to promote it. And the story just getting bigger and bigger. What we're seeing right now, I think, is that story is losing steam. And it started to lose steam at a very particular moment, I think, which is in August 7th, GPT5 came out. And Sam Altman kept pitching GPT-5 as AGI, artificial general intelligence. And people had already seen that GPT-4 was not the magic they had been promised.

Everybody tried it. It was fun to play with. But when businesses actually used it, it didn't necessarily give them a return on investment. MIT had this study recently showing 95% of companies were not getting return on investment. So what everybody thought, though, was, okay, this scaling is going to put us out of this temporary misery. We're going to get these great models. it's going to be AGI, and Altman kept selling that story over and over again. And I kept saying, no, it's going to be late and it's not going to be as good as you expected.

And people thought, you know, called me literally the Grinch of AI. But then August 7th came around and the model was not that good. You know, Sam came on onto the live stream and said, this is going to be, you know, as good as a PhD in every field. And then people went home and tried it. And within hours, they realized that it was not what he was promising. And that has really caused, I think, a shift. You know, as we're recording this, Nvidia is down, I think, 9% in the last couple days.

You never know how the market's going to react. But you're starting to see a lot of people talk about whether AI is a bubble. I first said that in August 2023. John Thornhill of the FT picked it up at the time. But not that many people were talking about a bubble at that point. But now you see new stories about it every day. Like earlier this week, Michael Burry, who's famous for the big short, revealed that he had a billion dollar short on NVIDIA and maybe a couple other companies. And that's probably

making people a little bit nervous. There are many other things. Just yesterday, Open AI came out and said they want to have loan guarantees for all this money they're putting on infrastructure. Theoretically, they're kind of like a capitalist company, right? But they don't want to stand on their own two feet. They want the governments to back them up as they make these crazy infrastructure bets. Open AI is basically now encumbered for over a trillion dollars that they've promised on infrastructure, and they're only making, well, they're losing $3 billion a month and their revenue, even if you don't look at the cost, it's only like $10 billion or $13 billion a year or something like that. It boggles the mind that they think they can pull this off.

And what we learned yesterday is they don't think they can pull this off. That's why they're asking for the loan guarantees. Well, that's, I mean, anybody who's rational should look at this and say, if Open AI is talking about getting loan guarantees, they don't think they're going to make their numbers. And they're not. are crazy. And a large part of the economy is now based on those numbers, right? I mean, most of the growth in U.S. GDP is from data centers, but the data centers are not making anybody money. And so it really doesn't make a lot of sense. And I think people have woken up to that, starting, you know, with GPT5, which was disappointing. And of course, Lama 4 and GROC 4 were also

disappointing. So it wasn't even the first data point. But suddenly people saw that data point, started to be disappointed, and now, you know, we're a few months later, and it's really, I think, spread. A metaphor that I like is, if you ever saw the old Bug Spunny cartoon, Wiley Coyote would run off the edge of a cliff, and he wouldn't fall until he looked down. And any kind of bubble is like that. Like, it's fundamentally irrational, but it doesn't fall apart until people notice. Think about the Dutch tulips in the 1630s. Like, a lot of people thought it's crazy that a tulip should cost as much as a house, but nobody knew when it would end, even if it was like obviously crazy. And it ends at a psychological moment. Someone's like, I'm not paying that, and it spreads. And I think we're

close to that moment. It's very hard to predict the timing for when a bubble would pop, but we have many more people saying, hey, are we over the cliff? And in some sense, it's a self-fulfilling prophecy. You know, if the financial basics aren't there, the fundamentals aren't there, and I think they're not. And enough people notice, then, you know, you can have a big crash. I think I've heard you say that, you know, GPT5 should have maybe been called GPT 4.6 or something. Do you think that with this current infrastructure of models that we have, we can ever reach what we thought should be a GPT5 level model? I mean, it depends on how you count on it, of course. I actually placed, you know, or offered a variety of public bets on this kind of thing.

And people will be like, what are the criteria. What I said at the time was basically, it's up to Sam to decide what he wants, but he has to put something out there that will satisfy people. So in fact, GPT 4 and a half was something called Project Orion, and that was supposed to be GPT5. They realized it was disappointing, and so they called it GPT 4 and a half. It was like the expectation was that it was going to be this quantum leap much better than any system that it proceeded, and it wasn't, so they didn't call it that. It's an interesting question why when the system that we now called GPT5 came out, they decided

to call it that anyway. Some people think it was an economic decision. Because it obviously was not over the bar that they himself had placed. Some people think it's because GPT 4 and a half was so expensive to run, they wanted to have a new model and kind of publicize it that would be cheaper to run. I don't really know the inside story. But your question is, you know, when are we going to get that quantum leap in a way? And I'll note that just a few days ago, maybe it was 10 days ago, I think it was Altman. It was someone at OpenEI, I said we're moving, I think it was in a conversation with Altman.

Someone else, I think Jacob, maybe chief science officer, said, you know, we're moving to a different system. So we're not going to call GPT6 by the same kind of standard. So they're already kind of weaseling out of having any commitment to how they're going to name these things. And they've been weaseling out since GPT4, basically, with a whole bunch of different things. And the real question is, are we going to see a model that is so much better that is going to solve hallucinations, reasoning errors, and so forth? I don't think that the paradigm of these transformer systems based on lots of data is really going to get us to the magical moment that people are expecting. I do think some form of AI will get us there, but that we don't yet know how to build it. An analogy that I like to make is that we're sort of like in the age of alchemy.

Like people have the rough intuition that, you know, you can combine data and compute and that will help. But like, we don't really have a serious understanding, I think, of how to build a robust, intelligence that we could actually trust. And I think we need new discoveries for that. And the tragedy of this era is that we're spending so much money on one bet, that one bet, trillions of dollars, on the one bet is that scaling, adding more data, and adding more compute will bring us artificial general intelligence.

I think there's actually lots of evidence against that bet at this point. But worse, it's a lack of intellectual diversification. So like, if you go to any financial planner, they're going to say we want you in some stocks, but we also want you in some bonds and some real estate. we want you to be in a bunch of different things. Can't place all your money in a single thing.

That's not safe. And that's what we have done with the world's economy at this point, is we made a single bet, which is that scaling will get us to artificial general intelligence, and we're putting all this money into infrastructure and so forth. We're not putting a lot of money in other bets. There are some companies like DeepMind that have put some into some other bets, but by and large, almost all of the research dollars,

and it's really research, has gone into one idea, that and one idea alone. I've never seen the field of AI so narrow as it's been in the last three years. And there's lots of dynamics behind why, but it's an incredibly narrow bet that's probably wrong. And until we put real money in alternatives, we're probably not going to do a whole lot better. One of the problems with how these research dollars are being spent, right, is that they're going into private companies. So we're not seeing, you know, publicly peer-reviewed papers all the time. Do you think that that is sort of a major problem is that we're losing a lot of knowledge of how

these systems that we already have work to learn about ways that they could be improved or merged with other systems? It is a problem. I mean, in some ways, I think that it wouldn't make that much difference if we had more transparency because it's just the wrong bet. But to the extent that we might understand where things are going, yeah, it's terrible. And I started griping about this in maybe 2019 on Twitter, about how the AI companies were publishing things or posting things without peer review that looked like peer review articles, and they were tricking people. And we entered this era where there's really not a lot of transparency. And so there are basic questions that really have a lot at stake right now.

They're not just even science questions now because there are questions that affect world policy. Like how well do these systems generalize beyond the data on which they're trained? And we can't answer that because we don't know what data they're trained on. And so much of the world economy is now resting on an assumption that these systems are intelligent. But part of intelligence is being able to generalize beyond the data that you've seen. And because we have all of these reports that are not actually peer reviewed by a scientist, and because the data themselves are not disclosed transparently and so forth, we can't answer their questions.

You know, we can make guesses. We can make educated guesses. Or here's another thing that's really at stake, right, is, are there, these things safe? So, you know, some people have written about, you know, so-called doomer scenarios. Could AI kill us all? Well, how can you intelligently answer that question if you don't know what's in the cutting-edge systems? You don't really know how they work. So one particular example is all these systems now have both a transformer, which is the kind of core to systems like chat GPT, but they also have reinforcement learning systems. And we don't have any transparency about how those work. And those

reinforcement learning systems are used for so-called guardrails to protect us from having the systems do bad things, like disclose how to make biological weapons to people that shouldn't be told how to make those weapons. And we don't know how those guardrails work. We don't know how thorough they are. We can sort of poke at them from the outside, but we don't know their internals. And it's, you know, maybe you don't want everybody to know those internals, but those internals are not really available to any kind of like scientific committee or government committee. So this is another example where the lack of transparency could be really, really dangerous. Yeah, I mean, I report on science all the time, and AI seems to be that one bit of it that

I'm not getting from sort of peer-review papers, and it's all to do with what these companies are doing. It's very much that. They have very powerful media operations, so they can hype these things. They can fool journalists who don't know better, and they'll put out some 50-page summary that doesn't actually give the details that has signed to. reviewer who care for. You know, I've had many different careers. I'm not easy to pigeonle, but one of those careers was as a scientist who wrote lots of reviews of academic articles.

And I'll look at these things and I'll be like, I can't believe they would put this thing out. They haven't done this. They haven't done that. We need this control group. We need this reveal. And none of it goes through peer review anymore. And if it were all kind of lab internal, they weren't doing anything in the external world with it, then you could just say, well, it's the company's research. But the fact is that society now read. on these things. They have made it society's problems because they release it so broadly. I mean, you know, something like

Chat TPT, if they make a change, that can affect hundreds of millions of people and they may do that without any kind of pre-flight check like we would expect for a medicine without any way of, you know, external observers auditing what they do and so forth. Study

and play. Come together on a Windows 11 PC. And for a limited time, college students get the best. of both worlds. Get the Unreal College deal, everything you need, to study and play with select Windows 11 PCs.

Eligible students get a year of Microsoft 365 premium, and a year of Xbox GamePass Ultimate with a custom color Xbox wireless controller. Learn more at Windows.com slash student offer. While supplies last, ends June 30th, turns at AKA.m.S. slash college PC. You said this place was steps from the water. We just haven't found the steps yet.

How much did we save? Enough. Enough to get lost. Or you could book a stay with Hilton. Welcome to your oceanfront room. Just steps from the water. The Hilton sale is on now. Book on Hilton.com or the Hilton app

and save up to 20% to get the stay you expected. When you want savings, not surprises. It matters where you stay. Hilton for the stay. This podcast is sponsored by name, audio and trocal. With over 100 years of combined expertise, name and focal have been bringing music to listeners just as the artist intended. Since day one, this mantra has shaped every innovation in hi-fi design, technology and acoustic engineering,

balancing craftsmanship and tradition with pioneering thinking. Name audio pushes cutting-edge technology to ensure digital precision whilst sustaining Pratt, pace, rhythm and timing, the elusive quality that makes music feel alive and gives it a emotional texture. Today, in partnership with French acoustic specialist focal, name audio creates systems that deliver exceptional sound and unforgettable listening experiences at home. Try it for yourself at a focal powered by name boutique. Visit focal powered by name.com for more information. So what about so-called agentic AI? You know, that's kind of packaged up as something that can deliver these massive productivity games for your company. Is that just kind of a

the same thing, a marketing ploy, is something else. You remember Gandhi's line about Western civilization? Well, it would be a very good idea. Atantic AI, like, it will be a very good idea at some point, right? It would be fantastic when I can tell an AI system to do anything that I would ask a personal assistant to do and have it on call 24-7. That'll be terrific. But right now, it just doesn't work that well.

What it does is it gives you a kind of simulacrum, is maybe the word, a facsimile or a preview, of what agents really would be like. But what they do now is they tell you kind of what you want to hear. So there was an earlier version of this a year or two ago that everybody got super excited about, I guess it was two years ago, called Auto GPD. And that was basically a bunch of large language models

calling other large language models. And people are like, wow, I can use it to do all my finances and whatever. And they quickly discovered that it would say stuff like, I'm on it because that was contextually appropriate to the statistics of the words and then I actually do the thing

that they said, I'm on it. So you'd be like, you know, buy me 5,000 shares of Nvidia, and it would say I'm on it, but it didn't even have your passwords. It wouldn't actually place the trade, and you'd think that it had done it, right? You don't want an agent that can do that. So at the beginning of this year, everybody was hyping up agents. And I published a piece called, I think, 25 predictions about 2025. And I said, everybody's saying this is going to be your of agents, but it's not going to really be a year of agents.

And what's happened is every company has rolled out something that's kind of agent-like, but people aren't that excited about them when they use them because they don't work that reliably. You know, they give you this vision of the future when it'll be really grand when, you know, the systems can take care of these things. But the reliability is not there. The reliability is not there because the basis of these things is large language models. And those are fundamentally unreliable technologies. You don't want to build agents out of your least reliable technology.

I mean, build them out of like spreadsheets or something that actually work if they're debug properly. Like, it's a little bit, it's somewhere between like wildly overambitious and crazy, because it's just crazy to expect that this technology is going to be able to work that well. I mean, if you know the system's going to hallucinate when you write your biography, why do you think it's going to be okay to have it run your portfolio? Like, it doesn't make sense. Or book a holiday or something. You might end up in a different country.

I mean, it's amazing how hard even shopping is for these systems. Like, they'll shop from some websites and then someone else has written their JavaScript differently, and they won't even be able to figure out, these agents won't even be able to figure out the prices because someone used a little bit of unorthodox HTML. They don't understand the stuff that they're manipulating. And so, they'll work in the best case context where they have a lot of data and so forth, but then something slightly different and they don't work and they don't give you, you know, they don't give you a clue that they're not working, a separate problem with these systems and why, another reason why they're terrible as agents is they don't have a real

measure of confidence. So everything that Chat TPT tells you, It says authoritatively. I don't know if we can say this on air here, but I told 60 minutes that the output is what I call authoritative bullshit. You know, it all comes out in this authoritative voice with full confidence. What you really want is for your agent to say, well, I tried to place this trade, but I didn't get the confirmation that I was hoping for. So maybe you want to check that yourself.

But that's not what these things do. Or you want the LM to say, you know, I'd love to help you with that, but that's actually outside my domain. They don't know how to do that. So if AI is a bubble, how big is this bubble? How is the market set up in a way that could make this particularly bad? So the metaphor that I've been using lately is blast radius. I think there's going to be an explosion here or an implosion.

I don't know how broadly it's going to affect things. And I'll sort of paint a range of scenarios, but we don't really know yet. And I will tell you that the thing I most feared actually is starting to look pretty plausible as of yesterday. So the least that could happen is a bunch of rich dudes who put in a bunch of money on this, lose their shirt on it. You know, Mark Zuckerberg loses $30 billion he put into this, goes nowhere. Like, nobody's really going to bleed for Mark Zuckerberg if he loses $30 billion. So, like that would be the minimal cases.

Some people like Zuckerberg, Andresen Horowitz, you know, they lose some money. Then people might just say, well, you know, serves them right for overhyping this and what's the big deal. But the next thing you have to realize is that the venture capitalists have been taking a lot of money from pension funds. And that means that the pension funds are probably also going to take a hit. If Nvidia takes a hit, which it might. In fact, it's been down 9% in the last week as I came on the show. That's bad because a lot of pension funds are in that, including not just the ones that invested as limited partners in VC funds, but just people buying index funds and so forth. So a lot of people might take a hit for their pension funds and so forth.

So that's the next bad thing that could happen. Another bad thing that could happen is a lot of banks may be involved in a lot of leverage in order to put the money in these investments in the first place. And if that's the case and not enough is publicly disclosed for us to really know what the fallout is going to be, you could have something like the 2008 liquidity crisis. And then you have government bailouts for that. And then the thing that I warned about in January is that these companies might, come along and say, well, we're losing money here, but you need us for China, so we need

to bailout. And basically that happened yesterday. The chief financial officer of OpenAI said, we need loan guarantees to build all that infrastructure. We need government loan guarantees. So who pays for that? Taxpayers do, right? So taxpayers might pay to bail out the banks. They might pay to bail out the pension funds, and they might pay these loan guarantees,

which are not quite a bailout, but effectively a bailout. a backstop is what they call. So you might have a situation where the taxpayers are really paying a lot of money to make all of this right. And I'm starting to find that increasingly plausible. And you could say, well, that's just Gary. He doesn't love large language models. But the companies themselves yesterday, it's two of them.

Invidia and Open A, I both basically laid the groundwork for governments propping them up because they're doing the numbers and they realize the numbers don't make sense. And that means we're all going to get stiff unless we fight it really, really hard. Why can't, you know, some of our listeners might be wondering, why can't these companies make money with what they have? So they have millions of users. Why is that not a profitable model? So there's a couple things there. First of all, they, I mean, Open AI has hundreds of millions of users, but not all of them pay for it. So most users, the vast majority, are free users. And those free users cost Open AI a lot of money. I mean, you know, the idea of like giving razors away as you can sell the blades or whatever.

you know, Open AI is giving away a lot of this stuff to build market share, but it costs a lot of money to run. It costs a lot of money to run because you need all these GPU chips. Fundamentally, large language models are not efficient. They're not efficient in the sense of how much computation they need to do to figure out, you know, even a simple question. So it might take them 10 seconds to figure out a simple arithmetic problem that a calculator could do in a billionth of a second with a lot less processor. So they're expensive to train, which is to say we put in all the input data and we teach the model to predict things of that.

And they're expensive to run, which is to say once we've trained them, we now test them or we give them some problem to solve. Both of these processes turn out to be enormously expensive. And they're vastly more expensive than almost anything else in the software business. So most things in software business, basically you can run the software almost for free. Like there's hardly any cost to running a web browser. I guess there's some cost to like streaming a video. But like most software does not cost very much to run.

But these systems, because they run on massive amounts of GPUs, actually turn out to be quite costly to run. So they're very costly to run. They're not making that much revenue. So some people are paying to subscribe, but not that many people are paying to subscribe. And so, I mean, you just look at the numbers. And, you know, somebody just did a calculation looking at Microsoft,

numbers to infer what's going on with Open AI, it looks like Open AI is losing, I think it was about $3 billion a month. That's a lot of money. And it's because they're subsidizing all your use. Every time you use Open AI software, it's basically costing them money. And then you have things like SORA, the video stuff is really inefficient, especially the longer the video, it actually does go up, I think, exponentially. Or there's some data that suggests it goes up exponentially. So, you know, a two-minute video would cost them in, I don't think they'll do that yet, but would cost them in a enormous amount of money. I think I saw somebody suggesting that maybe 15 second videos or costing them $60 each. I don't want to swear on the numbers, but something like that. It's pretty

expensive. And so you have lots of people doing it. They're trying to build market share. They're trying to get people addicted to it basically with a kind of TikTok competitor. That's costing them a lot of money. Those are two parts of the problem. The other part of the problem is businesses would pay a lot of money to replace their employees with AI. I mean, that's a brutal thing to say, but we know it's true. but the fact is the systems aren't reliable enough. And so that means that the businesses are not getting the value that they thought of out of it. And so, you know, there's some data, some businesses are starting to pull back a little bit. I think there's growing skepticism.

So if the product were perfect, it really would be worth a lot of money. But the product is not perfect. It's not going to be perfect. And so that's going to put an upper bound on how much it's worth into whom. So it's really worth some money to coders. But that's like a billion-dollar-year market or 10. billion market or whatever, it's not a trillion dollar market. And they're talking about spending trillions on infrastructure. So, you know, if you only make 10 billion and you spend 100 billion,

you have a problem. One of the things these companies do have an abundance, though now, is data from users. Do you think that that could be something that these companies start to leverage? And can it prevent the bubble from bursting just from that? Well, we'll see. I mean, I think they're getting pushed towards surveillance. And I started warning this actually on a BBC show about a year ago, I think it was the first time maybe I talked about it, 15 months ago. Because they can't make this stuff powerful enough to be so-called artificial general intelligence, they have to look elsewhere, right? They don't have the sophistication about artificial intelligence to make these things kind of magical, solve everything systems. But they are collecting an immense amount

of personal data, especially open AI, right? Because chat GPT has the market share and people use it like as a therapist, for example. And so they get all of this very, very, very important. And so they get all of this very personal data. And so I think what Open AI is trying to do, at least in part, I mean, they're trying several different things. But one of the things they're trying to do is, is to move more towards basically surveillance capitalism. I think they're definitely thinking about targeted ads and so forth. And they have all this personal data. And for the personal data, they don't have to kind of solve the grand problems of artificial intelligence. They just have to get people to type stuff in and they're doing that. And so, you know,

you look at, for example, they want to build hardware devices that I think monitor you 24-7, with a camera and a microphone, analyze everything you say, well, that's, that's surveillance. They may also sell it to governments. You know, I won't be at all surprised if one of their biggest businesses is that. They're also now turning to porn and they're turning to these, you know, AI slop TikTok videos. So they're trying anything they can because they know they have these insane numbers to meet. Like when you take on a trillion dollars in server time commitments, you need to find

some way to make money. And one of those ways is clearly going to be surveillance. Do you think that that could be enough to kind of cover their losses, though? Or is that still just not enough based on the amount of investment that we've piled in now? I don't think so. I don't know for sure. I'll just say it as an aside, by the way, I think one of the reasons Zuckerberg's probably putting so much money into AI is because he perceives the threat. His business model is surveillance capitalism, all the stuff he type in to Facebook and Instagram and so forth.

And so he sees, I think, rightly, or he should see, that Open AI is trying to horn in on that business of, you know, targeted ads. based on personal information. Time will tell whether those economics can work. It's obviously not working yet. If there's any big business for them, it's probably that. They'll make some on porn, I guess, which they call erotica.

But, you know, the numbers are so vast in terms of what they're doing on capital outlay that it seems hard. Like, I won't say it's absolutely impossible, but even that feels to me like a long shock, given the economics. And one other thing to point out is that, There's nothing that Open AI is doing that seems particularly unique.

So lots of other companies can do it. They're not as popular. Anthropic is starting to eat their lunch on the business side. It's mostly on the personal side, but someone could undercut them, right? We saw the so-called DeepSeek moment in January or so. When Deepseek was suddenly undercutting them on price a lot, and people started using it. And so unless Open AI comes up with a technical mode, they're always going to be vulnerable. I think they do the best job of marketing.

They have the biggest market share. That's their strength. But if they got undercut on price enough, then a lot of that can go away. So they have a chance there, but I don't know how big the chances. So you said obviously it's very difficult to predict sort of when a bubble will burst, but you are a man who likes to make predictions and bets. When would you say, you know, how long do you think we have, what would be, you know, the cutoff point?

That's the one prediction I'm reluctant to make. So we talk about the many things I've been willing to predict. One of them was that these companies would start to do this too big to fail maneuver and start looking for a backstop or something like that. The problem with predicting dates on financial stuff I have come to realize is that as the famous saying goes, the market can remain irrational for longer than you can remain solvent. So we don't really know. We can look at all the signs and are people waking up and I think they are, but the market can remain irrational. I think Tesla is irrationally overvalued, but I have thought that for five years. And, you know, either I'm wrong or, you know, a reckoning is coming, right?

So with Tesla, like, how much are they really worth now, given that BYD can produce cars probably cheaper? You know, a lot of it is now based on a belief that they're going to make a lot of money in robots, which I think doesn't make sense. But how long will it take people to realize that the promises that they're making around robots are not realistic? I don't know. You know, a lot of those kinds of predictions are really about market psychology.

When do people recognize that there is a problem? And I certainly see signs, even with the robots, but especially with the large language models that markets are waking up, or at least, you know, many individuals are waking up, like Michael Burry's billion dollar short is, you know, a big statement that at least somebody no longer has confidence in all of this. But, you know, the dynamics are hard. They're also hard to predict in the modern era because you have things like Twitter where there's armies of bots that can pump things up and, you know, look at GameStop, like clearly overvalued, but, you know, it went way up and then it went down and up and that. So predicting the kind of crowd psychology mixed in with the misinformation, somewhere which is generated by AI itself and so forth, it makes it hard to really like nail a day on that.

The kinds of things that I think I can nail time on is like, will we have artificial general intelligence by, 2027. I can say with absolute certainty, or nearly absolute certainty, know who we won't, because that's a technical question about where we are, what problems need to be solved, and so forth. So I feel really confident about that, whereas predicting exactly when a market might crashes so much about crowd psychology that I think it's very difficult to name exact time. Before people think, you know, you're all pessimistic and doom and gloom, you have provided sort of a route that you think could put us on a track to artificial general intelligence. You talk about sort of neurosymbolic AI.

Could you explain sort of what that is and how it might work better than the LLMs that we're working with at the moment? Sure. And let me just insert. I absolutely think a better artificial intelligence is possible. You know, I think current AI is morally and technically inadequate. The moral stuff is for governments and, you know, people, individuals to demand better. The technical side, I think, is fully solvable.

I don't know exactly how, right? logically speaking, humans run 20 watts, not 20 gigawatts, and are able to reason quite well, not in every circumstance, but we could go into the details of human flaws. I actually wrote a whole book about that. But in the limit, you know, trained humans can reason quite well, and they can reason in novel situations, et cetera. So I certainly think it's possible. I have always, you know, for 30 years, pushed towards what I call neurosymbolic AI. And I think that that's part of the answer, but not all the answer. So, like, I think there's so many problems to be solved that nobody actually knows all of them, even probably what all the problems are, and certainly not solutions to them.

But I would start with neuro-symbolic AI as a critical next step that people are finally starting to take. So the term comes from a combination of two disciplines that have historically been into odds with each other, or approaches, I should say. One of those approaches is neural networks, which is the stuff that's popular right now, most of what we've been talking about, chatbot, chat GPT, et cetera. And the other is classical AI, which just looks like computer programming. It's called symbol manipulation,

because you have symbols that stand for things, like in algebra or something like that. You know, let x equal y plus two. Y is a variable. You substitute something into that variable and you calculate what X is. So classical computer program is almost entirely made up with stuff like that.

And neural networks don't do that very well and kind of by design don't do it very well. And there's reasons to have both. So the classical stuff is really good at, for example, representing databases and ontologies like a Robin is a bird, a bird is an animal, and concluding, therefore, that Robin is an animal. Classical AI techniques are perfect at that stuff. They never hallucinate. And they're not going to make up that Harry Shear was born in the UK when he was actually born in Los Angeles. They're able to stick to the data.

What we need is some kind of rapprochement between those two approaches, some way of bringing them together. People are starting to play around with some. I think they have partial ideas. people have been able to do it in narrow domains. So alpha fold is a neuro symbolic system, alpha geometry, a bunch of systems that DeepMind has built are kind of very customized neurosymbolic systems for particular purpose. And people are starting to put in code interpreters, which are symbolic inside of large language models and getting some results out of that. So that's one aspect of what we need to do.

But it's not the only thing. It would be like saying we need computer programs. Well, of course you need computer programs. But knowing that you have computer programs doesn't give you all the world software, right? You need other things. You need, for example, operating systems, and you need certain ideas about design. So there's a famous one. It's called Model View Controller, for example.

You need to have networking and all kinds of different things. So, you know, it was a vital step towards modern software, for example, to have machine language and then assembly language and compilers, high-level programming languages. So neuro-symbolic AI is kind of like a broad agenda or a broad framework, but there are many ways to develop it. And we don't yet have that much understanding of it because we've been investing almost entirely in pure neural networks and not spending enough effort investing and understanding kind of what are the best practices, the best tools for using neuro-symbolic AI. So I would start there. The other thing I would really emphasize is what I would call world models. So, for example, we're doing this call over video.

And I have a little bit of data about the world in which you inhabit. So you are probably in an office. There are shells behind you. There are boxes that maybe contain magazines. You're wearing a sweater. You've got some kind of headset on. You're nodding your head when you understand some of my points. You're smiling.

So probably I'm giving you answers of roughly the length that you want. I'm not going on too long or I would try to infer that. So we build models of the world. They're not even always of the actual world. So my kids are watching Harry Potter and reading Harry. Potter. So they have a model of that world where there are wands and magic and flying broomsticks. And they're able perfectly well to separate those worlds from this world. If you bust out with a magic wand in the middle of our call and it does magic, I'm going to be very surprised. It's not

going to be consistent with the model I have of the world that I believe you to be in. But we use these models all the time. Like when we watch a movie, let's say we haven't read the Harry Potter books and we watch a movie. We make a model of like, what is Snape doing and why? What are his motivations? And we're constantly developing those. We don't really know how to do that in AI yet. So in AI, we know how to build in a world model from scratch. So for example, GPS navigation systems literally have a model of the world. It's an abstracted model, but they know distances between cities or distances between different locations, different streets. They may know something about the traffic on them. So it's literally a model of the world. And the way that they work is by doing computation over that

model of the world. It's not a complete model of the world. So it doesn't know anything about who Kim Kardashian is, for example. So, you know, it has some kinds of information and not others. But we don't know how to make AI systems derive or induce learn new models from data. So we can build these very specific models like streets. But we don't know how to have an AI system, watch a video, let's say a Harry Potter movie, and figure out what's going on there. You know, we even know how to have a system read secondhand sources about Harry Potter,

But we don't know how to have it put all that stuff together for itself. The way that a child can learn about numbers and figure out what integers are and start to count and understand what infinity is and so forth. Or in the way that I put together, you know, a model of your world right now. We don't know how to get AI to do that. I think that's the big problem for the next, I'll say, five to 20 years. I think when we solve that one, we might actually have artificial genital intelligence, or at least we will have taken a tremendous step closer to.

getting to artificial general intelligence. I don't think we're that close right now. I don't think enough people are thinking about that problem, although in the last year I've seen more people think about it. I don't think we're putting enough resources on it. Large language models have allowed people to duck that question and they kind of work around it. They don't really build those models. So when the field really fully engages in that world model problem and in that neural symbolic AI problem and a few others, then I think we'll really make progress. do we almost need the AI bubble to burst in order to make way for that approach, do you think, to make space for it? That is a really good question.

The tentative answer is probably yes. So I'll give you another example, historical, which is for a long time, the early 20th century, people thought that genes were proteins. They put way too much effort into trying to identify which protein genes were made of. And that was just wrong. And then Oswald Avery did these experiments in 1940s, proved by the process of elimination that it was actually. actually this weird sticky acid that nobody knew anything about called DNA. And they knew a little bit about it. And once he figured that out, that kind of enabled Watson and Crick with Rosalind Franklin's

kind of unwitting help to figure out the structure of DNA. And then, you know, Crick and others were able to figure out the code that described the relation between DNA sequences and proteins. And, you know, it was off to the races. Now we can sequence it. We can make new proteins if we want, all kinds of things that we can do. But it, you know, there was like 20 years where the field was kind of in the wilderness looking at the wrong thing.

And then once it got to the right thing, it moved faster, right? So we need people to realize that this thing we're doing now is not the right thing in order to have the intellectual oxygen and energy to go do the right thing. And so, yeah, if the bubble crashes, two things happen. One is people just invest less money in AI, which is not good if you want to see us, get to AI, which is a complicated question. But the other is it opens people's minds to other things. And I think a lot of the worries that we have about AI right now are because the AI

that we have is not reliable. So people are worried about doom and will it kill us all. If we had better forms of AI, those questions might not be as serious. Like, you know the famous paperclip scenario where the AI turns us all into paper clips. Like if you had a smart AI, you should just be able to say, you know, don't do anything harmful to the entire species or to, you know, large numbers of people and have it actually obey that. The problem right now is AI is too stupid to follow our instructions reliably. If we had a more reliable AI, some of those fears might go away. It's complicated. But in order to get to a better

AI, we have to give up on this E-Day fix of we're going to get it by scaling. And maybe that only happens when the money starts to dry up and people are like, well, let's think about this again from first principles. And that's what we need. Do you want us to build AGI? Do you think that's a good thing for the world? I think it could be. But I also think, that large language models were kind of dress rehearsal, and it's a pretty depressing dress rehearsal. Like one way you're like that. I don't know if I want this play to go on. So there's a dress rehearsal in the sense that we learn some things, like how governments respond to what they think might be artificial general intelligence. And, you know, one of the ways they responded

is by not regulating the companies making it, which is precisely the wrong decision to have made. They have, you know, started to think about handouts for the companies. That's not a great decision neither. What we have learned also is that people in government and media and most people in general are not that good at evaluating the actual quality of AI. And there's not enough effort or enough kind of investment in independent scientific experts giving evaluation. We've learned a lot of things about how we're just not handling this very competent. We've also learned that it's very hard to get countries together to build treaties. And, And if we're going to have a more powerful AI, we probably need to have treaties about enforcement

and what should be allowed, et cetera. And we've kind of fumbled all of that. And so that makes me concerned. The positive argument is I think a smarter, more trustworthy AI, if we could build it properly, could actually help enormously with disease, with climate change, with all kinds of technology, with education. Like, I think the upside could be very high. But we're not going to get to that upside if we just let, you know, attack oligarchy dictate terms to governments in ways that really aren't looking out for humanity's best interests. So I'm more ambivalent than I was earlier in my life. You know, I've always loved AI. I know it doesn't sound like that when I keep, you know, ripping generative AI. But I'm ripping generative AI because I want AI to succeed. Sometimes I think about myself as like a parent of two teenagers. I don't quite have that yet. But, you know, or a teenager that's

not really doing what you want, and you want the best for it, but you can see right now it's not making the best choices. So that's kind of how I feel about AI right now is I want it to succeed. I think, you know, there is partly because of the limits of human rationality, which we haven't talked about, but my book Kluge is about, I really see the potential for AI to make the world a better place. But I see the way that it's being enacted right now, both the political side and the technical side. And I'm like, whoa, hold on. Like, we need to do a lot better if we're really going to get to the positive outcome rather than the negative one. So that was Gary Marcus, a scientist, author, an entrepreneur, known as a leading voice in AI.

If you'd like to stay in the loop with Gary's musings on AI, you can subscribe to his substack at Gary Marcus. Thank you for listening to this episode of Instant Genius, brought to you by the team behind BBC Science Focus. If you like what you just heard, then please consider subscribing to Instant Genius on your preferred podcast platform. If you'd like to watch our presenters and guests speaking in person, then you can also check out our YouTube channel at ScienceFocus. The current issue of BBC Science Focus magazine is on sale now. Pick up a copy wherever you buy your favourite magazines or download us on your app store of choice. You can also find us on Apple News or online at ScienceFocus.com. This podcast is sponsored by name, audio and focal.

The texture and emotional depth of music can be lost through digital sources or poor, not. Name Audio believes you can have digital precision with analog warmth. Alongside French acoustic specialist vocal, name creates high-end audio systems combining innovation with craftsmanship so you can listen to music, just as the artist intended. Discover more at name audio.com. Ambition comes in all shapes and sizes. At First Citizens Bank, we roll with your goals because we're built for what you're building. Fit for your ambition for citizens back. Santa Monica College is the number one transfer college to the UCs for 35 straight years.

With caring faculty, dedicated counselors, and affordable tuition, SMC helps you reach top universities with confidence. Summer classes start June 22nd. Learn more at SMC.edu.