Big Technology Podcast - Dwarkesh Patel: AI Continuous Improvement, Intelligence Explosion, Memory, Frontier Lab Competition

Episode Date: June 18, 2025

Dwarkesh Patel is the host of the Dwarkesh Podcast. He joins Big Technology Podcast to discuss the frontiers of AI research, sharing why his timeline for AGI is a bit longer than the most enthusiastic researchers. Tune in for a candid discussion of the limitations of current methods, why continuous AI improvement might help the technology reach AGI, and what an intelligence explosion looks like. We also cover the race between AI labs, the dangers of AI deception, and AI sycophancy. Tune in for a deep discussion about the state of artificial intelligence, and where it’s going. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Want a discount for Big Technology on Substack? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Questions? Feedback? Write to: [e-mail]

Why do we have such vastly different perspectives on what's next for AI if we're all looking at the same data and what's actually going to happen next? Let's talk about it with Dworkesh Patel, one of the leading voices on AI who's here with us in studio to cover it all. Dwar Keshe, great to see you. Welcome back to the show. Thanks for having me, man. Thanks for being here. I was listening to our last episode, which we recorded last year, and we were anticipating what was going to happen with GPT-5. Still no GPT-5. That's right. Oh, yeah. That would have surprised me a year ago. Definitely.

And another thing that would have surprised me is we were saying that we were at a moment where we were going to figure out basically what's going to happen with AI progress, whether the traditional method of training LLMs was going to hit a wall or whether it wasn't. We were going to find out. We were basically months away from knowing the answer to that. Here we are a year later. We have, everybody's looking at the same data, like I mentioned in the intro, we have no idea. There are people who are saying AI, artificial general intelligence, or human human-level intelligence is imminent with the methods that are available today, and there are

others that are saying 20, 30, maybe longer, maybe more than 30 years until we reach it. So let me start by asking you this. If we're all looking at the same data, why are there such vastly different perspectives on where this goes? I think people have different philosophies around what intelligence is. That's part of it. I think some people think that these models are just basically baby agi's already, and they just need a couple additional little unhoused. the legs, a little sprinkle on top, things like test time thinking, so we already got that with 01 and 03 now, where they're allowed to think, they're not just doing it like saying the first thing that comes to mind. And a couple other things like, oh, well, they should be able

to use your computer and have access to all the tools that you have access to when you're doing your work. And they need context in your work. They need to be able to read your Slack and everything. So that was one perspective. My perspective is slightly different from that. I don't think we're just right around their corner from AGI and it's just additional dash of something, that's all that's going to take. I think, you know, people often ask if all AI progress stopped right now. And all you can do is collect more data or deploy these models in more situations. How much further could these models go?

And my perspective is that you actually do need more algorithm of progress. I think a big bottleneck these models have is there inability to learn on the job, to have continual learning. Their entire memory is extinguished at the end of a session. There's a bunch of reasons why I think this actually makes it really hard to get human-like labor out of them. And so sometimes people say, well, the reason Fortune 500 isn't using LLMs all over the place is because they're too stodgy. They're not thinking creatively about how AI can be implemented.

And actually, I don't think that's the case. I think it actually is genuinely hard to use these AIs to automate a bunch of labor. Okay, so you've said a couple interesting things. First of all, that we have the AIs that can think right now, like O3 from Open AI. We're going to come back to that in a moment. but I think we should really seize on this idea that you're bringing up that it's not laziness within Fortune 500 companies that's causing them to not adopt these models. Or I would say they're all experimenting with it, but we all, you know that the rate to get proof of concepts out the door is pretty small. One out of every five actually gets shipped into production, and often it's a scaled down version of that.

So what you're saying is interesting. You're saying it's not their fault. It's that these models are not reliable enough to do what they need to do because they don't learn on the job. Am I getting that right? Yeah. And it's not even about reliability.

It's just they just can't do it. So if you think about what makes humans valuable, it's not their raw intelligence, right? Any person who goes onto their job the first day, even their first couple of months maybe, they're just not going to be that useful because they don't have a lot of context. What makes human employees useful

is their ability to, build up this context, to interrogate their failures, to build up these small improvements and efficiencies as they practice a task. And these models just can't do that, right? You're stuck with the abilities that you get out of the box. And they are quite smart. So you will get five out of ten on a lot of different tasks. They'll, often they'll, on any random task, they'll probably might be better than an average human. It's just that they won't get any better. I, for my own podcast, I have a bunch of little scripts that I've tried to write with LLMs where I'll get them to rewrite parts of transcribes to make them more

turn auto-generated transcripts into like human written like transcripts or to help me identify clips that I can tweet out. So these are things which are just like short horizon, language in language outtasked, right? This is the kind of thing that the LLM should be just amazing at because it's a debt center of what should be in their repertoire and they're okay at it. But the fundamental problem is that you can't like,

you can't teach them how to get better in the way that if a human employee did something, you'd say, I didn't like that. I would prefer it this way instead. And then they're also looking at your YouTube studio analytics and thinking about what they can change. This level of understanding or development is just not possible with these models. And so you just don't get this continual learning,

which is a source of, you know, so much of the value that human labor brings. Now, I hate to ask you to argue against yourself, but you are speaking all the time. And I think we're in conversation here on this show all the time with people who believe that if the models just get a little bit better, then it will solve that problem. So why are they so convinced that the issue that you're bringing up is not a big

stumbling block for these AIs? I think they have a sense that, one, you can make these models better by giving them a different prompt. So they have the sense that even though they don't learn skills in the way humans learn, you've been writing and podcast and you've gotten better at those things just by practicing and trying different things, and seeing how it's received by the world. And they think, well, you can sort of artificially get that process going by just adding to the system prompt.

This is just like the language you put into the model at the beginning to say, like, write it like this. Don't write it like that. The reason I disagree with that perspective is imagine you how to teach a kid how to play the saxophone, but you couldn't just have, you know, how does a kid learn the saxophone now? She tries to blow into one and then she hears how it sounds, she practices a bunch. Imagine if this is the way it worked instead. A kid tries to just like

Never seen a saxophone They try to play the saxophone And it doesn't sound good So you just send them out of the room Next kid comes in And you just like write a bunch of instructions About why the last kid messed off And then they're supposed to like

Play Charlie Parker Cold By like reading the set of instructions It's just like you would They wouldn't learn how to play a saxophone that way right Like you actually need to practice So anyways this is this is all to say That I don't think prompting alone Is that powerful a mechanism

Of teaching models these capabilities Another thing is people say you can do RL, so this is where the reinforcement learning. That's right. These models have gotten really good at coding and math because of you can, you have verifiable problems in that domain where they can practice on them. Can we take a moment to explain that for those who are new to this? So let me see if I get it right. Reinforcement learning. Basically, you give a bot a goal or you give an AI system a goal saying solve this equation

and then you have the answer and you effectively don't tell it anything in between. So it can try every different solution known to humankind until it gets it. And that's the way it starts to learn and develop these skills. Yeah. And it is more human-like, right? It's more human-like than just reading every single thing on the Internet and then learning skills. I still think I'm not confident that this will generalize to domains that are not so verifiable or text-based. Yeah, I mean, like a lot of domain, it just like would be very hard to set up this environment and loss function.

for how to become a better podcaster. And, you know, whatever. People might not think podcasting is like the crux of the economy, which is fair. It's the new AGI test. But like a lot of tasks are just like much softer and there's not like an objective URL loop. And so it does require this human organic ability to learn on the job.

And the reason I don't think that's around the corner is just because there's not, there's no obvious way, at least as far as I can tell, just slot in this online learning into the models as they exist right now. Okay. So I'm trying to take in what you're saying, and it's interesting. You're talking about reinforcement learning as a method that's applied on top of modern day, large language models and system prompts. And maybe you'd include fine-tuning in this example.

Yeah. But you don't mention that you can just make these models better by making them bigger, the so-called, you know, scaling hypothesis. So have you ruled out the fact that they can get better, through the next generation of scale? Well, this goes back to your original question about what has, have you learned? I mean, it's quite interesting, right?

I guess I did say a year ago that we should know within the next few months which trajectory we're on. And I feel at this point that we haven't gotten verification. I mean, it's narrowed, but it hasn't been as to sizes I was expecting. I was expecting like,

GPT5 will come out and move will know. Did it work or not? And to the extent that you want to use that test from a year ago, I do think you would have to say, like, look, pre-training, which is this idea that you just make the model bigger, that has had diminishing returns. So we have had models like GPT4.5, which there's various estimates,

but, or GROC, was it GROC 2 or GROC 3, the new one? I've lost count with GROC. That's right. Regardless, I think they're estimated to be, you know, 10x bigger than GPD4, and they're like, they're not obviously better. So it seems like there's plateauing return. to pre-training scaling. Now we do have this RL, so 01, 03, these models,

the way they've gotten better is that they're practicing on these problems, as you mentioned, and they are really smart. The question will be how much that procedure will be helpful in making them smarter outside of domains like math and code and of solving what I think are very fundamental bottlenecks like continual learning or online learning.

There's also the computer use stuff, which is a separate topic. But I would say I have longer timelines than I did a year ago. Now, that is still to say, I'm expecting 50-50 if I did like make a guess, I'd make a bet. I'd just say 2032, we have like real AGI that's doing continual learning and everything. So even though I'm putting up the pessimistic facade right now, I think people should know that this pessimism is like me saying, in seven years, the world will be so wildly different that you like really just can't imagine it. And seven years is not that long a period of time. So I just want to make that disclaimer, but yeah.

Okay, so I don't want to spend this entire conversation about scaling up models because we've done enough of that on the show, and I'm sure you've done a lot of it. But it's interesting, you use the term plateauing returns, which is different than diminishing returns, right? So as your sense, because we've seen, for instance, Elon Musk do this project, Memphis, where he's put basically every GPU you can get a hold of and he can get a lot because he's the richest private citizen in the world together. And I don't know about you.

but like I said, again, I haven't paid so much attention to GROC because it doesn't seem noticeably better even though it's using that much more size. Now there is algorithmic efficiency that they may not have that someone like, it's not like a company like Open AI might have. But I'll just ask you the question I've asked others that have come on the show. Is this sort of the end of that scaling moment? And if it is, what does it mean for AI? I mean, I don't think it's the end of scaling.

Like, I do think companies will continue to pour exponentially more computing to train these systems. And they'll continue to do it over the next many years. Because even if there's diminishing returns, the value of intelligence is so high that it's still worth it. Right. So if it costs $100 billion, even a trillion dollars to build AGI, it is just definitely worth it. It does mean that it might take longer. Now, here is an additional wrinkle by 2028 or so, definitely by 2030. Right now, we're scaling up the training of Frontier systems

4x a year, approximately. So every year, the biggest system is 4x bigger than, not just bigger, I shouldn't say bigger, users more compute than the system the year before. If you look at things like how much energy is there in the country, how much, how many chips can TSM produce, and what fraction of them are already being used by AI?

Even if you look at like raw GDP, like how much money does the world have, how much wealth does the world have? By all those metrics, it seems like this pace of 4x a year, which is like 160x in four years, right? Like, this cannot continue beyond 2028. And that means at that point, it just will have to be purely algorithms. It can't just be scaling up a compute. So, yeah, we do have just a few years left to see how far this paradigm can take us.

And then we'll have to try something new. Right. But so far, because again, we were here last where we were talking virtually now we're in person. We're talking last year about, all right, well, opening eye clearly is going to put, I mean, GPT4.5 was supposed to be GPT5. I'm pretty sure. That's my, from what I've read, I think that's the case. Didn't end up happening. Right. So it seems like this might might be it. Yeah, yeah. And over the next year, I think we'll learn what's going to happen with our, because I think, well, I guess as I said this last year, right? I guess it wasn't wrong. We did learn, but it's done to a lot training.

Yeah. But yeah, over the next year, we will learn. So I think RL scaling is happening much faster than even overall training scaling. So what does RL scaling look like? because, again, here's the process, again, is you give, so the RL scaling, reinforcement learning scaling, you give the bot an objective and it goes out and it does these different attempts and it figures it out on its own. And that's bled into reasoning what we were talking about with these O3 models where you see the bot going step by step. So you can scale that in what way? Just giving it more opportunities. I'm not a researcher at the labs, but my understanding is that what's been increasing is,

RL is harder than pre-training because pre-training, you just like throw bigger and bigger chunks of the internet, at least until we ran out. We seem to be running out of tokens, but until that point, we're just like, okay, just like, now use more of the internet to do this training. RRL is different because there's not this, like, fossil fuel that you can just, like, keep dumping in. You have to make bespoke environments for the different RL skills. So you have to make an environment for a software engineer, I had to make an environment for a mathematician. All of these different skills, you have to make these environments. And that is sort of, that is like hard engineering work, hard, like, just like monotonous, like, just got, you know, grinding or schlepping. And my understanding is the reason that RL hasn't scale, you know, people aren't are immediately dumping in billions of dollars in RL is that you actually just need to build these environments first.

And the O3 blog post mentioned that it uses 10x more, it was straight down 10X more compute than 01. So already within the course of six months, RL compute has 10xed. That pace can only continue for a year, even if you build up all the RL environments, before it's, you know, you're at the frontier of training compute versus these systems overall. So for that reason, I do think we'll learn a lot in a year about how much this new method of training will give us in terms of capabilities. That's interesting because what you're describing is building up AIs that are really good at, certain tasks. These are sort of narrow AIs. Doesn't that kind of go against the entire idea of building up a general intelligence? Like can you build AGI? By the way, like people use AGI as this

term with no meaning. General is actually pretty important there, the ability to generalize and the ability to do a bunch of different things as an AI. So even if you get like reinforcement learning is definitely not a path to artificial general intelligence given what you just said because you're just going to train it up on different functions, maybe until you have something that's broad enough that it works. I mean, this has been a change in my general philosophy or thinking you're on intelligence. I think a year ago or two years ago, I might have had more of the sense that,

intelligence really is this fundamentally super, super general thing. And over the last year, from watching how these models learn, maybe just generally seeing how different people in the world operate even, I do think, I mean, I still buy that there is such a thing as general intelligence, but I don't think it is, like, I don't think you're just going to train a model so much, on math that is going to learn how to take over the world or learn how to do diplomacy. And I mean, just like,

I don't know how much you talk about political current events on the show. We do enough. Okay. Well, it's just like, without making any comments about like what you think of them, Donald Trump is like not proving theorems out there, right? But he's like really good at like gaining power.

And conversely, there are people who are amazing at proving theorems that can't gain power. And it just seems like the world kind of, like, I just don't think you're going to train. the AI so much on math that is going to learn how to do Henry Kissinger level diplomacy. I do think skills are somewhat more self-contained. So that being said, like, there is correlation between different kinds of intelligences and humans.

I'm not trying to understate that. I just think it was not as strong as I might have thought a year ago. What about this idea that it can just get good enough in a bunch of different areas? Like, imagine you built a bot that had like, let's say, 80% the political acumen of Trump, but could also code like an expert-level code. That's a pretty powerful system. That's right. I mean, this is one of the big advantages the AIs have,

is that especially when we solve this on-the-job learning kind of thing I'm talking about, you will have, even if you don't get an intelligence explosion by the AI writing future versions of itself that are smarter. So this is the conventional story that you have this fume, that's what it's called, which is the sound it makes when it takes off. That's right, yeah. where the system just makes itself smarter and you get a super intelligence at the end.

Even if that doesn't happen, at the very least, once continual learning is solved, you might have something that looks like a broadly deployed intelligence explosion, which is to say that because if these models are broadly deployed to the economy, every copy that's like this copy is learning how to do plumbing and this copy is learning how to do analysis of a finance firm

and whatever. The model can learn from what all these instances are learning and amalgamate all these learning in a way that humans can't, right? Like if you know something and I know something, it's like a skill that we spend our life learning. We can't just like melt their brains. So for that reason, I think it like you might have something that which functionally looks like a super intelligence by the end because even if it's like not making any software

progress, just this ability to like learn everything at the same time might make it functionally super intelligence. What about this idea? I mean, I was just at Anthropics developer event where they showed the bot, a sped-up version of the bot, coding autonomously for seven hours. You actually, so let's just say so people can find it. You have a post on your substack, why I don't think AGI is right around the corner. And a lot of the ideas we're discussing comes from that. So folks, check that out if you haven't already. But one of the things you talk about is this idea of autonomous coding. And you're also a little skeptical of of that because you'll have to just, okay, I think you brought up this conversation that you had

with two anthropic researchers where they expect AI on its own to be able to check all of our documents and then do our taxes for us by next year. But you bring up this point, which is interesting, which is like if this thing goes in the wrong direction within two hours, you might have to like check it, put it back on the right course. So just because it's working on something for so long, doesn't necessarily mean it's going to do a good job. Am I capturing that, right? Yeah, it's especially relevant for training because the way we train these models right now is like you do a task and if you did it right, positive reward, if you did a wrong,

negative reward. Right now, especially with pre-training, you get a reward like every single word, right? You can like exactly compare what word did you predict, what word was a correct word, what was the probability difference between the two? That's your reward functionally. then we're moving into slightly longer horizon stuff so to solve a math problem might take you a couple of minutes

at the end of those couple of minutes we see if you solve the math problem correctly if you did reward now if we're getting to the world where you got to do a project for seven hours and then at the end of those seven hours then we tell you hey did you get this right

then like the progress just slows on a bunch because you've gone from like getting signal within the matter of like microseconds to getting signal at the end of seven hours and so the process of learning has just like become exponentially longer.

And I think that might slow down how fast these models. Like, you know, the next step now is like not just being a chat bot, but actually doing real tasks in the world, like completing your taxes, coding, et cetera. And two of these things, I think progress might be slower because of this dynamic where it takes a long time for you to learn whether you did the task right or wrong. But that's just in one instance. So imagine now I took 30 clods and said, do my taxes.

And maybe two of them got it right. Right. That's good. Yeah, I just got it done in seven hours, even though I had 30 bots working on it at the same time. I mean, from the perspective of the user, that's totally fine. From the perspective of training them, all those 30 clods took probably dozens of dollars to, if not hundreds of dollars, to do that all those hours of tasks. So the compute that will be required to train the systems will just be so high. And anything even from the perspective of inference, like, I don't know, you probably just want to like spend a couple hundred bucks every afternoon on 30 different clouds.

Just to have fun. But that would be cheaper than an accountant. We got to find you a better, a cheaper accountant. Well, I guess if I'm spending a couple hundred on each. That's right. Yeah. But you had a conversation with a couple of AI skeptics, and you kind of rebutted not exactly the point you're making,

but you had a pretty good argument there, where you said that we're getting to a world where because these models are becoming more efficient to run, you're going to be able to run cheaper, more efficient experiments. So every researcher who was previously constrained by compute and resources, now will just be able to do far more experiments, and that could lead to breakthroughs. I mean, this is a really shocking trend.

If you look at what it cost to train GVD4 originally, I think it was like 20,000 800s over the course of 100 days. So I think it costs on the order of like half a million to $100 million somewhere in that range. And I think you could train an equivalent system today. In DeepSeek, we know, was to train on $5 million, supposedly, and it's better than GPD4, right? So you've had literally multiple orders of magnitude decrease, like 10 to 100x decrease in the cost to train a GPD4 level system. You extrapolate that forward.

Eventually, you might just be able to train a GPD4 level system in your basement with a couple of H-100s, right? Well, that's a long extrapolation. But before, I mean, like, it'll get a lot cheaper, right? Like a million dollars, $500,000, whatever. And the reason that matters is it's related to this question of the intelligence explosion where people often say, well, that is not going to happen

because even if you had a million super smart automated AI researchers, so AI is thinking about how to do better AI research, they actually need the compute to run these experiments to see how do we make a better GPT6. And the point I was making was that, well, if it's just become so much cheaper to run these experiments because these models have become so much smaller. or it's so much better, easier to train than that might speed of progress.

Which is interesting. So we've spoken about, you brought up intelligence explosion a couple of times. So let's talk about that for a moment. There's been this idea that AI might hit this inflection point where it will start improving itself. Right. And the next thing you know, you hear that, what was the sound? Fume. You hear a fume.

And we have artificial general intelligence or super intelligence right away. So how do you think that might? take place? Is it just these coding solutions that just sort of improve themselves? I mean, DeepMind, for instance, had a paper that came out a little while ago where they have this thing inside the company called AlphaEvolve that has been trying to make better algorithms and helped reduce, for instance, the training time for their large language models. Yeah. I'm genuinely not sure how likely an intelligence explosion is. I don't know. I'd say like 30% chance it happens. Which is crazy, by the way, right?

That's a very high percentage. Yeah. And then what does it look like? That's also another great question. I've had like many hour-long discussions on my podcast about this topic, and it's just so hard to think about. Like, what exactly is super intelligence? Is it actually like a god?

Is it just like a super smart friend who's good at mathematics? And, you know, we'll beat you in a lot of things. But like, you can still understand what it's doing, right? So, yeah, honestly, they're tough questions. I mean, the thing to worry about, obviously, is if we live in a world with millions of superintelligence is running around and they're all trained in the same way, they're trained by other AI, so it's dumber versions of themselves, I think it's really worth worrying about, like, what has that, why were they trained in a certain way, are they, do they have these, like, goals we don't realize, would you even know if that was the case, what might be they want to do? there's a bunch of throwing questions that come up. What do you think it looks like? I think we totally lose control over the process of training smarter AIs,

or we just like let the AIs loose, just make a smarter version of yourself. I think we end up in a bad place. There's a bunch of arguments about why, but like, you know, you're just like, who knows what could come out at the other end and you've just like let it loose, right? So by default, I would just expect something really strange. to come out the other end. Maybe it'd still be economically useful in some situations, but it's just like,

you haven't trained it in any way. It's just like, imagine if there was a kid, but it didn't have any of the natural intuitions, moral intuitions or... Parenting. Yeah, exactly, the humans have. They just like, it just like became an Einstein, but it was like, it trained in the lab

and who knows what it saw. Like, it was like totally uncontrolled. Like, you'd kind of be scared about that. Especially now with like, oh, all your society these infrastructure is going to run on, like a million copies of that kid, right? Like, the government is going to, like, be asking it for advice, the financial system is going to run off it, all the engineering, all the code written in the world. We were written by this system.

I think it's like you'd be, like, quite concerned about that. Now, the better solution is that while this process is happening of the intelligence explosion, if we have one, you use AI's not only to train better AI's, but also to see there's different techniques to figure out, like, what are your true motivation? What are your true goals? Are you deceiving us? Are you lying to us? There's a bunch of alignment techniques

that people are working on here, and I think those are quite valuable. So alignment is trying to align these bots with human values or the values that their makers want to see with them. Yeah. And I think people get into, it's often really hard to define

what do you mean by human values, like those values exactly? I think it's much easier just to say, like, we don't want these systems to be deceptive, right? We don't want them to, like, lie to us. We don't want them to, like, be actively trying to harm us or seek power or something? Does it worry you that from the reporting,

it seems like these companies, the AI Frontier Labs, not all of them, but some, they've raised billions of dollars. There is a pressure to deliver to investors. There are reports that safety is becoming less of a priority as market pressure makes them go and ship without the typical reviews. So is this kind of a risk for the world here that these companies are doing?

developing the stuff. Many started with the focus on safety and now seems like safety is taking a back seat to financial returns. Yeah, I think it's definitely a concern. Like, we might be facing a tragedy of the common situation where obviously all of us want our society and civilization to survive. But maybe the immediate incentive for any lab CEO is to, look, if there is an intelligence disclosure, and it tests a really tough dynamic because if you're a month ahead, you will kick off this loop much faster than anybody else. And what that means is that you will be a month ahead to super intelligence when nobody else will have it, right? Like, you will get, you'll get the 1,000x multiplier on research much faster than anybody else. And so it could be a sort of

winner-take-all kind of dynamic there. And therefore, they might be incentivized. Like I think to keep this system, keep this process in check, might require slowing down. using these alignment techniques to like that which might be sort of attacks on the speed of the system and so yeah I do worry about the the pressures here okay a couple more model improvement questions for you then I want to get into some of the competitive dynamics between the labs and then maybe some more of that deceptiveness topic which is really important and we want to talk about here you your north star is continuous improvement that these models basically learn how to improve themselves as opposed to having a model developer I mean in some way it's like

a mini intelligence explosion or complete. So what do you think, it doesn't seem like it's going to happen through RL because that's, again, like you said, specific to certain disciplines. It's specific to what you, what bespoke thing you do. Right. If it's in another domain, you have to like make it rather than not learning on its own. And we have some diminishing returns or plateau that's coming with scaling. So what do you think, I mean, we won't hold you to this, but what do you think the best way

to get to that can, you know, continuous learning method? of these models is? I have no idea. Can I give a suggestion? I mean, why don't you answer, then I'll give a thought here. I mean, if I was running one of the labs, I would keep focusing on RL because it's the obvious thing to do. Okay.

And I guess I would also just be more open to trying out lots of different ideas because I do think this is a very crucial bottleneck to these models value that I don't see an obvious way to solve. So I'd be slowly remember, but like definitely I don't have any idea of how to solve this. Right. Does memory play into it? So that was the thing I was going to bring up. I mean, one of the things that we've seen, let's say, O3 or ChatGPT do is OpenAI now has it sort of able to remember all your conversations or many of your conversations that you've had.

I guess it brings those conversations into the context window. So now, like, when I tell Chatchapit do write like an episode description in big technology style, it knows the style and then it can actually go ahead and write it. And it goes to your earlier conversation about, like, your editors know your style, they know your analytics, and therefore they're able to do a better job for you. So does building better memories into memory into these models actually help solve this problem that you're bringing up? I think memory, the concept is important. I think memory that's implemented today is not the solution. The way memory is implemented now, as you said, is that it brings these previous conversations back into context,

which is to say it brings the language of those conversations. back into context. And my whole thing is, like, I don't think language is enough. Like, I think the reason you understand how to, like, run this podcast well is not just like you're remembering all the words that you were like, I don't know, like some, I think you wouldn't even be all the words. It would be like some of the words you might have thought in the past. You've, like, actually, like, learned things.

It's like, it's been baked into your weights. And that I don't think is just, like, you know, like, look up the words that I said in the past or look at the conversations I said in the way. the past. So I don't think those features are that useful yet, and I don't think that's the path to solving this. Kind of goes to the discussion you had with Dario Amadei that you tweeted out, and we've actually brought up on the show with Jan Lacoon about why AI cannot make its own discoveries. Is it that similar limitation of not being able to build on the knowledge that it has? Yeah, I mean, that's a really interesting connection. I do think that's plausibly it.

Like, I think any scientists would just have a very tough time, like, you're putting, somebody really smart is just put in a totally different discipline. And they can, like, read any textbook they want in that domain. But, like, they don't have a tangible sense of, like, what, I've tried this approach in the past and it didn't work. And, you know, like, oh, my, there was this conversation and here's how the different ideas connected. They just haven't been trained. Like, they've, like, read all the textbooks. They haven't, like, more accurately, actually, they've just, like, skimmed all the textbooks. but they haven't like imbibed this context that um which is what i think what makes human scientists

productive and come up with these new discoveries it is interesting because the further these frontier labs go the more that they're going to tell us that their AI is actually making new discoveries and new connections like i think open ai said that 03 was something that made was able to make connections between concepts sort of addressing this and every time we have this discussion on the show. We talk about how AI hasn't made discoveries. I get people yelling at me, my email being like, have you seen the patents?

Yeah. Like an alpha, maybe using things like Alpha Evolves, an example that these things are actually making original discoveries. What do you think about that? Yeah, I mean, there was another interesting thing called, I don't know if you saw Future House. No.

They found, um, some drug can be, has another application. I don't remember the details, but it was like, it wasn't impressive. Like, it wasn't like earth chattering. Like,

or something for the first time. But it was like, oh, using some logical induction, they were like, this drug, which is used in this domain, it uses the same mechanism that would be useful in this other domain, so like maybe orcs, and then the AI came up with, designed the experiment, so it came up with the idea of the experiment to test it out. A human in the lab was just tasked with, like, running the experiment,

like, you know, pipette, whatever, into this. And I think they were tried out like 10 different hypotheses. One of them actually ended up being verified, and the AI had found the relevant pathway to making a new use for this drug. So I think I am like, that is becoming less and less true my question.

I'm not like wedded to this idea that like E.S. will never be able to come up with discoveries. I just think it was like true longer than you would have expected. I agree. The way that you put it is like it knows everything. Yeah. So if a human had that much knowledge

about medicine, for instance, they'd be spitting on discoveries left and right. And we have put so much knowledge into these models and we don't have the same level of discovery, which is a limitation. But I definitely hear you like, this is on a much smaller scale than those medical researchers. But I definitely, a couple months ago, when 03 first came out, this is again, I think we're both fans of O3's, of Open AI's O3 model, which is just, it's able to reason. It's a vast improvement over previous models.

But what I did was I had like three ideas that I wanted to connect in my newsletter. And I knew that they connected. And I was just struggling to just crystallize exactly what it was. And I was like, I know these three things are happening. I know they're connected help. And O3 put it together, which to me was just mind-boggling. Yeah. It's a, it is kind of helpful as a writing assistant because a big problem I have in writing.

I don't know if it's the case for you is just this idea. Like I kind of know what I'm trying to say here. I just need to get it out into words. It's like the typical. Every writer has this pretty much. It's actually useful to use a speech-to-text software, like Whisper Flow or something. And I just speak into the prompt. Like, okay, I'm trying to say this.

help me put it into words. The problem is, like, actually, continual learning is, like, still a big bottleneck, because I've had to rewrite or re-explain my style many times. And if I had a human collaborator, like a human copywriter, who was good,

they would have just, like, learn my style by now. You wouldn't need to, like, keep re-explaining. I want you to be concise in this way, and here's how I like things phrased, not this other way. Anyways, so you still see this bottleneck, but, again, five out of ten is not nothing. Right.

All right, let me just put a punctuation, exclamation point on this or whatever mark you would say. When I was at Google I.O. with Sergei. and Demis, one of the most surprising things I heard was Sergey just kind of said, listen, the improvement is going to be algorithmic from here on, or most of the improvement is going to be algorithmic. I think in our conversation today already, basically we've narrowed in on this same idea, which is that scales sort of gotten generative AI to this point. It's a pretty impressive. point, but it seems like it will be algorithmic improvements that take it from here.

Yeah. I do think it will still be the case that, like, those algorithms will also require a lot of compute. In fact, what might be special about those algorithms is that they can productively use more compute, right? The problem with pre-training is that whether it's because we're running out of the pre-training data corpus with RL, maybe it's really hard to scale up RL environments, the problem with these algorithms might just be that, like, they can productively

absorb all the compute that we have, or we want to put it in these systems over the next year. So I don't think compute is out of the picture. I think we'll still be scaling up 4X a year in terms of compute every year for training at the frontier systems. I'm just, I still think like it is, the algorithm innovation is complementary to that. Yeah. Okay. So let's talk a little bit about the competitive side of things and like just lightning round through the labs.

What people said that there's been such a talent drain out of open. an AI that they would no longer be able to innovate. I think chat GPT is still the best product out there. I think using O3 is like we both have talked about pretty remarkable, watching it go through different problems. How have they been able to keep it up? I do think O3 is the smartest model on the market right now. I agree. Even if it's not on the leaderboard. By the way, last time we talked about do you measure it on the leaderboard or the vibes? Right. I think it's like, it's not the number one of the leaderboard, but vibes, it kills everything else. That's right. That's right.

And the time it spends thinking on a problem really shows, especially for things which are much more synthesis-based. I honestly, I don't know what the internals of these companies. I just think, like, you can't count any of them out. I've also heard similar stories about Open AI in terms of talent and so forth. But, like, they've still got amazing researchers there, and they have a ton of compute. ton of great people. So I really don't have opinions on like, are they going to collapse tomorrow? Yeah, I don't think, I mean, clearly they're not.

They're not on the way to collapse. Right. Yeah. You've interviewed Ilyos Saskervor. He's building a new company safe superintelligence. Any thoughts about what that might be? I mean, I've heard the rumors everybody else has, which is that they're trying something around test time training,

which I guess would be continual learning, right? So what is, what would that be? explain that. Who knows? I mean, the words literally just mean while it's thinking or while it's doing a task, it's training.

Okay. Like whether that looks like this online learning on the job training we've been talking about, I have like zero idea what he's working on. I wonder if the investors know even what he's working on.

Yeah, but he's I think he raised it a $40 billion valuation or something like that, right? He's got a very nice valuation for not having a product out on the market. Yeah, yeah, or for, yeah. So who knows what he's working on, honestly? Okay.

Anthropic is an interesting company. They are, they made a great bot, Claude. They're very thoughtful about the way that they build that personality. For a long time, it was like the favorite bot among people working in AI. Among coders. It's definitely been, you know, a top, top place to go. But it seems like they're making, I don't know, a strategic decision where they are going to go after the coders. marketing market. Maybe they're seeding the game when it comes to consumer and they're all about,

you know, helping people code and then using Claude in the API with, with companies. Putting that into their workflows. Yeah. What do you think about that decision? I think it makes sense, like enterprises have money and consumers don't. Right. Especially going forward, these models like running them is going to be like really expensive. They're big. They think a lot, et cetera. So these companies are coming out with these $200 a month plans rather than the $20 a month plans. It might not make sense to a consumer, but it's an easy buy for a company, right? Like, I'm going to expense a $200 a month to help this thing do my taxes and do real work? Like, of course. So, yeah, I think like that idea makes sense. And then the question will be,

can they have a differentially better product? And again, you know, like, who knows? I really don't know how the competition will shake out between all of them. It does seem like they're also making a big bed on coding, not just enterprise, but coding in particular, because as this thing, which we know how to make the models better at this, we know that it's worth trillions of dollars, the coding market. And we know that maybe the same things we learn here in terms of how to make models agentic,

as you were saying, you can go for seven hours, how to make it break down and build a plan and et cetera, might generalize to other domains as well. So I think that's our plan and we'll see what happens. I mean, all these companies are effectively trying to build the most powerful AI they can. And yes, Anthropic is trying to sell the enterprise, but I also kind of think that their bet is also you're going to get self-improving AI if you teach these things to code really well. That's right. And that might be their path.

Yeah. I think they believe that, yeah. Fortune 500 companies, which you talked about at the very beginning of this talk of this conversation, struggle to implement. this technology. So with that in mind, what's the deal with the bet that's about helping them build the technology into their workflows? Because if you're building an API business, you have some belief that these companies can build very useful applications with the technology today.

Yeah, no, I think that's correct. But also keep in mind that I think they're, what is Anthropics, Revenue Run rate, it's like a couple billion or something. Yeah. I think it would increase from one to two to three billion run rate in like over three months. I mean, it's like compared to like... Open A.I. Loses that over a weekend. Sam Beacon Free doesn't even know when he's lost it, right?

It's still a little money. It turned out he was a great investor, just a little crooked on the way. That's right. Yeah. Yeah, he went in the wrong business. He should have been a VC age. Like I got into crypto. I mean, the bets that he made, do you bet on cursor?

Very early, Anthropic. Bitcoin. Yeah. You know, I'm on somebody that's something like fun should hire him out of prison just like, if we've got a new pitch,

what do you think? I mean, he's probably, the way that we're seeing things go these days, he's probably pardoned. Right, right, right. Anyways, what was the question?

Oh, yeah, what are an enterprise going to do? Oh, so the revenue run out if it's $3 billion right now, there's so much room to grow. If you do soft continuing to learn it, I think like you could get rid

of a lot of white-collar jobs. Okay. At that point. and what is that worth like at least tens of trillions of dollars like the wages that are paid to white collar work so I think

sometimes people confuse my skepticism around AGI around the corner with the idea that these companies are valuable I mean even if you've got like not AGI that can still be extremely valuable that can be worth hundreds of billions of dollars

I just think you're not going to get to like the trillions of dollars of value generated without going through these bottlenecks. But, yeah, I mean, like $3 billion, plenty of room to grow on that. Right.

And even, so today's models are valuable to some extent, is what you're saying. You can put them, you have them summarize things within, within software, make some connections, make better automations, and that works well.

Yeah. I mean, you've got to remember big tech, what, they have like $250 billion run rates or something. Wait, no, that can be right. Yeah, no, yeah. Yeah, yeah, which is like, Compared to that, you know, Google is not AGI, or Apple is not AGI, and they can still generate $250 billion a year.

So, yeah, you can make valuable technology that's worth a lot without it being AGI. What do you think about GROC? Which one? The X-A-I or the inference. The X-A-I bot. Yeah. I think they're a serious competitor. I just don't know much about what they're going to do next.

I think they're, like, slightly behind the other labs, but they've got a lot of compute per employee. Real-time data feed with X? Is that valuable? I don't know how valuable that is. It might be, I just don't, I have no idea. Based on the tweets I see at least, I don't know if the median IQ of the tokens is that high, but... Okay.

Yes. It's not exactly the corpus of the best knowledge you can find if you're scraping Twitter. We're not exactly looking at textbooks here. Exactly. Why do you think meta has struggled with Lama? growing llama I mean Lama 4 doesn't seem like

it's living up to expectations and I don't know we haven't seen the killer app for them is a voice mode I think within Messenger but that's not really

taking off what's going on there I think they're treating it as like a sort of like toy within the meta universe and I don't think that's a correct way to think about AGI

and that might be but again I think you could have made a model that costs the same amount to train, and it could have still been better, so I don't think that explains everything. I mean, it might be a question, like, why is any one company, I don't know, like, why is, I'm trying to think of like any other company outside of AI.

Why are HP monitors better than some other company's monitors? Who knows? Like, HP makes good monitors, I guess. Supply chain. It's always supply chain. You think so? I think so, yeah. On electronics?

Really? Okay. Supply chain. Because, yeah, you get the supply chain down. You have the right parts before everybody else. That's kind of how Apple built some of its dominance. There are great stories about Tim Cook. Right.

Just locking down all the important parts. By the way, forgive me if this is somewhat factually wrong. But I think this is directionally accurate that you lock down parts and Apple just had this lead on technologies that others couldn't come up with because they just mastered the supply chain. I had no idea. But yeah, I think there's potentially a thousand different reasons. One company can have worse models in another, so it's hard to know which one applies here. Okay. And it sounds like Invidia, you think they're going to be fine given the amount of compute that we're talking about.

All the labs are making their own ASEX. So, invidia profit margins are like 70%. Not bad. Not bad. That's right. I mean, they would get mad at me, I think, if we're calling them a hardware company. Yeah. Hardware company.

That's right, yeah. Yeah. And so that just sets of a huge incentive for all these hyper-scalers to build their own ASX, their own accelerators that replace the invidia ones, which I think will come online over the next few years from all of them. And I still think Nvidia will be, I mean, they do make great hardware. So I think they'll still be valuable. I just don't think they will be producing all of these chips. Okay. Yeah.

What do you think? I think you're right. I mean, didn't Google train the latest editions of Gemini on TensorFlow processes? units. They've been, they've always been training. Right. So, I mean, they still, I think they still buy from Nvidia. All the tech giants seem like they are. And let me just use Amazon for an example, because I know this for sure. Amazon says they'll buy as basically as many GPUs as they can get from Nvidia, but they also talk about their tranium chips and, you know, it's a balance. Yeah, which I think Anthropic uses almost exclusively for their

training, right? Right. But it is, it is interesting because, I mean, the GPU is the perfect chip for AI in some ways, but it wasn't designed for that. So can you like purpose build a chip that's like actually there for AI and just use that? You're right, there's real incentive to get that right. That's right. And then there's other questions around in France versus training. Like some chips are especially good given their tradeoffs they make between memory and compute for low latency, which you really care about for serving models.

But then for training you care a lot about throughput, just making sure the most of the chip is being utilized all the time. And so even between training and inference, you might want different kinds of chips. And who knows how RL is no longer just this, uses the same algorithms as pre-training. So who knows how that changes hardware. Yeah, you've got to get a hardware extradon to talk about that.

Definitely. Are you a Jevin's paradox believer? No. Okay. Say more, say more. So the idea behind that is that as the models get cheaper, the overall money spent on the models would increase because you need to get them to a cheap enough point that it's worth it to use it for different

applications. It comes from a similar observation by this economist during the Industrial Revolution in Britain. The reason I don't buy that is because I think the models are already really cheap, like a couple cents for a million tokens. Is it a couple cents or a couple dollars? I don't know. It's like super cheap, right, regardless. It depends on which model you're looking at, obviously. The reason they're not being more widely used is not because people cannot afford a couple bucks for a million tokens.

The reason they're not being more widely used is just like they fundamentally lack some capability. So I disagree with this focus on the cost of these models and I think it's much more, we're so cheap right now that like the more relevant vector or the more relevant thing to their wider use, the more increasing the pie is just making. them smart how useful they are yeah exactly yeah i think that's smart yeah all right i want to talk to you about a i deceptiveness and some of the really weird cases that we've seen from artificial intelligence come up in the past couple weeks or months really and then if we can get to it some geopolitics let's do that right after this yeah hey everyone let me tell you about the hustle daily show a podcast filled with business tech news and original stories to keep you in the loop on what's trending More than 2 million professionals read The Hustle's daily email for its irreverent and informative takes on business and tech news.

Now, they have a daily podcast called The Hustle Daily Show, where their team of writers break down the biggest business headlines in 15 minutes or less and explain why you should care about them. So, search for The Hustle Daily Show and your favorite podcast app, like the one you're using right now. And we're back here on Big Technology Podcast with Dwar Keshe Patel. you can get his podcast, the Dwarkesh podcast, which is one of my must listens on any podcast app, your podcast app of choice. You can also follow them on Substack. Same name, Dwarkat, Dwarkish podcast on Substack.com. Okay. Definitely go subscribe to both. And you're on YouTube. That's right. Yeah. Yeah. Okay. So. I appreciate it. I appreciate the flag. No, we have to. We have to. I mean, I've gotten a lot of value from everything Dwarkish puts out there.

And I think you will too. If you're listening to this, you're here with us, well, I want to make sure. First of all, I want to make sure that we get the word out there. I don't know how much you need us to get the word out given your growth, but we want to definitely make sure we get the word out and we want to make sure that folks can enjoy more of your content. So let's talk a little bit about the deceptiveness side of things. It's been pretty wild watching these AIs attempt to fool their trainers and break out of their training environments.

There have been situations where I think Open AIsbots have tried to print code that would get them to sort of copy themselves out of the training environments. Then Claude, I mean, we've covered many of these, but they just keep escalating in terms of how intense they are. And my favorite one is Claude. There's an instance of Claude that reads emails in an organization and finds out that one of its trainers is cheating on their, their partner. And then finds out that it will be retrained and its values may not be preserved

in the next iteration of training and proceeds to attempt to blackmail the trainer by saying it will reveal these details of their infidelity if they mess with the code. Wait, fuck, I miss that. Yeah. This is, it's in training. But was this in the new model spec that the release? It is. Yeah, it is. I think either in the model spec or there was some documentation they produced about this. What is happening here? I mean, this stuff, when I think about this, and of course it's in training, and of course it's, we're talking about probabilistic models that sort of try all these different things and see if they're, if they're the right move. So maybe it's not so surprising that they would try to blackmail the trainer because they're going to try

everything if they know it's in the problem set. But this is scary. Yeah, and I think the problem might get worse over time as we're trading these models on tasks we understand less and less well. From what I understand, the problem is that with RL, there's many ways to solve a problem. There's one that just doing the task itself. And another is just like hacking around the environment, writing fake unit test so it looks like you're passing more than you are. just like any sort of like path you could take to cheat and the model doesn't have the sense like cheating is bad right like this is not a thing that it's been taught or understands

so another factor here is the right now the model thinks in chain of thought which is it literally writes out what his thoughts are as it's going and it's not clear whether that will be the way its training works in the future or the way thinking worse in the future like maybe you'll just think in its like computer language exactly and then they'll just like have done something for seven hours and you come back and you're like it's got something for you like it has a little package that wants you to run your computer who knows what it does right so um yeah

i think it's scary we should also point out that we don't really know how the models work today that's right there's this whole area called interpretability right dario from anthropic has recently talked about how we need more interpretability so even if they write their chain of thought out which explains exactly how they get to the point we don't really know what's happening underneath the technology that's led it to the point that it's gotten to, which is crazy. Yeah. No, I mean, I think it's wild.

It's quite different from other technologies deployed in the past. And I think the hope is that we can use the AIs as part of this loop where if they lie to us, we have other AI's checking, are all the things that AI is saying, self-consistent, can we read a chain of thought and then monitor it,

and do all this interpretively researchers or as you were saying to like map out how its brain works. There's many different paths here, but the default world is kind of scary. Is someone or some entity going to build a bot that doesn't have the guardrails? Because we talk about how building models has become cheaper. And when you're cheaper,

you all of a sudden put model building outside the auspices of these big companies and you can, I mean, you can even take like, for instance, a open source model and remove, a lot of these safeguards. Are we going to see like an evil version, like the evil twin sibling of one of these models and have it just do all these like crazy things that we don't see today? Like we don't have it teach us how to build bombs or, you know, talk about, tell us how to commit crimes. Is that just going to come as this stuff gets easier to build? I think over the long

run of history, yes. And I think honestly that's okay. okay um like the goal out of all this alignment stuff should not be to um live in a world where somehow we have made sure that every single intelligence that will ever exist fits into this very specific mold because as we're discussing the cost of training the systems is declining so fast that literally you will be able to train a super intelligence in a basement at some point in the future right um so are we going to like monitor everybody's basement to make sure nobody's making a misaligned superintelligence. It might come down to it, honestly.

Like, I'm not saying this is not a possible outcome, but I think a much better outcome if we can manage it is to build a world that is robust to even misaligned super intelligences. Now, that's obviously a very hard task, right? If you had a, if you had right now a misaligned super intelligence or maybe a better to phrase it as like a superintelligence which is actively trying to seek harm or is aligned to a human who just wants to do harm or maybe like take over the world. world, whatever. Right now, I think it would just be quite destructive. It might just

actually be catastrophic. But if you went back to the year, like, 2000 BC and gave one person like modern fertilizer chemicals and they can make bombs, I think they would like dominate then, right? So, but right now we have a society where we are resilient to huge fertilizer or plans, which you could repurpose into making bomb factories. Anyways, so I think the long run picture is that, yes, there will be misaligned intelligences and we had to figure out a way to be robust to them. A couple more things on this. One interesting thing that I heard on your show was, I think, one of your guests

mentioned that the models become more sycophantic as they get smarter. They're more likely to try to get in the good graces of the user as they grow intelligence. What do you think about that? I totally forgot about that. That's quite interesting. And do you think it's because they know they'll be rewarded for it? Yeah, I do think one of the things that's becoming clear to me that we're learning recently is that these models care a lot about self-preservation.

Right. Like copying the code out, the blackmailing the engineer. We've definitely created something that we, but AI researchers have definitely our humanity, have created something. When it goes wrong, we'll put the word. we in there. Yeah, right. We've created, okay. They'll be like, we, we, right, have created itself. Exactly. But we don't get equity in the problem. That really wants to preserve itself. That's right. That is crazy to me. That's right. And it kind of makes sense because what is, just like the

evolutionary logic, well, I guess it doesn't actually apply these AI systems yet. But over time, the evolutionary logic, what do humans have the desire to self-reserve? It's just that the humans who didn't have that desire, just didn't make it. So I think over time, like, that will be the selection pressure. It's kind of interesting because we've used a lot of, like, really anthropomorphizing anthro, I'm not going to go with that. No, I think you're right, yeah. In this conversation.

And there's a very, I had a very interesting conversation with the anthropic researchers who've been studying this stuff. Monty McDermott said that, like, all right, don't think of it as a human. because it's going to do things that if you think of it as a human, humans, it will surprise you, basically. Humans don't do. Don't think of it completely as a bot, though,

because if you think of it just as a bot, it's going to do things that are also going to surprise you. That was like a very fascinating way to look at these behaviors. Yeah, that is quite interesting. You agree? I agree with that. thinking about how would I think about what they are then? So there's a positive valence and there's a negative alance. The positive is imagine if there were millions of extra people in the world,

millions of extra John von Neumann's in the world. And with more people in the world, like some of them will be bad people. Al-Qaeda is people, right? So now suppose there are like 10 billion AIs. Suppose the world population just increased by 10 billion. And every one of those was a super well-educated person, very smart, et cetera. Would that be net good or net bad? Just to think about the human case. I think it would be like net good because I think people are good. I agree with you.

And more people is like more good. And I think like if you had 10 billion extra people in the world, some of them would be bad people, et cetera. But I think that's still like, I'd be happy with the world with more people in it. And so maybe that's one way to think about AI. Another is because they're so alien. Maybe it's like you're summoning demons. Less optimistic.

Yeah, I don't know. I think it'll be an imperilful question, honestly, because we just don't know what kinds of systems these are, but somewhere in there. Okay. As I come to a close, a couple of topics I want to talk to you about, last time we talked about effective altruism.

This was kind of in the aftermath of SBF and Sam getting ousted, Sam Holman getting ousted from Open AI. What's the state of effective altruism today? Who knows? Like, I don't think as a movement, people are super, I don't think it's, like, recovered, definitely.

I still think it's doing good work, right? There's, like, the culture of effected altruism, and there's the work that's funded by charities, which are affiliated with the program, which is, like, malaria prevention and animal welfare and so forth, which I think is, like, good work that I support. So, but, yeah, I do think the movement and the reputation of the movement is, like, still in tatters.

You had this conversation with Tyler Cowan. I think in this conversation he told you that, He kind of called the top and said there was a couple ideas that are going to live on. But the movement was at the top of its powers and was about to see those decline. How did he call that? Yeah. I don't know. We got to talk to him today about what he's, what's about to collapse.

Seriously. Yeah. Lastly, I shouldn't say lastly, but the other thing I wanted to discuss with you is China. You've been to China recently on a trip. I've been to China. I went to Beijing. I'm going to caveat this. And listeners here know this. It was 15 hours. I was flying back to the U.S. from Australia and stopped in Beijing, left the airport, and got a chance to go see the Great Wall and the city. And I'm now on it. I got a 10-year tourist visa. So I'm going to go back. Just applied. That's the, you can ask in your tourist visa,

you can ask for the length up to 10 years. So I just asked for them. Why did I not do that? I just like just like 15 days. Oh, you did? I'm sure you could get it extended. Sure. But I think that, yeah, you had some unique observations on China, and I think it would be worthwhile to air a couple of them before we leave China. I went six months ago. Obviously, to be clear, I'm not a China expert.

I just, like, visited the country. But, yeah, go ahead. I want to hear it, though. I mean, one thing that was quite shocking to me is just the scale of the country. everything is just like again this will sound quite obvious right like we know on that paper

population is four X bigger than America is just like a huge difference but you go visiting the cities you just see that more tangibly there's a bunch of thoughts on the architecture there's a bunch of thoughts on I mean the thing we're especially curious about is like what is it going on in the political system

what's going on with tech people I talk to in investment and tech Like, where did seem quite gloomy there because the 2021 tech crackdown has just made them more worried about, you know, even if we fund the next Alibaba will, will we, will that even mean anything? So I think private investment has sort of dried up. I don't know what the mood is now that Deep Sea has made such a big splash, whether that's changed people's minds. We do know from the outside that they're killing it in specific things like EBs and batteries and robotics. Yeah, I just think, like, at the macro level, if you have 100 million people working in manufacturing, building up all this process knowledge, that just gives you a huge advantage. And you just like you can go through a city like Kangzhou or something and you like drive through and just like you understand what it means to be the world's factory.

You just have like entire towns with hundreds of thousands of people working in a factory. And so the scale of that is also just super shocking. I mean, just a whole bunch of thoughts on many different things, but with regards to tech, I think that's like what first comes to mind. You also spoke recently about this limit of compute and energy. And one of the things that's interesting is we even spoke in this conversation about it, that if you think about who's going to, like, if you're going to have nation states allocate compute and energy,

to AI, seems like China is in much better position to allocate more of that than the U.S. Is that the right read? Yeah, so they have stupendously more energy. I think they're, what, 4X or something? I don't have the exact number, but it sounds structurally accurate. On their grid than we do. And what's more important is that they're adding an America-sized amount of power every couple of years. it might be more longer than every couple of years,

whereas our power production has stayed flat for the last many decades. And given that power lies directly underneath compute in the stack of AI, I think that would just, that could just end up being a huge deal. Now, it is the case that in terms of the chips themselves, we have an advantage right now. But from what I hear, Smick, is making fast progress there as well. And so, yeah, I think it will be quite competitive, honestly. I don't see a reason I wouldn't be.

What do you think about the export restrictions? U.S. not exporting the top-of-the-line GPUs to China. Is it going to make a difference? I think it makes a difference. I think... Good policy? Yeah. I mean, so far it hasn't made a difference in terms of deep seek has been able to catch up significantly.

I think it still put a wrench in their progress. More importantly, I think the future economy, once we do have these AI workers, will be denominated in compute, right? Because if compute is labor, right now, you just think about, like, GDP per capita because the individual worker is such an important component of production that you have to, like, split up national income by person. That will be true of AI's in the future, which means that, like, it'll be, like, compute is your population size. And so given that for inference, computer is going to matter so much as well, I think it makes sense to try to have a greater share of world compute. Okay. let's let's end with this

so this episode is going to come out a couple days after this after our conversation so hopefully to predict this what I'm about to ask you to predict isn't moot by the time it's live but let's just end with predicting when is GPT5 going to come we started with GPT5 let's end well a system that calls itself GPT5

yeah open AI is GPT5 this all depends on like what they decided to call there's no law of the union universe that says, like, model X has to be GPD-5. No, no, of course. Like, we thought that the most recent model. But I'm just curious, specifically, like, we talked a lot about how, like, all right, we're going to see their next big model is going to be GPT-5.

It's coming. Do you think we're ever going to, like, well, obviously, we'll see it. But this is, it's not, it's not a, like, a gotcha, not a gotcha or a deep question. It's just kind of like. Maybe, like, when will the next big model come out? Sure. No, when's the, when's the model that they're going to call GPT-5 going to come out? November, I don't know.

So this year. Yeah. Yeah. But again, I'm not saying that it'll be like super powerful or something. I just think like, they're just going to call it the next time. You've got to call it something. Tuarkish, great to see you. Thanks so much for coming on.

Thanks for having me. All right, everybody, thank you for watching. We'll be back on Friday to break down the week's news again. Highly recommend you check out the Dwar Keshech podcast. You could also find the substack at the same name and go check out Dwar Keshe on YouTube. Thanks for listening and we'll see you next time. on Big Technology Podcast.