Big Technology Podcast - AI Scaling, Alignment, and the Path to Superintelligence — With Dwarkesh Patel

Episode Date: May 15, 2024

Dwarkesh Patel is the host of the Dwarkesh Podcast, where he's interviewed Mark Zuckerberg, Ilya Sustkever, Dario Amodei, and more AI leaders. Patel joins Big Technology to discuss the current state and future trajectory of AI development, including the potential for artificial general intelligence (AGI) and superintelligence. Tune in to hear Patel's insights on key issues like AI scaling, alignment, safety, and governance, as well as his perspective on the competitive landscape of the AI industry. We also cover the influence of the effective altruism movement on Patel's thinking, his podcast strategy, and the challenges and opportunities ahead as AI systems become more advanced. Listen for a wide-ranging and insightful conversation that grapples with some of the most important questions of our technological age. ---- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Want a discount for Big Technology on Substack? Here’s 40% off for the first year: https://tinyurl.com/bigtechnology Questions? Feedback? Write to: [e-mail]

One of the sharpest minds in AI joins us to look at the research, the business, the dangers and the conspiracies, all coming up right after this. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We're joined today by Dwarkesh Patel. He's the host of the Dwar Kesheh podcast, and he's had DeepMind CEO Demis Saba-San recently, along with Anthropic CEO Dario Mode, the Open AI chief scientist, Ilya Setskever, and recently met a CEO Mark Zuckerberg. So that's the person we're dealing with here today.

To Arkesh, welcome to the show. Thanks for having me. Super excited to be on, Alex. I think that you've been doing some great interviews on AI, looking at the research, where things are going, and really asking the right questions to the right people. So you've interviewed Ilius Discover, Open AI's chief scientists, or really, I should say,

it's former chief scientist after he parted ways with the company yesterday. How important was Elia to Open AI's ability to see, stay competitive in the AI field? I actually think that's a really interesting question. I remember I was chatting with someone and they said something along the lines of, well, now that they've lost Dahlia, you know, I think the great people matter a lot. And since this person has lost, it might be downhill for open AI. And then, you know, I think like the default perspective is, listen, you've got thousands of

scientists who are doing AI. Surely any one of them is replaceable. I think that's probably correct, but I'm not in the field enough to know that. And it is a sort of interesting empirical question of what is the bus factor of a place like Open AI? If you lose a chief scientist, how much does that slow down your progress? And it would be interesting if it doesn't slow it down that much. It would be super interesting if it slows it down a lot.

But yeah, that's a really interesting question. Have you seen any signs of them slowing since his departure? No. I mean, the big question people have had is since she's, GPD4, which was released more than a year ago, the, we haven't really gotten anything better, right? So we've gotten Claude 3, we've gotten Gemini. They're not significantly better, if at all, than GPD4, and certainly not the newer version of GPD4. So the question is, is AI progress plateauing or are people just waiting to build out the giant data centers,

which are necessary for training a GPD-5 level model? And actually, I think this year will be super interesting in terms of. learning about AI because by the end of this year, we'll get to see hopefully what a GPD5 level model looks like. And we'll learn whether we're on the path to some kind of superintelligence. If GPD5 is so amazing, we're like, okay, well, we're building a god of fewer is out. Or if GPD5 is not that much better than four, and I think the main thing we're going to learn is between 4.5 and 5 level models, we're going to hit what's known as the data wall, which is to say that as you make these models bigger, you need more and more data.

to keep training them. And we're running out of internet data. And so we're going to learn whether synthetic data, RL, these other techniques can substitute for the data that bigger models will need. And I mean, by the end of this year, I'm expecting to learn a lot about what the course of AI is going to look like. What is your sense as to what the answer will be with GPT5? Here's some predictions I have. I'm not sure if that's gets to the heart of what will be impressive about it. I think it'll definitely be better at reasoning, which is trivial to say, because the training methods that we've seen them talk about, like, I'm sure you heard to talk about Q Star, and what it seems to be is training the model to rewarding it on getting

the correct reasoning trace to get the right answer, and that seems to lead to better reasoning, or at least in public, there's equivalent papers that are released publicly called Quiet Star that claims that it does, then we're going to see much more multimodal. data and I think that'll look a lot like the equivalent of supervised fine tuning, but for a bunch of people recording their screen and doing workflows with their screen, navigating UIs. So I think you'll have agents that can act coherently as your assistant for potentially minutes, if not hours on end, in a way that you can just tell them to go do something on the internet and they can actually do it.

Where like GPD4 right now, web browsing isn't really a big feature, like people like perplexity use it, but it's mostly to summarize. It's not to like actively go out and look for information. So I'm hoping, well, I'm expecting that to be one of the things you see with GP5. I guess the big question is like how much smarter is it, right? Like I'm mentioning all these off abilities it might have, but like how much juice will I have? And I honestly don't know.

Right. And so how do you think we're going to be able to assess that? Like, is it just going to be like how much better it does on tests or do you think it's just going to be a feel when people are using the model? Yeah, I trust the feel more, honestly. It's so interesting. We have these evaluations. but what's your sense on them?

I mean, like, people will come out and say, here's what we got in MMLU and so on, but they're getting saturated and they're not often not that great to begin with. So I'm more eager to see what it feels like to talk to one of these things than learn what its MMLE score is. Yeah, I agree.

I mean, it is interesting how Fiel plays such a big role into evaluating these models because they are, you talk about this a lot, scientific, right? And we can sort of test them scientific, and we have these big things like parameters and, you know, the size of the compute that we use. But ultimately, feel is like one of the great things that we use to test how good these models are. And in fact, like, the thing that people always look at in terms of model performance is this chatbot arena where they put two answers from different chatbots side by side and say, okay, well, which one is the best?

And that's kind of, it seems janky, but it's also the thing that people take almost as gospel now in the AI industry. Yeah. And in fact, I think even chat about Arena has some deficiencies in terms of evaluation because from what I understand, you're doing these pairwise comparisons, but you're doing them, you ask a question and two of them respond. And what I'm more curious about is what is it like to have a long form conversation with this thing, where it's not just what is the immediate response like? But if we keep talking, you know, can you kind of like understand the context of the problem I'm talking about? Can you keep following up on different threads? That's more relevant to my workflow than just immediately producing some bullet points. Yeah. We're going to get into this evaluation and research a little bit more as we keep going. But just to go back to OpenAI, it is interesting to watch Microsoft now start to develop their own models, almost to compete with OpenAI. And there was a story in the information recently that Microsoft is working on a 500 billion parameter model just for context. This is kind of like the size of the model.

And OpenAI's GPT4 was apparently trained on something like a trillion parameters or apparently has a trillion parameters. But Microsoft is now making this move where it's trying to build, I think for the first time since GPT4, its own model that competes with it. How should we read into what Microsoft is doing there? Do you think it is a loss of faith in OpenAI, it's a hedge against Open AI, sort of a necessary move, even though it has such an important partner? What's your take? Yeah. One of my friends who works at DeepMind told me that Microsoft is basically reversing what Google has managed to do over the last few years. And in fact, making the same mistake that Google initially made, which was to have its training distributed or split up between two different corporations or institutions. For Google, it was a brain, Google Brain and DeepMind. And so Microsoft has a company which is in the lead, right, open the eye. and I guess instead of doubling down on it they're trying to hedge their bets in this way

I think if you think that AI is like another product where you have multiple vendors so you can be sure that if one of them decides to go to a different route you have some leverage over them that might make sense for another kind of product the thing with AI is if you buy scaling in this picture

that is you make the models bigger they get much smarter then I don't think it makes sense to hedge your bets in this way I think you should just double down Give one of them $100 billion and just say, like, go make me super intelligence. You know what I mean? Because then you're just splitting up your efforts.

And, yeah, like, it would be much better to have one, GPD4 than two companies that have, you own two companies that have a GPD 3.5. And I'm sure Microsoft knows this. So what do you think could possibly be their reasoning for doing what they're doing? Partly, I think they got spooked by what happened with the board. last December and November. I think definitely that there's that because the clause in the Open AIA Charger says that if the board, which is non-profit and controls the company, decides that we've built

AGI, then Microsoft has no leverage over Open AI anymore. So they got to look by that. And I think partly it's probably Microsoft doesn't buy the scaling hypothesis as much as us internet weirdos. because they probably think like, oh, you know, we want to like, we want to diversify our bets here and we'll have multiple companies build a GPD 4.5 level model and we'll see how that goes. I mean, I'm sure there's better reasons too, right? Like you can build this in-house talent. I'm sure there's a lot of practical knowledge you understand by building smaller models and you're getting a lot of that knowledge in-house by training these models.

And so Microsoft's own ability to train and understand and deploy these models improves. Yeah. And can you just handicap the field for us? I mean, how should we think about the efforts of deep mind versus meta versus anthropic versus open AI? Is there a clear leader there or is there any sort of key differentiators that are important to know? I know it's like a broad question, but feel free to zero in as you as you'd like.

Yeah, that's a good question. It doesn't seem like there's a strong leader at the moment. I think in terms of revenue, probably open AI is leading by a lot. I think just the amount of people use chat GPT versus any other service. But, I mean, subjectively, you use Clod and it's often better, not significantly worse in any case. I think the main way in, at least in which they're today different, is Clod seems to have better post-training, which is to say, which is the jargon for basically saying, like, what kind of personality does it have?

And how does it break down your question? And how does it act, how does it act as a persona of a chatbot? and so all this RLHF stuff you hear is part of the post-training and Klaught has a more automated way of doing that or Anthropic which controls Klaught does. Gemini has longer context obviously but so you know million tokens which is huge I think the big thing we'll probably learn in the next few months

is who's really ahead because everybody's just been really seeing models that are as good as GPD4 right now and we'll learn who can go the distance so to speak and I think the big question will be open AI probably has the compute because of Microsoft, but then again, maybe the Microsoft is splitting up its compute. Google has a, definitely has a compute because, you know, Google, a huge company. The question is, I guess we don't know yet whether Anthropic can keep up beyond this year with models that might cost tens of billions of dollars.

Right. And I think that's sort of the underappreciated part of Google's attempt here is that they are doing this reverse Microsoft or correcting their mistakes like we talked about, where they've brought Google Brain and DeepMind together under one organization and said, you know, resource, you know, your internal conflicts be damned. Resources are going to be pooled right now. Yeah. You know, I interviewed the guy who wrote The Making of the Manhattan Project, or sorry, the making of the atomic bomb Richard Rhodes. And he was telling me when I interviewed him about Soviets after Hiroshima and Nagasaki, Stalin called his top physicists. And he said,

comrade you will give us the bomb you have the entire resources of the state at your disposal you will give us the bomb or you and your family will be camp dust and i'm sure the last sentence wasn't uttered inside of google but that maybe the attitude they've taken in terms of their compute allocation might be much in favor of we're going to take this seriously we're going to invest a bunch of compute into making this happen and also i think you shouldn't ignore the fact that google is the company that actually has a successful accelerator program for AI chips already with their TPUs, which other companies are trying but don't yet have to replace, you know, Nvidia GPUs. Right. Okay. Lastly, XAI, which is Elon Musk's

effort. Do you think that there's any chance that they can be competitive here? I mean, we think about resources and they definitely have Musk's money. And they have, I think there's like this big GPU cluster that Tesla has. So I wonder if that's going to factor. And then is there any other dark horse that might come in and start to matter. I think the second party question is super interesting. I mean, on XAA, I honestly don't know. I mean, Tesla is separate organization than XAI, so I don't know how much those could transfer, but I have no idea.

With regards to who the other actors could be, I think people are in the case where AI really is super powerful, where Jupy-5 is amazing. I think what happens is a lot of different countries, national security apparatus, start to realize what a big deal this is, and they're not just going to sit around for people to like, you know, they're going to make moves. And I think what that looks like is there's a lot of different countries in the world with a sovereign wealth fund with $100 billion, right? And do they all just go around saying, like, what are we doing sitting on this money? Obviously, AI is the thing to do. Each, you know, the UAE spins up, which I hear they're already doing a bunch of data centers

in the Middle East to start training even bigger models. And China starts deploying energy infrastructure to do big training runs. So I think in the world where AI continues to get much better at a fast pace, I think you're looking at a much more involved. I guess I'm trying to say the players will be nation state level players because that's also the kind of funding you'll need to keep scaling these models. And what does a nation state do with this technology? Yeah.

I mean, obviously the military uses are clear, or at least it will be clear, right? You can use it for R&D on military stuff. but I mean just like basic things like you have a drone operator who's human level who can they know you can just mass manufacture millions of drones and like a human equivalent model that's on running locally you can run these drones and you have a million drone swarm headed towards Beijing or something I don't know that's just one example but you can imagine there's a lot of things you can do I mean the stepping back the bigger picture is why are some countries wealthier and more powerful than other countries well it's often because they have

more people, right? So, Taiwan, Taiwan would lose a war against China. Why is that? Without the help of the U.S., but because China has more people. If AI substitutes for people can increase the effective population of a country, then you can imagine that it would just be a huge leverage that a country would have over other countries in terms of its own economic output or even its ability to withstand geopolitical competition. You've referred to AI as, you know, potentially like the last invention, I think I might be cribbing a little bit. But what happens if a nation state is the one that achieves artificial general intelligence? And we're going to talk about more, more about AGI in a bit. But let's say, you know, China is able to invent it. Then what happens?

Are there implications there? Totally. I mean, I, for that particular phrase of the last invention, I think, belongs to somebody else. But yeah, it really matters who wins here. I mean, if China wins, it depends on how fast things happen in the world where they happen within a few years I think what you're looking at is China has a ton of leverage over the United States because one of the things the future AI could unlock is things like pocket nukes right and so you if China is ahead they could be like listen we've got these we've just basically got this mass army that would be able to manufacture of billions of extra soldiers and And we can build mass manufacturer drones or robots or whatever for them to run on.

And I think that gives them a lot of leverage, right? So I would worry about that. I think it's important that the U.S. win that and stay ahead. I don't know the status of Chinese AI currently. It seems like the newest model was a deep sequence one. It seems like it was really good. So before we get there, we're going to have a lot of, I mean, before anyone gets there, the tech industry in particular is going to have a lot to work through.

And you already mentioned a little bit. We have data constraints. We also have compute constraints. I think we should talk a little bit more about whether these resource constraints are actually going to be things that matter and how they might factor. And Mark Zuckerberg spoke with you about, about how energy is going to be something. We'll talk about that. And that really opened my eyes to sort of like, oh, are we going to be like hitting a wall here with AI and wrote about it in the newsletter? Sort of an interesting question.

So why don't we just go into the component parts and talk a little bit about each? And the first one, and I think, you know, clearly a very important one is compute. And I start to like raise my antennas here when I hear rumors of Sam Altman wanting to raise $7 trillion. And of course, that's like an economic question. But it's also like, if that's what it's going to take to make this stuff work, then are we ever going to get to the place where a lot of these people want to get to talking about like adding more compute and data and energy? and eventually you get to the point where you can train better large language models

and see what the scaling law really looks like at its limit. What do you think? Yeah, I think compute will be less of a bottleneck than energy. It's where the $7 trillion. Yeah, I imagine, well, backing up, the reason I think compute will be a lesser bottleneck than energy is because right now you have one company in Vida which is making the sort of flagship GPUs.

And other than Google, nobody has a clear competitor. And so the thing that was bottlenecking Nvidia so far is that some of their components that the need for these GPUs, co-os and HBM, they just weren't able to get enough allocation or get TSM to build facilities for these because TSMC was like, I don't know if we buy all this AI stuff. Because then they had to make this huge investment into building it out. But now it seems like the fabs are building it out.

And also all these companies have accelerator programs where they're going to try to ship their own chips. So I think compute will become more and more available. And that's what Zuckerberg said on the podcast, that now the compute constraints are decreasing. Then the question that Zuckerberg pointed to was, well, will there be energy? And the key constraint with energy is not necessarily, is there enough energy in the world, but more so for training, is there enough enough energy in one place because to do a training run it has to usually at least from what it seems like publicly the training methods we have you got to do it in one place so um if a nuclear power plan releases one gigawatt of energy can you and if you know training with hundreds of thousands of

GPUs would take um would consume one gigawatt of energy then do you have can you like get all the energy into one place where in the america is that place if not in america where where do you go do you go the Middle East. You like get some aluminum refinery in Canada because those consume a lot of energy and you can just like buy out the aluminum refinery and turn that offline. And I don't know, but you can try out different ideas. But I think energy will be the big constraint. And you basically Zuckerberg talked about the fact that you might need a moderately sized nuclear power plant to be able to do this. And you asked, well, what about Amazon? And he said, ask Amazon. Then I asked Amazon did a little research, right?

It actually, Amazon actually has purchased a nuclear, a small nuclear power plant in Pennsylvania. Correct me if I'm wrong here. And it's 960 megawatts, so close to that gigawatt size. And they're gonna use 40% of the energy there. I assume for AI training. So is that sort of what this energy battle looks like moving forward? And is that even enough energy given that everybody wants

to add more compute and more data and more energy into the process to actually be able to build these models. Well, it's certainly not enough. And in fact, I brought that, yeah, I brought that up with Zuckerberg. You need some in one place for the training, it seems like, but they might have ways to get around that. Then you also need to deploy these models.

And those you can, so wherever you deploy the model, it doesn't necessarily, you don't need like a huge amount of energy in one place necessarily, but you do need a lot of different places that each consume energy. And that could result in the demand for energy increasing globally. I can pull this up somewhere, but I did some back at the envelope calculations on if you believe the scaling laws, then you believe the, you can like just look at how much energy does an H100 consume. Then you can look at every generation, how much cheaper or how much less energy because of efficiency gains do we get in terms of these GPUs. Anyways, you can just like go down the list of and then how much basically it would cost.

in terms of energy to train a GPD4 level model, 4.5 level model, 5, whatever. And you get into the gigawash pretty soon. And especially if these models are going to be widely deployed in the world, then, yeah, it's going to consume a ton of energy. One last thing is data. And you mentioned it right at the start, which is that data might be a major constraint. I mean, these companies are already going to are already working with synthetic data, like to train Lama 3, meta-use synthetic data, like data created from basically, you know,

from AI itself. and this is they try to buy Simon and Schuster or they talked about buying Simon and Schuster in the company when they like were like we cannot get this to be as good as chat GPT and by the way this is meta the company that owns Facebook which has like the entire social internet to train on so how do we get around or how does the industry get around that yeah I mean the synthetic data thing comes back to energy and compute in a way right because well how do you make synthetic data you use the existing model what does it take to or use the existing model. It takes energy and compute. So in fact, it will make training more

expensive because instead of just doing one backward pass, you now potentially have to do many forward passes because at each forward pass are going to come up with some output. Then the model has to decide which of those outputs was the best. Now we're going to train on the best of those outputs. It could like be a 5x tax on training. So I mean, that's separate from the question of Are these model methods scalable enough such that they can make the model smarter? And I don't think we have public evidence of this yet. But I don't know. What's your vibe on this?

Because when I talk to researchers at these lab, they seem pretty confident that this will happen. There's no evidence that, I mean, yeah, synthetic data, obviously, the MET, Islam of 3, they said they used it and so forth. But actually, like, really making it smarter in a significant way, I guess we don't have that much evidence for it. I mean, I think I'm learning as we're talking here. and sort of thinking about it, thinking it through and being like, okay, so just like, look at the headlines we've seen, seven trillion dollars for compute. I mean, of course, but we might get more efficient, nuclear power plant for energy, more data than the world possibly has. And then I'm like, how, and we're not quite sure whether scaling these models up, right, adding more energy and more compute and more data into whatever we're training LLMs or the industry is training LLMs, you know, to make them better. We're not sure if that's going to work. And I'm just like, how is this sustainable?

Right. I mean, the thing you had to add on top of that is, what is the revenue of these AI companies so far? And it's actually, I'm guessing it's not great. Like, probably on the order of billions of dollars cumulatively across the industry. And they probably want tens of, I mean, I guess Sam Altman wants $7 trillion of winning.

But you know what I mean? So, like, the difference between how much. So I think it will depend on whether other, hyper-scalers do big companies like Amazon, Meta, Google, Microsoft, buy that this is the path to go on and investing a lot of money into. And it seems like they do. Maybe at some point they stop because the models, maybe GPD-5 isn't that much better, and so they lose their patience. And then I guess a nation-states never get into it either. But then the question fundamentally is, I think in the world where you can get an AGI for $100 billion worth of training, I just like can't see.

why GPD-5 wouldn't be really good and also why people wouldn't continue investing. And in the world where we need much better algorithms or something, yeah, I agree. We might plateau out around here. But that goes back to what we were saying earlier about. We'll learn a lot by the end of the year, what trajectory we're on. Right. And I think that what the industry seems to be betting on is that there's going to be more efficient models. They'll just be able to code them up better.

And so they won't need to take as much compute or data, for instance, to improve, even though they will. have to expand and one of the things that's consistent in your interviews and elsewhere from the people in the industry is that they believe that this stuff is predictable that this scale is predictable totally this was sam altman just a couple weeks ago at stanford he says we can say right now with a high degree of scientific certainty gpt 5 is going to be a lot smarter than gpt 4 gpt 6 is going to be a lot smarter than gpt 5 and we are not near the top of this curve and we kind know what to do. And this is not like it's going to get better in one area. It's not like, it's not that it's always going to be better at this eval or this subject or this modality.

It's just going to be smarter in the general sense. And I think the gravity of that statement is still like underrated. Okay. So like that seems to me to be the case why everybody keeps putting money in to this stuff. It's not necessarily for what it does today. It's what I can do maybe a couple of generations from now, and that will eventually give you the R.I. I'm curious, I mean, you've had these conversations with these folks, you know, at really the ground level of the science. Do you, do you buy this that it's almost going to be a linear progression in terms of how good it is from generation to generation? Yeah, I mean, like one way to think about it is you're mentioning the scaling laws, and that's a relationship that basically, as you've done more

computed in these models, their loss gets better in a very predictable way. And the loss in this case corresponds to their ability to protect internet text. How that translates into capabilities is another question. But if imagine a model that can predict any internet text, it can predict how to write like really great scientific papers or whatever. Well, then that's like, you know, it's like human level intelligence. Anyway, so I think you could make the case that there might be some break or plateau previous to GPD3 or something where GPD 2 is really impressive, but here's the kinds of things they won't be able to do and sort of pre-register that prediction and stick by it.

It would just be really bizarre to me that GPD1 to GPD2, GB2 is actually kind of really interesting artifact for the small amount of investment it took to make it. GPD 3, a couple million dollars, and you've got this like this thing that's like early stages of intelligence, whoa, then you get a GBD4. and oh my god this is actually useful they can generate billions of dollars of revenue a year it would just be bizarre to me that like you're halfway through the human range of intelligence and now it stops getting better so i do sympathize with sam's statement in the sense of like why would it stop here right if it was going to stop why it seems like it should have stopped

before it started getting better in a human intelligence way at all so you don't think we're going to hit this this stop in the road well the the reason that could happen is because of the data wall, which is to say that we can't keep training them in the same way we've been training them before. You, GPD2 to GPD3 to GPD4, you can just dump more data and compute in these models. If you run out of more data,

then the question is like, well, you know, we could have made something smarter, but we just didn't have the data for it. Right. And then the operative question becomes synthetic data or RL. And I think the intuition there of why that will work is, first of all, that should work better once the models are smarter because the sort of self-play setup is contingent on the model being smarter enough to be like, that was the wrong way to proceed. Let's back up and proceed to this another way and let's learn from this. Why did I make this mistake? Let's make sure to do it the right way in the future. That seems like they're getting smart enough to be able to do that. On a per token basis, they're actually really smart, potentially as smart as really smart humans. It's just that five minutes out they lose their train of thought. Can they bootstrap themselves in a way to help them back up every five minutes

and learn to how to do that so that they can stay coherent for longer. Seems plausible. And I mean, I had one of these takes in one of my blog posts that the way humans got better was the sort of self-play are set up as well, right? Where we learned language and, or at least the initial stages of language

where our vocal cords got, something called a Fox, Fox P2 gene. And then, so from there, you can talk to other humans, you can interact with them. That's sort of like a self-flay loop that led to humans getting smarter and so forth. Do you ever think it's weird that there is this belief in this predictability of improvement? And yet when you speak with the people who are working on these projects,

they tell you that they don't really know why it's making that improvement. Like Dario Modi was basically like, I'm not quite sure what's going on inside these LLMs to make them as smart as they are. Totally. And I think that's where that's why you should have, shouldn't be like sure that they're gonna, uh, we're on the tract of AGI because, yeah, fundamentally, we don't know what kinds of things these are. Um, you know, it could just be, I don't know, it's less, it's more implausible now than it

was maybe a couple years ago, but it could be some sort of curve fitting thing where I think if you ask me like, what is the reason AI progress, like looking back on it, if let's say GPD6 isn't that much better than GPD4 and you had to look back on it and say like, why, did that happen? I think the most reason, the thing I'd expect to say is that right now we are we are kind of fooled by

how much data these models consume whereas, you know, they've like literally seen all of internet texts and trained out at multiple times and then in retrospect you'd be like, well, of course they knew how to do the nearest adjacent thing because it was in the data site, but you should have seen that they're not that good at being creative or novel.

And so yeah, clearly they weren't going to keep improving in that way. So we've talked about AGI a couple of times, artificial general intelligence is this big phrase that's thrown around a lot. And I think that oftentimes people hold multiple definitions of it in their brain at the same time. And it's definitely something that's kind of one of the more amorphous finish lines, so to speak, that you've ever seen in the business world, that everybody seems to be working

toward it, but no one can really define it. What is your definition of it? And do you think that we're going to reach it? The way I've been thinking about it, which is less to do with like maybe AGI in the world and more so about I think its long-term impact is the kind of model which can automate or significantly speed up AI research. And why define it that way, given the fact that there's so many other jobs? jobs in the world. Because I think one of the things you really have to think about is once

it can automate AI research, then you can have this sort of feedback loop where it's helping train the next version, but it's like looking at finding better activation functions and like, you know, designing architecture that has better scaling curves, becoming over better synthetic data and so forth. So I think once you get to that point, then it's off to the races in the sense of like you could have an intelligence explosion that the kind of things that you see in sci-fi books and so that's what that's what i've been thinking about when i think in terms of aGI can it speed up a ii research um and yeah i think that's like a plausible thing whether in the next five to ten years are people working on that problem in particular i mean

the people making these models are ai researchers themselves and i can imagine them being selectively trying to um clearly they hear about their use case which is helping them with their job so I can imagine the model getting better at that than it gets better at other things. Fascinating. So what type of breakthroughs do you think we're going to need to get there, right? We've talked a little bit about reasoning. And from my understanding, like the way that models do reasoning is kind of like look at a task instead of, I mean, this is what Jack Clark told us a couple weeks ago, look at a task instead of just like spit back information, be like, how many steps do I need to perform to like really get this task right and then just go step by

step. Is that one way that we're going to get there or that they'll get there or is there something else that's going to happen? Yeah, I agree. I agree. That definitely seems like an important component of the puzzle. One big one and this is similar is that they aren't yet useful in long when you need them to kind of go do a job. You can't be like GPD4. I'll be back in a while, but can you like manage my inbox for me in the meantime or can you go go book a trip for me you know just like things that require them to sort of autonomously hold themselves together for a while and act as an agent and so just that kind of coherency where they go from five minutes of being able to be in dialogue with you to you just like tell them to do something you come back a couple hours later and they've just done a bunch

of inference to make it happen I think that will be a huge bottleneck or a huge unlock I mean. Yeah. I mean, this idea of memory in the bots, right? Like it's something, and maybe this is a little different, but it's something that I keep thinking about where like I'm speaking with Claude every day. And yet every morning, it's like 50 first dates. I have to introduce myself to Claude again. Yeah. Totally. And although there's one weird thing that Cloud started doing where like I did a podcast last week about or a couple weeks ago about the data that you get from voice and the emotion that you get from voice when you can listen to something as opposed to just have text.

And Claude, obviously, when it gets a transcript of a podcast, is only getting the text of the voice. And so I uploaded that conversation about the data that you get from emotion into Claude. And now it keeps hallucinating the audio quality of further transcripts that I've put in, almost as if like it wishes it understood what the audio sounded like

because it knows that that's an important data point. But anyway, let's just put that aside for a moment. The memory thing is interesting. Do you think there are easy ways to then, like, have, have a persistent conversation with one of these bots? Or is that going to be, like, another tough problem that we won't solve for a while? It could plausibly be very tough because I don't think it's a member, it's a matter of just keeping, like, storing memory. I think it's, like, what kind of thing are you? And are you a Chad bot or are you, is your persona, like, I am an entity that,

You know, it's not just about like I'm storing these things somewhere. It's like you have to train it to act as an agent. And compared to just pre-training tokens on the internet where, yeah, it knows how to complete statements. Does it know how to act as an agent? There's not necessarily a good way to structure that. So people have been talking about Long Horizon RL, which is the training method. You need to get something like this where you go tell it to do something and then you reward it at the end for having achieved that outcome. But the difficulty with those kinds of approaches and the difficulty with RL in general is sparse reward and non-stationary distributions, which is like, you failed to book me the right appointments based on like reading out my inbox and like talking with me about it.

Why did you fail? There's like so many different reasons you could have failed that's hard to attribute to any one of them. You know what I mean? It's like hard to learn from that. It's kind of an interesting question, honestly, like why humans are so good at learning from these sparse rewards or making long-term plans. Because when you look at it from an ML perspective, it's kind of a, it's a cursed sort of problem to solve. Yep. All right.

Well, I want to take a break here. And then when we come back from the other side, I want to talk about the Dwarkesh podcast, how you've started it and where it's going. And then particularly, I'm very interested in the AI risks because that's something that you're focused on. And it's something that I've like dismissed oftentimes in terms of like looking at the big. risks and I've promised myself to do a better job of taking it seriously. So why don't we do that on the other side of this break? Hey everyone. Let me tell you about The Hustle Daily Show, a podcast filled with business, tech news, and original stories to keep you in the loop on what's trending. More than 2 million professionals read The Hustle's daily email for its irreverent

and informative takes on business and tech news. Now they have a daily podcast called The Hustle Daily Show, where their team of writers break down the biggest business headlines in 15 minutes or less and explain why you should care about them. So, search for The Hustled Daily Show and your favorite podcast app, like the one you're using right now. And we're back here on Big Technology Podcasts with Dwar Keshe Patel. He's the host of the Dwar Keshe podcast. All right.

So you recently tweeted your bank account before the advertising checks from the Mark Zuckerberg interview hit. And it reminded me of a similar screenshot from my bank account not too long ago. So you have checking negative $17.56 and savings 14 cents. So congrats on the savings. It reminds me of before the advance of the advance of my book hit, I was negative quite substantially in my bank account also. And it's kind of this moment. I think we had similar moments where you're like, okay, I think things are going to be going on the right direction, go all the way in on this on this content plan effectively and then trust good things will come. and they have for you. So apparently, I mean, and I, you know, saw the tweet and I was like, all right, we're definitely talking this, talking about this on the show. So apparently things

are heading in a good direction for you, financial wise, at least that's the sense I get from Twitter. But I really want to know, given that, like it got to that point and now you're making your move. What is your background, Dwar Keshe? And what got you interested in starting a podcast largely focused on some of the deepest questions in AI. Yeah. Well, I started the podcast in college, a sophomore year, because that's when COVID hit. And all my classes went on offline, and so I was super bored. And then I just, I was super into economics and history at the time.

So I invited some, I was emailing some economist. And my first guest, actually, Brian Kaplan, who's now a good friend. I asked him, well, you know, I'd love to chat with you on the podcast. I didn't tell him I didn't have a podcast I didn't even have the name for the podcast yet And then he's like yeah sure I'll come on And then so we recorded an inaugural episode And from there I was super interested in economics history

For a while through college I was mostly doing topics like that And then I graduated about two years ago Still kept doing it It was honestly I graduated a semester early So it was a way of basically taking a gap semester To figure out what I wanted to do I mean I was studying computer science

But yeah I wasn't sure what to do I didn't want to become a code monkey in fact I mean there's a longer story there but so yeah and then things just kept going well in terms of the growth of the podcast itself and so I was like well this seems like a thing worth doing and investing my time into and it wasn't really like financially making money but you know whatever this seems fun and I'm learning a lot I'm meeting a lot of interesting people so kept it going dot dot dot interview alias let's go over at some point and then like other cool things happened and then dot dot some more and then interviewed mark Zuckerberg and now i'm getting

uh checks for ads on the podcast yeah it's a great summary and i recently listened to you on an effective altruist podcast uh and i thought it was a great a great conversation i and it sounded like the effective altruist movement influenced you a lot in the beginning to start the podcast uh or at least to focus the podcast on artificial intelligence yeah so what What did you find interesting about the movement? Do you ascribe to the EA theory and how present is it in your life right now? Yeah. I definitely think they've been like right on a lot of things, right in the sense of like,

this is a big focus and they've realized it before a lot of other people. Like this AI stuff, right? The EAs have been talking about this stuff for decades. And like it's been a big part of the movement for a long time, right? So you got to give them some credit for that. We were talking about pandemics and viruses and the dangers that deposes society from that long before COVID. A lot of them saw that coming.

You know, I'm actually curious about something. Listening to that podcast, what was your sense to, like, the kinds of things where we were buying into EA assumptions, did it feel like we were buying into too many assumptions? Did it feel like a reasonable, you know, I feel free to be a red team this and be harsh, but like, what was your sense on, for somebody outside? I'm, yeah, I'm assuming you're not necessarily in EA. What was your sense of like, were we assuming too many things in that conversation? Yeah, no, I'm not EA.

To me, it was, I don't think I had enough grounding to be able to really answer that question well. I think EAs should, like, talk a little bit more is my perspective. And I've tried to reach out to many of them, especially when there was this like six-month pause that was funded by open philanthropy. they call for the six-month pause and AI development and was sort of like met with like a dismissive no on that front. So I think that's kind of where some of the skepticism comes from. But I will tell you this because I think you saw the screenshot that I uploaded five of your podcasts into Claude and was like, all right, tell me a little bit about what's important to Dwar Keshe.

And then I was like, do you think Dwar Keshe? is an effective altruist and Claude says there's a high probability that Dwarkish Patel is an effective altruist or at least strongly influenced by EA ideas. Based on the strong alignment between his express views and EA priorities, I would estimate the probability is quite high, perhaps in the range of 70 to 90%. Oh, it's impressive that it gave you a problem, like an actual probability number. Oh, I asked for a probability. That was prompting. Yeah. Oh, okay. That's That's actually a super interesting use case of these models to like info dump into the long context, a bunch of stuff about them. And like, what are the odds that they're like, you know, believe a certain thing?

That's actually a really fascinating thing to do. Yeah, look, like I just like to not use, like, subscribe to a label just because it kind of constrains your ability to think. I'm not sure what EA necessarily means and then also, you know, like there's certain things that maybe traditionally considered EA that I probably disagree with. But I'm definitely happy to say like, look, the movement influenced me a lot. And I think like they've had some really interesting ideas that I found fruitful and useful. No, it's interesting. I feel like they are asking some of like the really the right questions about this technology. Like it is powerful and how do you steer it and then it all and then you know in fairness they've also been

like moments where EA has been associated with some stuff and people have been like what like obviously like sandbank-man-freet is not a distillation of VA philosophy but he was a definitely like a firm believer and a funder totally and then um you know with the whole sam altman coup like clearly that was also part of it so i'm curious like i actually don't know how much yeah like i actually don't think yeah was that big deal of the board stuff i think like from what i've heard it was a it was related to something separate. Yeah. Well, I'll say this.

The people who were some of the, I mean, the two of the board members who I think were in favor of the ouster were connected in some way. Whether there was a direct like this is an EA thing or not is still an open question. But that's what I'm saying. Like there was that presence. So I really want to get to like the core of the question here, which is what are the things that you do disagree with from EA? And then I bring up these examples not to impugn the movement, but to ask you if you think there are holes in the mood, in like the broader philosophy.

And they've just manifested in these in at least one strange moment. I mean, I could like go on for days about things I disagree with about them, depending on like what EA definitely necessarily counts as. You can like look at their cause prioritization list. Like it depends on what sense are we disagree like what do we mean EA in terms of like when you go to effective altruism.org and like what they say the cause priorities are like it's hard to you know they'll say things like we care about the poorest people on the planet and about animal welfare and about existential risk and I'm like yeah those all seem like good like things that care worth caring about like probably some of the most important things in the world. Maybe we mean like the impacts they've had because of SBF or the board stuff right is it a good sort of. scalable culture. I'm actually curious, like, in what sense? Because like, on this sort of like social cultural angle, there might be other things to say. So I'm curious which angle, yeah. It's a great, it's a great sort of return question because I think you're right that it has been something that has been a label for a lot of different things. And I don't think there's like

a charter. Right. So for me, when I think about it mostly, I think it more, mostly in terms of like this expected value equation where like people should structure their lives to maximize the expected value that they're going to have on the planet or the expected value of their presence that will have in terms of adding goodness to the world. Hmm. Yeah. I mean, there's definitely problems with that. And I certainly don't think of like so here's one particular problem that I was talking about on the podcast is that it's hard to forecast in the case of any individual how to make decisions using this framework like I would have never started the podcast if I was thinking from a maximizing expected value perspective

right um obviously you could have been like well use your CS knowledge to do something more useful like you're going to spend your time working on the podcast come on um so yeah it might not be it might not be practically practically useful in many ways And, of course, there's, like, the dangers of people who think they know what the expected value of something is and actually don't and just having uncertainty over that. On the other hand, I think, like, at current margins, like, how does society currently, if you think of other charities or think of your own tax dollars, how are they allocated? And wouldn't you prefer at the current margins, whether if they were allocated using a more sort of rational expected value framework? Like, you know, your taxes are going to be used, like, if you live in California, especially, there's going to be wasted on a bunch of useless shit. like all these nonprofits and whatever.

And like, wouldn't be better if like they thought about like, let's, let's pull up the spreadsheets on how much good these, uh, these nonprofits are and these like different, uh, institutions we were funding with our tax dollars are doing.

And I think that kind of mentality actually would be broadly useful in the world. Yeah. I don't know. I think it's worth considering sort of the longer term risks of AI, right? Which you mentioned they were early on. And I think that they've probably brought more focus to. So on that front. I'm curious, like, what you think makes some, I guess it's the CEOs, a lot of the CEOs that we hear from, and maybe some of like the intellectuals in the EA movement, but elsewhere.

What do you think makes them so afraid of AI or so cautious about where it could lead us? Yeah, I think it's kind of a sort of straightforward thing of like looking at maybe a couple of years out from the people who are just thinking of this as a normal economic transition where you say, okay, we'll have things that are as smart as humans and as smart as the smartest humans when it comes to science and tech. Well, what happens if you plug this into our basic economic growth models? You have, you know, you just have a huge effective population. And so there are people doing science and R&D for you. You're like rapidly going through the tech tree because you have like billions of researchers. And maybe like there's certain physical bottle next to this, but you can, you just, you just have like trillion, billions of extra people helping you do further AI research, whatever, and there's enough of them that if they wanted to, they could do a coup, they have certain advantages in the sense that you can, like, they can easily copy themselves in the sense of their weights, they can, they can increase their population rapidly. And there's like, they're harder to kill, to put it that way once they're deployed than humans are.

Like, you put out a bioweapon or a nuclear war and a bunch of humans will die. If these things keep a seed version of themselves somewhere, then, you know, like, it'd be hard to, like, if you had to go to war with them, it's a sort of asymmetric. And then you go from there to, listen, we fundamentally, as you were saying earlier, we fundamentally don't understand what's happening in these models, but we know they're really smart. And sometimes they, like, with a Gemini thing, they go, they do things we didn't expect them to do. or like, you know, like, that was a great example where I'm sure Google didn't want this sort of embarrassing image to come out, but that's just what ended up happening at the end. Like, I imagine these things are super integrated into like our cybersecurity and are trained on this long horizon RL, so they are their coherent agents over a long period of time.

You can like, you know, put all that together. And it's like, well, that could go wrong. Yeah, it's interesting because for me, it's always felt far-fetched because I'm working in like the current versions of chat GPT and Claude. But then if we get to this place where these machines are effectively improving themselves, right, which you mentioned, like, that's not only a potentiality. That seems like a likelihood that people are, that the developers of these systems are going to get them working on improving them, then, and again, we still don't fully know where

they're going, then it seems like that could have some unintended consequences. Totally, yeah, yeah. I mean, I'm still expecting a great future because of AI. My expectation is, like, the median outcome is good. I think we should worry about the cases where things go really off the rails and do what we can to reduce the odds of that. And so what, and is that like the whole practice of alignment? Is that what people talk about when they're like, if we're going to set these things

going, like we should at least align their values to be closer to the ones we want as humans? Yeah. It meets so many things at this point that, like, even I'm sometimes confused by what exactly means. I mean, one of the goals is that it should do what the user wants it to do unless the user wants it to do something that would be, like, hurt other people, basically. But there's, like, problems with that definition, obviously.

What if the user wants to use, like, in super intelligence to make a bioweapon or to do a coup against the government or something? But, yeah, something like, basically, like, we don't want the AIs to, like, then have their own drives and want to take over. something. And do you think that this is a reasonable concern? And if so, do we have a reasonable chance of stopping it? I think both it's a reasonable concern and we have a reasonable chance of stopping it. One of the things I've discussed in one of my recent episodes with Trenton and Cholto is there's these researchers who have discovered interesting properties that these models

have in terms of how they represent their drives or they're, you know, what they're thinking. And so you can like see if they think if they're being honest or not or if you're just reading their internals, whether they think they're being honest or not. And as they get smarter, maybe you can parse out fundamentally it's a bunch of parameters, right? So it's much more interpretable than a human brain or something. So maybe we can learn ways to sort of understand what they're up to and train them to do the right We have an advantage in the sense of, like, listen, if you break the law, we might put you to jail or something.

But with these AI models, we can literally change their brain if they do something wrong. And like all their children and have changed brains as a result, just like the entire lineage has changed. Yeah, exactly. It is. And I guess we could shut them off. I don't know. Well, hopefully. That is a good point.

I mean, they will be broadly deployed. So if they really go off the rails, then I think it might be tough. Damn. and so but but you do think that we're going to end up with a positive outcome here contingent on people still doing a bunch of alignment research and also these systems being deployed in a way that is um you know you don't want just like some china just racing ahead of everybody else and then just like doing a coup of the whole world because they have much more advanced AI right so contingent and then also like we want to make sure that the models are

deploying. They serve the needs of the user and don't do crazy things. But given that, you know, just like fundamentally more stuff, more abundance, more prosperity. I think that's good. Okay. As we come to a close, I did open some questions up from the Twitter folks to ask me what they want me to ask you. So I have actually one question that aligns a little bit with this discussion topic and then one that's kind of more about the podcast. One is how have your political views changed, if at all, since you started the show? or let me even, you know, put a different frame on that or similar. You've interviewed like Mark Andreessen as well,

who has like very different perspectives from some of the early AI guests. So has that sort of changed the way you think about AI at all? I mean, generally politically, I'm very libertarian. And I was probably even more libertarian than I am now. Like I was like an anarcho capitalist and, you know, whatever. So or at least a soft version of that. So politically. I mean, the way in which that's changed is that I'm open to the possibility that potentially

some kind of regulation might be useful on AI, but I still have libertarian instincts and I'm not sure if it will be done the right way and maybe it's better for private companies to proceed and come up with incentives and constraints by themselves. What type of regulation do you think might be appropriate? I was talking this week with an editor I work with and we were like maybe regulating the way that kids and AI can interact given you really have no idea where that's going to go. once you put this in the hand of a child. Yeah, potentially, but I think it'll honestly be better than what they're currently doing,

which is YouTube and TikTok and Twitter, you know, Facebook and so forth. So I would prefer my kids are playing with chatbots than they're playing with what they currently have access to. I don't have kids. What if I did? So I think in the world where you have really fast AI progress and you are coming up to this point we're talking about where EIs can help improve themselves, then I think what you want to do is you might need a sort of government. level actor to be like all right everybody pause for a second anybody pushes a button basically of

like helping the having the i help us with the i research they could get fundamentally better a i's than everybody else has and kind of take over the galaxy basically so before we before we let somebody do that we got to make sure we're in a place where we're ready to proceed right and not just let some random person do that so in that world i think regulation makes sense yeah then the second question we had was, someone says, give us the Dwarkesh interview prep playbook. That's his innovation. And if he's able to explain it in the way that can be replicated or at least approximated by others, well, have many more interesting interviews. Okay, I'm honestly self-interested in this as well. So how do you do it? Well, I know it sounds like a terse to say this,

but I honestly, just like I prep a lot. And I think there's also a flyable. There's also a fly wheel up by doing interviews. I learn a lot of things. And because of that, I can get better interviews, learn more things. I think the main flywheel honestly is that I make the podcast better, smarter people listen, some of those smart people I become friends with. And they teach me a bunch of things. And now I can do an even better interview. I can get connected to a bunch of other smart people. They teach me more things. So I think that's like, that's probably a big part of the flywheel that people may not know about. yeah i've definitely had this here like we've talked about certain companies and stuff and then people

who listen have reached out and been like there's something you should probably know about this and given that i enjoy the show let's talk through it's always helpful yeah yeah yeah in what ways is different from me is do you have some trick that i uh i should be aware of or like other are there tools of trade no i really think that's it i mean you're going to get a great show i think there's a few ways you'll do it one i mean and you already know this but you prep like crazy. You're not afraid to ask tough questions. And yeah, when somebody wants to call and talk through the topics afterwards, take the call. Totally. And then one more thing, this is more of like a media question, but video has been pretty big for you. So what was the thought about

doing video? Because it's also expensive and time consuming to produce. So I'm curious if you had an ROI calculation about doing video from the beginning. And you also oftentimes show up with with a video, I guess a video camera and record in person interviews. So talk a little bit through your strategy there. I will say for anybody doing a podcast, I highly recommend video and like if you can do it in person. I mean, especially for me doing it full time because it increases the expected value so much where with an audio podcast, the discoverability is so low that you kind of know how many listeners you'll get. But the tail outcome where something goes really viral. I'll give you an example. My most popular episode right now is Sally

Payne. It has like 800 something thousand views on YouTube. And, you know, like she's, she was totally unknown before the podcast, at least by the White Republic. But the episode was so compelling that now you can make clips of it. Now the clips wouldn't be as compelling if they were made of a not in-person episode, let alone if there was no video at all. So you make these amazing, you make these great clips. They bring people to the video. And then you can have like close to a million and people watch it just because so on any average video you might do in the beginning that might not be the case but you just have this asymmetric return potentially from having that artifact and then as far as how to make it better like ever tell people to just like honestly like watch a bunch of like

podcast with mr beast i think he has good advice yeah and so do you set up those cameras yourself or you bring somebody in i've usually half and half about yeah but yeah like i got the workflow down and I set it up. And more recently, I've been having a friend help me. Nice. Yeah, that's sort of like the bag of podcaster tricks. Like it's not just sitting down in front of a microphone. We all have to learn these days.

We all have to learn sound. We have to learn video and figuring out the right. The main thing is like, I don't know this is your experience, but like clips is the main thing. Like you had to spend a ton of time. And you got to do all this Mr. B stuff of you make a clip with the wrong first five seconds and it's not going to do well at all. But then if you spend a bunch of time thinking through what is like the hook to begin with,

then it could go super viral. So that takes up a bunch of time, right? Definitely. Yeah, so much time that I mean, it's videos like become like part of the strategy for me, but also it's low down because of the amount of work that it takes to get into. But we also mean, we do two shows a week. So it's kind of like, yeah, it's a question of sometimes do the show

or do the clip. So totally, totally. Do you do, is it, uh, are you doing the show full time or, uh, yeah. So big technology is full time for me. It's the show. It's the newsletter on substack and then YouTube. You're full time also, right? that's right yeah i mean it's a really it's a great life if you're able to to do it because of what we talked about just finding all these interesting people to just get to spend time with and learn for yeah yeah yeah 100% all right last question for you out of all the interviews you've done i don't want to ask you your favorite but i want to ask you who was the most impressive person that you've

you've spoken with someone who you walked away with and said all right this person really gets it i mean i'm sure there were multiple but who's at the top of the heap that Carl Schulman, I would say. He's just this really interesting person who has these models about how the AI takeoff will happen. The stuff I've been saying about, do you have AI research and whatever? He has just thought it out much more. I can go through the numbers in terms of, I mean, literally things down to the level of, okay, well, suppose you get something really smart. How fast could it do a takeoff? And then, well, e. coli can double every 20 minutes. And it has this many moving parts inside of it. So we have a sort of lower bound there. of how you can just do that. And then what would it look like to convert the entire Sahara into solar power and like how many data centers

could make that of that and stuff like that? Yeah, oh, that's fascinating. All right, Dorcas, awesome stuff. Thank you so much for joining. Awesome. Thanks so much for having me. This was fun. All right, everybody. Thank you for listening. We'll be back on Friday with our show with Ranjan Roy, breaking down the week's news

and we'll see you next time on Big Technology Podcast.