Limitless: An AI Podcast - AI DEBATE: Runaway Superintelligence or Normal Technology? | Daniel Kokotajlo vs Arvind Narayanan

Episode Date: June 4, 2025

Two visions for the future of AI clash in this debate between Daniel Kokotajlo and Arvind Narayanan. Is AI a revolutionary new species destined for runaway superintelligence, or just another step in humanity’s technological evolution—like electricity or the internet? Daniel, a former OpenAI researcher and author of AI 2027, argues for a fast-approaching intelligence explosion. Arvind, a Princeton professor and co-author of AI Snake Oil, contends that AI is powerful but ultimately controllable and slow to reshape society. Moderated by Ryan and David, this conversation dives into the crux of capability vs. power, economic transformation, and the future of democratic agency in an AI-driven world.------💫 LIMITLESS | SUBSCRIBE & FOLLOWhttps://limitless.bankless.com/https://x.com/LimitlessFT------TIMESTAMPS0:00 Intro1:38 IS AI Normal Tech?27:13 Capability & Power39:48 Manageable or Existential?47:32 AGI Milestone55:28 AI by 20301:07:03 Making Sense1:11:05 Closing & Disclaimers------RESOURCESDaniel Kokotajlo https://x.com/dkokotajlo Arvind Narayananhttps://x.com/random_walkerTwo Paths For AI - Danielhttps://www.newyorker.com/culture/open-questions/two-paths-for-ai AI as Normal Technology - Arvindhttps://knightcolumbia.org/content/ai-as-normal-technology AI 2027 - Danielhttps://ai-2027.com/ AI Snake Oil - Arvindhttps://www.aisnakeoil.com/ https://www.amazon.com/Snake-Oil-Artificial-Intelligence-Difference/dp/069124913X Pause AIhttps://pauseai.info/pdoom ------Not financial or tax advice. See our investment disclosures here:https://www.bankless.com/disclosures⁠ 

All right, everyone, I've been looking forward to this conversation all month. Maybe for many months, I've got to say. So the central question on today's episode is, is AI normal technology? Maybe it's something like the Industrial Revolution, we've been through that before. Or is it very, very abnormal? Is there incredible variance? Is it something between utopia and annihilation? It could go either way.

It's like nothing we've ever dealt with in the past. We've got two guests on today that are going to discuss, or I might use the word debate, this subject. We've got Arvind Narayanan. He is a Princeton computer science professor. He is the professional hype slayer, including some hype slaying he's done in crypto in the past. He is the writer of AI snake oil, and he is a believer that AI is normal technology, kind of like electricity or the internet. Arvind, welcome to the podcast. Thank you. I'm really excited. Thank you for having me. We also have Daniel Kokitello. He is an ex-open AI researcher. He's the author of an incredible report called AI 2027, which is like a month by month.

month forecast of AI progress all the way to AGI, which I bet you can guess the year in which he predicts that's coming. He thinks AI is very, very much not normal and might become super intelligent by the time you upgrade your next iPhone. So pretty soon. Daniel, welcome to the podcast. Thank you for having me. Excited to discuss. All right, guys, this is going to be a lot of fun. We've got seven topics to go through and we'll see if we get to all seven. We're going to do this in a compare and contrast way. I'm going to see the conversation. David's going to help. And then just fire you right into the topics. You guys ready? Let's do it. Okay. The first topic on the agenda, the question, is AI normal technology

or is AI very much not normal like nothing we've seen before? Is this kind of a tool that we can control or is this a new species destined for its own autonomy? Arvin, I've got a quote from you to kind of set the scene, maybe in your perspective. You said this, AI belongs with electricity and the internet. It's transformative, but yes, it's still a tool. Daniel, on the other side, you said this. This isn't another general purpose tech, talking about AI. It's a new species about to outthink us. Arvin, I'm going to start this with you on the topic. AI, normal tech. You put out a report entitled, like with that as the title. What's your case for AI being normal tech? We break down the way in which AI or really any technology is going to impact society into a kind of four-step process.

The first step is improvements and capabilities, new models coming out, AI is able to do new things, but there's no direct line from there to the impacts that we really care about. The next step after the first step of invention, as we call it, is innovation, turning that into product development. And from there to users learning how to use these new capabilities and ways that make sense for their workflows, what they want to do and their jobs. and the last and final and slowest step is adaptation. We talk about many reasons why business models need to change, laws and norms need to change for AI's impact on the world to really be felt.

And we can already see this process playing out in the last two years where, you know, we had this big leap in capabilities with chat GPT and other releases around that time frame. But when you look at the extent to which that has transformed people's work, flows, I think we're starting to see that that's going much, much slower than people initially predicted, and we have some numbers in the paper to back this up, and we predicted this trend broadly will continue. Daniel Arvin thinks it's going slow.

What's your take? I mean, I also think it's going slow right now. For example, you said people, it's gone slower than people predicted. It hasn't gone slower than I predicted, in fact. It's gone about the same speed that I predicted. I know you're moderating this debate, so you're in charge, but I wondered if I could also ask some questions to help the stage. So, Arvind, if you've read AI 2027,

I would love it if you could give your own sort of alternative story, basically. Like, if you had been in charge, like, suppose that someone gave you a whole extra year of life and said you have to spend your year writing your own counter scenario to AI 2027. What would that be? What would you depict? And maybe one way of putting it is, like, at what point would it start to diverge from what we predict in AI 27? And then, like, what would do it?

how it would gradually diverge more and more as time goes on. Please just, like, paint that picture for me because I think it's a useful way to sort of, like, set the stage for everything else that we're going to be talking about. Sure, yeah. So, I mean, our fundamental point is that in 2027, the world is going to look more or less like the world in 2025. And like you said, yeah, we think gradually things are going to look different,

and that divergence, we say, you know, is roughly on the time scale of decades. But qualitatively, what the diversions would look like for us, there's perhaps a greater role of human agency and policies and so forth in determining what kind of future we end up in, as opposed to predictions stemming directly from the capabilities of the technology. And so that's the reason why, you know, picking one scenario would be kind of sampling from a very broad range of possible futures. So it's a little difficult for me to paint a scenario, but broadly I can, I can. I can say that over, let's say, the next couple of decades roughly, we expect to see, just for concreteness to take one industry like software engineering. You know, people are going to figure out what it even means to be a software company or a

software engineer at a time when AI is much more capable than it is today at writing code. Today we can vibe code, create simple apps. What if we can create, you know, enterprise scale apps in the future with AI and minimal human involvement. So one scenario for what software might look like in that situation is it'll be silly to create software once and expect hundreds of different companies to use it because you're forcing everyone to be boxed into the abstractions that are built into the software. Instead, enterprise software will mean creating software for one company or one team or even one individual user. And the role of the software engineer of the software company will be primarily to talk to

that client understand the specifications. So creating software will primarily be, you know, that communication role as opposed to typing code into a keyboard. So that's one scenario for how one industry might change. Again, it's hard to give an overarching scenario for how the world overall might change in the next couple of decades. Thanks. So I agree that the future is uncertain, lots of branching possibilities. And so there's a lot of uncertainty that we sort of have to collapse down like the wave function, collapse down into like a particular path. for purposes of discussion. And so I totally understand, you know, we dealt with that a lot when we were doing a

2027 is like we're not like confident that this particular path is what's going to happen. But it sounded like you were saying that like, yes, the AIs are going to get a lot better at coding. They might be better at autonomous coding so that they can basically like do the full stack development process without human intervention. But then the humans might go to other roles such as interfacing with the customer to understand their requirements and so forth. And you think that might happen in the next few years, but that it'll take. like decades to sort of like gradually transform the actual industry to to upend the way things

are going now. So that's that's definitely a disagreement between us because I think that I mean to put it in like a little potted summary, the AI 2027 story is first you automate the coding, then your research speeds up so that you build new AIs that can automate the whole research process. Then your research speeds up even more so that you can get to what you might call superintelligence, which means it's like cognitively. superior to humans in every well-event way. It still has to learn from real-world data. It still has to like, you know, deal with physical bottlenecks, but it's, it needs less data than humans do, and it like can, like, work through the bottlenecks faster than humans can. And there's a

million of it. And it thinks, you know, 50 times faster, you know. And so then we depict that sort of process happening in 2027. And then in 2028, we depict it sort of exploding into the economy and actually transforming things. And so it was interesting. You said earlier, like, your first thing that you said in response to my question was, I think 2027 will look much like 2025. And I kind of wanted to say, I also think 2027 will look much like 2025. AI 2027 depicts, you know, the economy looking basically the same in 27 as it does today, because a lot of what's going on is that the capabilities are advancing rapidly within the companies, but they're not like being, they're not like transforming the world yet.

And then in 2028, that's when the world transformation happens. That's when the new factories start going up, et cetera. So, you know, you talk a lot about the bottlenecks and the frictions and so forth. And I think, I guess I'd want to say something like, yes, those things exist and they're extremely important. And that's part of why, like, there's so much of a gap between what AI systems are currently capable of and the actual transformations of the irrelevant industries. Like, there's just a time lag to, like, build new companies based on the new technology and to integrate it into everything. so that's why I think things won't look that different up until around the time you get superintelligence.

And then after you get superintelligence, they'll also be bottlenecks and there'll also be time lags. However, I think that they won't take decades to overcome. I think that when you've got the army of superintelligence is we're looking at something more like a year to get to a fully automated economy rather than something like 20 years. So having just like, you know, recapped sort of my position there, I would then ask you, do you think that we will get to superintelligence anytime in the next decade? Yeah, thank you, Dan. That's very helpful. Yeah, so let me pinpoint exactly where some of my disagreements are.

So in the paper, AI is normal technology, we dispute the whole premise of superintelligence. We don't disagree that AI capabilities on many individual dimensions are going to keep increasing and there are going to be superhuman AI capabilities, just like there are today in things like chess. However, we think that most tasks are like writing and not like chess. So the key difference between these tasks is that when it comes to chess, human performance is limited by our computational abilities. So it's very natural to see how AI can be dramatically superhuman

and, you know, has been for the last 20 years. In a task like writing, our abilities are not computation, Our abilities are limited by a few things. One is hard to agree on what constitutes good writing and what doesn't. It's limited by our knowledge of the world. And we talk in the paper about why there are going to be bottlenecks to acquiring knowledge that humans haven't been able to acquire because it requires real world interaction. It requires experimenting on people and so forth. And finally, our writing abilities are limited because, let's say you want to write to persuade some one, we think there's only so much that you can make a piece of text persuasive. There are fundamental limits there. And so computational limits are relevant only in a narrow set of tasks. In most cases, it's these intrinsic limits and knowledge limits. And we do think that over and over AI is going to run up against those. So this idea that you can run AI at 50x speed and have a million copies of it that make things go a million times faster, we dispute that premise.

Well, I certainly don't think it will go a million times faster. We give our quantitative estimates of how much faster it will go, and it never gets close to a million. Oh, okay. Sorry if I misremembered. I agree that the parallel thing, there's massive diminishing returns there and things like you said. But I think I want to push you again on the point about superintelligence. The way that I would define it is it's an artificial AI agent that is better than the best humans at every important or relevant cognitive task. And so it's fine. And so according to this definition, if there's a hard ceiling on how good you can be at writing that's only slightly better than the best writers that are humans, it's still possible for there to be superintelligence.

It's just that the superintelligence would be running up against that ceiling, but only on an absolute scale would still only be slightly better than the best humans. But as long as you think it's possible to be like ever so slightly better than the best humans, then it's possible to have an AI that's meeting that definition for what I'm saying. And so then the question is like on all the relevant domains. So, so, you know, obviously in chess, they're already superhuman, but like, you know, enumerate all your favorite domains, all the domains that you think are important. Will there be AI systems that are better than humans at all of those ones? And it doesn't have to be like massively better,

just like somewhat better than the best is basically what I'm asking. So if they're only a little bit better than humans, what leads to this dramatic acceleration? You know, I mean, if humans got a little bit better at something, we wouldn't think that would lead to, you know, a 10x of GDP or whatever. Well, let's talk about the different types of acceleration. So first, the research acceleration is the one that we've thought the most about. So on the website, we've got this appendix where you can go read the way we did our little

hasty back-of-the-envelope calculations to try to get the numbers that we used in the scenario. And we estimated something like a 25X speed-up for the AI research stuff when you hit the superhuman AI researcher milestone, which is kind of like relevant to this discussion because it's basically like slightly better than the best team in researchers. And the way we get that 25x, we did a couple different methods and then sort of aggregated them. And then we like shaded it downwards just to be conservative. So, well, I guess I won't get into that here. But I'll try to briefly summarize. One way of thinking about it is we pulled researchers at companies, how much slower their research would go if they had less compute. And then we sort of flipped it on its head and thought, well,

like if you're running these AIs at 50 times human speed, or say 30 times human speed, I think, which is the number we actually used, that's kind of like having a company going 30 times faster, but with 1.30th the compute. And so applying the numbers from the pole, we get some sort of slowdown factor of like, well, we're going this much slower

because you have 1.30th the compute. But then we're going 30 times faster because we're running things 30 times faster. That was one of the methods we used. We also use a more like granular method of just thinking through the research process and looking for ways you can be more efficient here and there

and adding up all the multipliers. Anyhow, so that's how we got the 25X number for the overall speed of research. We haven't done a similar analysis for every other field of human endeavor, like, you know, rocketry or plumbing. We just did it for AI research. When it comes to those other things

like rocketry and plumbing, I think I would say the general heuristic that I would use is something like think about the gap between people who are the best in the world at it and people who are merely good professionals. And then look at the shape of the distribution. Oftentimes we see a sort of like heavy-tail distribution

where like most people are around the level of a professional, but then like some really good people are like able to get things done in half the time. And then like some really, really, really good people, the best in the world are able to get things done in like a quarter of the time that it takes a typical professional or something. So you can sort of plot this distribution. And then you can try to speculate about where the sort of limits are. You know, where is that fundamental limit

to how fast you can build new rockets, for example? And the insight that we're relying on is basically that if you have this distribution that looks heavy-tailed, then you should expect that the fundamental limit is not right above the highest data point that you've observed. Like probably it keeps going for a little bit before we're hitting limits. Otherwise, it'd be the strange coincidence that, like, the fundamental limit is, like, right above what we just happened to have observed.

It would be different if the distribution looked different. Like, if the distribution was sort of clumping up, then that would be evidence that we're sort of, like, asymptotting towards some sort of limit. Anyhow, all this to say, the way that we got our headline figure for 2028 in the scenario was looking at historic examples of transformations of economies, such as what happened in wartime, when, like, the U.S. repurposed all their factories to build planes instead of, cars. And so we're imagining something similar where we're imagining a political content. This is another possible disagreement we have with you, is that you're talking a lot about the like the bottlenecks, the frictions, the regulations, the market being cautious to adopt new technology, etc. We're imagining a lot of that still happening, but to a much lesser extent than perhaps you think due to the race dynamics where they were projecting where like

the US is afraid of losing out to China. China's afraid of using out to the US. So the, so the president is basically partnering with AI companies to, like, cut regulations as fast as they can and, like, you know, smooth things over in the special economic zones and so forth. In that environment, we're thinking it's similar to World War II, where they also cut regulations, cut red tape, you know, did this sort of massive pivot from things. So we were thinking basically, like, again, to use the distribution analogy, we were thinking, like, how fast have good humans been able to do this in the past? How much faster are the best humans able to do it compared to, like, typical professional humans.

It seems like there's a decent gap there. It seems like the Elon Musks of the world are able to like, you know, get SpaceX off the ground much better than Boeing and the various like legacy companies. And so then that suggests that if you do have the army of superintelligences that are better than the best humans, you should be able to go faster still than the best humans. And how much faster? Well, maybe another factor of three, maybe another factor of five, something like that. And so instead of taking like a World War II length of time to do this transformation,

mission. Maybe it takes like one-third that or one-fifth that. That was sort of roughly how we got those numbers. But yeah, obviously there's lots of uncertainty. Cool. Thanks. So there's lots of things in there if I can respond to a few of them. So first, let me start with the policy implications. So I think there's a little bit of a self-fulfilling prophecy effect here. Right. So in our view, AI as normal technology is not just a prediction, but a prescription for how we should approach it. We should not approach it as if we're in wartime, and that's for all the reasons that we describe. And my worry is that with AI 2027, it's leading to the very same prescriptions that will lead to the arms race dynamics that you're so concerned about. So, you know, if we think that we have to do this because China is going to do this, it inevitably kind of leads you to that spiral.

I do agree that if it's not just cut regulations, we're talking about civil liberties, right? if we were to suspend civil liberties and suspend democracy in a very real sense and act as if we reoriented the whole economy toward a kind of military production, except here it's improving AI industrial capacity. You know, things could go qualitatively very differently. But why should we? I think it's the only reason we would do that is if we assumed that each side assumed that the other one is going to do that. So that's the first point. The second point is when it comes to the acceleration, this idea that the fastest humans are much faster than the median, and so we can achieve that speed up. That's a little tricky to me. So when we look at past technologies,

you know, when the internet was starting to become a thing, people thought that this was going to lead to a 10x improvement in all kinds of knowledge work, because instead of going to the library to look up a fact, you can look it up instantly on the internet. But that turned out that was only one step in a larger process. Right. So whatever step of a process we're looking at, you know, let's say one surgeon is much faster than another surgeon, but when you look at the entire context of a hospital, that surgeon's sufficiency is only one small part of it, and there are so many other bottlenecks. Same as with the internet. When we remove one bottleneck, other parts of the process become the bottleneck. Within each job, I think the slowest task is going to become the

bottleneck. And for the whole economy, the slowest sector, the least productive sector, is going to become the bottleneck. And we have some citations from economic works that talk about this. And third and last point, I would say, is in our earlier conversation, I think you acknowledge that there might be limits, most tasks being like writing, where AI is only slightly better than the best humans. And so that maybe seems a little bit at odds with the claim that there is this long tail and therefore AI can be far out in the extremes of the distribution, far better than the best human. Thank you.

I guess I'll take them in reverse order. So I agree that it's task-dependent or field-dependent. The way I like to think about it is you keep improving capability, but at some point you sort of asymptote towards whatever the hard limits are in terms of actual effects on the world. And it's going to be different from field to field. So I think in a lot of research-y-type fields, it seems like we have evidence that it's an extremely like fat-heavy-takes,

distribution and the very best researchers are just like orders of magnitude more productive than the typical researchers in terms of actual effects, like in terms of like how much like actual scientific progress they produce. Whereas in other field, like perhaps in plumbing, you know, the world's best plumber is only able to do, you know, twice as fast as like a typical plumber, maybe. I don't actually know that number, but like I wouldn't be surprised if something like that were the case, right? I think that for things like the transformation of the economy, we're sort of taking the zoomed out perspective of like, yeah, that's going to involve a lot of specific things, that will run into bottlenecks,

but then also sometimes you can sort of route around that bottleneck by finding a different way of doing things. It doesn't depend on that. Zooming out and being like, what about this whole transformation task as itself, the thing that we are measuring, well, it seems like there have been humans trying to do something like this in the past. Like the human is in charge of the transformation of the economy during Rollo 2

and stuff like that. Also, similarly, I think economic production, like I think that like what Tesla and SpaceX and Boeing and so forth are doing is kind of similar, where you want to like produce more, cars that are better and that are cheaper than your competitors, but you have to design them and you have to test them and you have to produce them. You have to purchase the relevant equipment and install it and debug it and so forth. And there's bottlenecks every step of the way through that

whole process. And there's ways in which like, I mean, that's why it takes so long is that like your assembly line is not going to work correctly on the first try and you're going to need to like purchase new equipment and like integrate it and so forth. But you can sort of zoom out of all of that and be like, okay, so how much better is SpaceX at doing that sort of thing compared to Boeing? And the answer is, like, a factor of five or something. Like, they seem to be able to, like, actually make progress in the aggregate, like, five times faster or something. So similarly, we're thinking that once you have the superintelligences, and you've given them the legal authority and, like, the money and the financing and the support

to basically be in charge of all of these factories and to be calling the shots, we're imagining a similar sort of gap between that situation and, you know, SpaceX being in charge as we see between SpaceX being in charge and like Boeing being in charge, you know. And that's where we got the sort of one-year takeoff, one-year transformation sort of thing. If I can also talk about some of the other things we're mentioning, so the self-fulfilling prophecy thing is like, it gets me to the core because I don't want to create this self-fulfilling prophecy.

And this is actually my number one concern about AI-2020. in the ways, like, like, if it turns out that I regret this on my deathbed, it'll probably be because of this. I understand that, like, these companies benefit from hype about their products and that, like, they have an incentive to hype it all up so that they can get more investors and so that they can, like, get the government to, like, wave red tape for them and stuff like that. However, I think that, like, well, this is what I think is going to happen, and it's what the companies are gunning for. Like, like, AI-227 is not that new. to the people who've been at Anthropic and Openia and GDM,

it's sort of like what they're aiming for and what they expect. And like, I think that they're going to get it if we don't raise up a storm about it. Like, like, I think that we can, I don't know.

Like, I think it would be different if, like, we were the only people. I think, I think basically, like, I'm torn about this, but I, but I've, like, made the,

made the gamble that, like, since it seems like this is where we're headed anyway, we need to, like, talk about it. and raise awareness about it and hopefully use that as steering towards something better rather than sort of just hoping this doesn't happen

because this is what the companies are trying to make happen and they are very powerful and they have lots of money and lobbyists and so forth. And so if I don't tell the president now, like what about China? You know, like if I don't raise these concerns,

the company is totally will a year from now, you know? And like when they have automated the AI research, they're totally going to be like saying, oh, you need to cut the red tape for us and you need to like, I don't know. So I guess I'm making the gamble that like AI 227 has a relatively small effect on the self-fulfilling prophecy direction

and that it has a bigger positive effect in the sort of like getting people to wake up and pay attention and then hopefully steer things in a better direction. But I'm not entirely confident that that's true and I sure hope it is. Yeah, I definitely agree that that's something we should all hope for. And just to put a little bit more clarity on perhaps where we agree and disagree. And this gets to some of the other.

items in the outline that Ryan and David shared. I think for us, if some of the things that I think you kind of take as inevitable happens, which is that there will be this capability of improvement, and then we will have kind of no choice but to put AI in charge of the factories, maybe give it legal authority. And for us, and then they're kind of red line as AI owning wealth and so forth. If all those things happen, to me, we've already lost. We've massively lost from a policy perspective. Even if some catastrophic or existential event doesn't happen, this is all, you know, not the kind of thing we want to be doing in a democracy, not the kind of thing that's compatible with civil liberties. So for us, those are where a lot of the intervention points are, as opposed

to, you know, alignment, international treaties and so forth. Those concerns only come up at a stage where for us, a lot of mists have already been made. So that's perhaps one point of difference. Another one is, oh, sorry, Ryan. Yeah, I agreed. And I'm wondering if we could zoom in on that for a second, then get back to kind of the second. But this is something I read very much in your work, Arvinida, and I wanted to get Daniel to really respond to this.

So throughout your work and kind of the case for this being normal tech, you make this distinction between capability and power, and you sort of view those as decoupled things, right? So you say this, capability is intrinsic, power is the permissions that we grant the AI, right? So we can keep power away from the capability of AI's and do things like maybe don't give it legal status, don't give it the ability to collect wealth, don't give it the ability to lobby governments, right? And you're imagining that we can control power and separate that from capability. I have actually not heard Daniel's take

on the decoupling of capability and power. And so I'd like to hear that, but I'm wondering if Daniel thinks that you can't really decouple these things. Capability leads to power. And so, Arvin, maybe we could get Daniel to respond there on your first point. So Daniel, this idea of capability and power being separate, what do you think about that? I agree that they are separable in principle. Like, there are different things for the reasons that Arvins has said. I do think that in practice, they're going to tend to come together for reasons illustrated in a 20227. I mean, I think just concretely, if you let the companies automate their research and get super intelligences, I mean, like Irvin said, you don't want to be in that situation in the first.

Like, if you get to that position, maybe you've already messed up because now they're sitting on this incredibly capable, incredibly valuable thing, and they're going to be able to credibly lobby the government and say things like China's building their own version and China is going to let it out into the economy and have it transform things and, you know, rapidly build up military capabilities and so forth. And so if you don't want to lose to China, then you have to let us do it over here on our side of the Pacific. Like this is a prediction about not the technology, but about the social dynamics. We are predicting politically that that's what the companies are going to be saying, and that's what the government is going to agree to, at least by default. And so that

means they will end up with the power once shortly after they have the capabilities, if that makes sense. And to what Irvin, you were saying earlier, like, I think, yeah, like in a sense, I agree with you that like if you're relying on alignment, you've already messed up as a civilization. Like, if you're getting to the point where you're like, yeah, we're going to have the army of super intelligences and it's going to be like autonomously doing all this stuff in the economy and the government's going to be supporting it rather than like rather than trying to fight it. And so that's why we have to make sure that it has the right values because otherwise it could go very badly for us. Like in some sense, you've already taken a ton of risk by the time you're putting yourself in that point

and you've sort of played the board game into like a very difficult to win position. I think I agree with you there. And here's maybe the thing to say is like A.227 is a prediction. It's not a recommendation. Like this is not what we think should happen. Yeah.

Yeah. I totally agree that the AI companies are going to be lobbying for all kinds of things that again, I want to keep using this language, are incompatible with a democracy. And so I think that's the thing we should be trying to stop and we should not in any way take it for granted. And I don't think

that AI companies are as powerful as we make them out to be. When we look at the numbers, especially in the U.S., on the extent to which people are apprehensive about AI as opposed to excited about AI, I think it's very clear that if the companies make a play for this kind of, you know, authoritarian partnership between big tech and the government, we will be able to mobilize the sufficient public backlash to that and we should build the infrastructure for potentially doing that now. Those are some of the things we should be focusing on. And for me, this difference between capability and power is not even primarily a matter of the AI companies themselves.

So we make this really big distinction between development and deployment of AI. So the entities that are going to be in charge of those limits on power are going to be the everyday organizations, you know, schools and educational institutions. government agencies, banks, you know, hospitals, legal firms, every other firm in the economy who are going to be using AI in their various sectors in ways that are either with oversight and compatible with the principles we want, or are going to be handing over crazy amounts of power to AI systems in the pursuit of, we think, small increases in efficiency. And once again, you know, it doesn't matter if the AI companies themselves.

themselves are hyping things up and are power hungry, we can ensure that the broad base of organizations and institutions in our society don't fall for this hype and abide by the principles that we think are important. I would like to try and take a stab at really defining the line that separates you to because there's a lot of agreeing going on about many, many things. And I can't quite put a pin in exactly where I feel there is disagreement. On Daniel's side of things, it feels like there is this, like, demon in a Pandora's box that we need to not allow out of the box. And it is this truth, this object that's there that we need to make sure is contained. And I think on Arvin's side of things, there's just declining of the existence

of the demon in the first place. And that's an okay way of articulating where this line is. But, like, Arvin's fundamental premise, I think, is like, this isn't a demon. inside of a Pandora's box. This is a regular Web 2 tech company. This is the next iteration of Facebook, meta, Twitter, Instagram, just now in the AI age. And they're going to have power and control and influence like we've seen from all the current cohort

of AI companies. But it's not fundamentally different from any of the power that we've seen, like Mark Zuckerberg, a crew. And then Daniel is like, no, this is an organism that exists, that it will have life. It will have a mind of its own. It will escape. And that will change the face of Earth,

forever. And these are our two extremes. And so it's easy for me to, like, identify when they're polar opposites. But I'm still struggling to find this, like, line where you guys actually can't cross each other's, cross into others' territory. Well, it sounds like, Arvind, perhaps you just don't think that there's going to be superintelligence that soon. And you also separately think that even if there was, it wouldn't transform the economy that fast. Is that correct? Yeah, I agree. And there we go. That's the answer. Again, I mean, I'm, I'm, I'm skeptical. of superintelligence ever existing for some definitions of superintelligence. And the reason why AI is slightly outperforming humans at most tasks to me is not superintelligence is that there's

kind of inversion of cause and effect here in the sense that I think if we ensure our policies and practices are such that humans remain in control and use AI as tools, then those increases and capabilities are going to be effectively increasing what Zayash and I think of as human intelligence, as opposed to an unated, technologically unated biological human versus AI, which we think is the wrong comparison in the first place. So, Daniel, where is Arvin wrong here? So the definition of superintelligence that I was describing in this conversation is not slightly better than humans at most tasks, but rather slightly better than the best humans at all relevant tasks. And I do think that if you achieve that milestone, then things are going to be quite transformative. I think that

in particular, one of the relevant tasks is operating autonomously as an agent in the world, you know, doing whatever it takes to steer towards your goals. And that's where the analogy, that's where, I mean, I didn't, I think it's your words, not mine to say it's like this alien demon or whatever. Those are my words. But, like, that's why that analogy is relevant is that, like, that something that meets the definition of superintelligence as I've described it is a agent. It's, it's not just a tool is something that can instead operate fully autonomously in the world and, like, you know, learn as it goes and pursue its goals and so forth. And in fact, is better at doing that than the best humans at that skill, you know? And so it is like an agent that is, you know, comparable to humans

in that sense and in fact, it's superior to humans in that sense. And I think that even if we, even if we keep it capped at just slightly better than the best humans at everything, already that would be quite enough to cause this sort of analogy to be correct. Why? Why? Why? Why? Well, one way of thinking about it is think about the, it would be sort of like an alien species landing that was like basically like humans except just a bit better and not just better than like the average human but better than the best humans, you know? If this alien species landed on Earth, even if they weren't like dramatically better at anything, but if they're just a little bit better than the very best humans at everything and there's a million of them, already that would be like, whoa, like they're going to create like a colony somewhere. It's going to be like a really economically productive colony. It's going to be probably militarily powerful, too. And then you realize like, oh, wow, like their population doubles every six months.

What? Like, whoa, let's play that forward a bit. And in a few years, they'll have more population than humanity. Holy cow. You know, like, like that's sort of what you get. That's the sort of thing that you'd imagine. Basically, like, if it was literally aliens with like, you know, tentacles landing on Earth and they had that level of capability and that population doubling time,

it would be like, wow, like, in a couple years, they're going to be a competitor species to humanity. And then when you add in the factor that, like, actually, I think they're going to be not just like a little bit better than the best humans at everything, but like that as like a floor, but then also much better than humans at many things, you know, then that like only strengthens it. And then I think an additional thing as well is that like this whole like political angle too is that like I think if aliens landed on Earth, I think there'd be a natural coalition of like humans against the aliens. you know, and people would sort of be naturally hesitant to, like, put the aliens in charge of the U.S. military effectively and to, like, you know, basically have them run all sorts of companies and things like that. Although actually, now that I say that historically in colonialism, analogous things happened all the time, so whatever. But, but like with AIs, it's like, well, I think that the companies are going to tell everybody and tell themselves that, like, actually they're under control and that, like, the AIs are just doing what they want them, what the humans want them to do. This is so different. The depiction of aliens landing on Earth being super intelligent, kind of doing whatever they want, is so different, Arvin, than normal tech. Like, help us square this.

Sure. I mean, that's very helpful to me. It helps me pinpoint exactly where I disagree. I think this alien analogy, population doubling, et cetera, all of that is a choice. And I think it's a choice that we will not make. We don't have to make, at least. Wait, Arvin, so you can see it's possible. You can see it's possible, but you think.

that humans can essentially opt out of that to not let it happen? It's exactly the capability is not equal to PowerPoint. So we're talking about capabilities, again, that I think are going to be slightly better than unaided humans and equal to humans operating with the help of AI. And in that scenario, it's entirely a choice to say we're going to treat this as an alien colony. And so what that means is that not just let one AI system operate in the world unsupervised, but let entire collaborators of AI systems operate unsupervised, you know, give them this power, give them control over resources. And more than that, the humans versus AI framing to me, I think, is a deeply dangerous one, dangerous in the sense that it will bring about a lot of

the safety concerns that we are worried about. And the reason for that is, when you talk about it as humans versus AI, you're assuming that all of these AI systems are going to act toward the same goal in collusion with each other. But in fact, we should be designing all our AI systems so that the best defenses against a malfunctioning AI system is another AI system. So we should never get to the point where it's even human versus AI. Arvin, with my metaphor of like there's this demon in the Pandora's box, this demon is super sentient AI that's very powerful. And if it escapes, then it's game over. I think the thing that you contest is this notion that this Pandora's box is confusing and uncontrollable to humans. And I think what you would say is like, actually, no,

it's a totally reasonable box that we can think about and discuss and reason about. And we do have control over that. And I think Daniel would say it's like, we actually have less control over the nature of the box that constrains AI than we think that we do. Would you agree with that characterization? Yes. So, sorry, let me try to figure out what this analogy actually means in terms of the technical specifics that I'm more used to thinking about. So, yeah, I mean, there's a sense in which AI is unexplainable and uninterpretable. And sorry to take this to a more technical discussion, but it helps me make sure that I'm not misunderstanding the analogy

and misrepresenting my own views because of that. And there are concerns like deceptive alignment. AI might pretend to be aligned when we're testing it, but then when it's deployed, it might act very differently. But what we're seeing is that, yes, those concerns are already coming up with, even with, you know, current far below super intelligent AI systems. We don't even have to wait for the future. But actually, our best defenses against that are other AI tools in order to tease out this kind of behavior. Right. And so even if we're not the ones building the demon, if you will, neuron by neuron, you know, it's emerging out of the training process. We are in control of the box. and we are in control of the box through the help of other AI systems, and we can ensure the balance of power stays that way, and we can ensure that as AI capabilities improve,

it's making not just the demon more powerful, but the box itself more powerful. So that's one way of bringing it back to your analogy. Daniel, what do you think when you hear that? A couple things. First of all, A.22027, this is going back to something you said earlier. Air 2027 doesn't actually depict the AI as escaping. So, I mean, I think that could be, could happen.

AIs could escape, but also that's not actually what I was predicting. I want to distinguish between two risks, which I just want to put on the table so that for future parts of this discussion, one is the loss of control stuff, which is like, what if the AIs are misaligned, what if they pretend to be aligned, that sort of thing. But then entirely separate from that, even if you're not worried about that at all, there's the concentration of power stuff of like who's in charge of all the AIs, you know, and who gets to tell them what to do, and who gets to put values into them, you know? And, okay, having put those on the table, now to answer your question, I think control research is great.

And it sounds like that's kind of what Arvin's talking about, where basically you should build your system on the assumption that an individual AI system might not be trustworthy. And you should in fact assume that it's untrustworthy and that it's, you know, trying to kill you if it can. And then you should build a system of checks and balances and security permissions and so forth so that it can't. You know, that seems like a reasonable way to design things. And so there's this whole like sub-literature, which sort of grew out of the alignment literature called AI control, which is people working on like that aspect of things. And I think that's great. And I think that's important.

And there I would just be like, well, the companies aren't going to do this and they're not going to do it well enough and they're going to lose control, is my prediction. And I think that like in practice, like, I think it's possible in principle to have this whole system set up. But it's just not like the companies aren't really investing that hard. into making sure that this whole system of checks and balances is ready and has been debugged and has been like robustified to superintelligence or to anything close to that level of intelligence by the time we have those things.

And instead I think that, well, it's kind of like it parallels the whole alignment thing. Like with alignment, there's all sorts of techniques that you can use that maybe will result in more aligned AIs, things like scalable oversight and like interpretability and so forth. But then like the important thing to ask is like, okay, let's like what techniques are we actually going to use in practice and how confident are we that they're actually going to work in practice. And similarly with control, I'd be like, yes, impossible you can have this whole system of checks and balances. But like in practice, during the intelligence explosion, when the companies are automating all their research and so forth, I don't expect them to like invest that heavily

in this compared to what they would need to do. So can I say one thing to this before you move on to the next point? So this actually highlights another big worldview differences between us. Because for us, the role of the, what you call AI companies, we call model developers, is much, much less than what it is for you. It doesn't matter to us what the AI companies do or don't do. A lot of this research is happening anyway. It's happening in academia. We're actually compiling a big list of the many and varied kinds of AI control methods. So that's one thing.

The companies are not that powerful. And secondly, for us, these control methods have to have to. happen closer to the deployment point. So it's going to be the law firms and the hospitals who are finding these ways of AI control that make sense for their own industries and their own deployment scenarios. So that's another reason. It doesn't matter that much what Open AI does or doesn't do. And to give just one practical example of that, even in the simplest case of deployment of a relatively really innocuous kind of bot compared to the kinds of things we're talking about, we see many cases in the real world of exactly how hard it is for deployers to get away with deploying AI systems without

oversight if they're not innovating on these kinds of control methods. And great example is, I think, from last year, Air Canada deployed a customer service chatbot. Again, super, super innocuous thing. You know, it didn't have abilities to actually book people's flights or anything like that, just answer questions. But this bot was not very accurate, as you might expect, and gave people incorrect information about loyalty programs or something like that. And the customer lost money because of that.

They actually sued the airline. And Canada's courts concluded that the airline is actually responsible to make whole the monetary losses that the customer suffered. So this kind of thing is not an isolated example, has been such a serious liability issue for companies that even deploying, again, the simplest, dumbest thing, like a chatbot, has been going so slowly. Right.

And therefore, our prediction, and this is consistent with the observation so far, is that deployers, if they start to do anything meaningful with these systems, are going to face such serious liability risks, even under existing law, which I think should be buttressed by future AI regulation, that they're going to be forced to innovate on control methods before they get anywhere with serious deployments. And one example of such innovation. Some of it is technical innovation, but a lot of it is going to have to be non-technical.

there is a startup now that's offering insurance for exactly this kind of mistake that might be made by AI chatbot. So for us, insurance is an interesting AI control innovation. So those are the kinds of things that are going to be needed and that are actually happening on a pretty massive scale. People don't necessarily seem to realize how much because, as Daniel said, control originally started as a subfield of like AI alignment, but it's not anymore. If you look at all the control stuff that's happening in various disciplines like

human computer interaction and computer security and in the industry, It's actually bigger than alignment, and I think that story deserves to be told, and we hope to do that in a follow-up to our paper. Daniel, Air Canada chatbot, like, you know, it does seem like Arvin is saying every time something goes wrong with AI, basically our existing institutions will hold, you know, slap it back into place. We'll keep the alien species in the zoo if we want to use analogies and metaphors and have it live there. Like, why is that so wrong? I mean, that is what we've seen in the past. I know earlier in the conversation you were saying, well, of course, things are going to look like 2025 all the way till 2027, and then something happens.

And then we get this AGI moment. That's maybe another subject for discussion if we want to get into it. It seems very much like Arvind doesn't think the AGI milestone matters at all. Yeah, Arvin, you've said this. AGI is not a milestone. There's no sharp capability threshold that yields instant impact. Whereas Daniel, that seems to be part of the premise of AI 2027, right? The AGI line, March 27, we hit it, we cross it, and we start the intelligence explosion clock.

I mean, I forget if we actually use the term AGI somewhere in the text of A. 2nd27, but we basically don't. Like, I think instead we break it down into different milestones, and then we say it's continuous progress that we're just going to summarize as this series of milestones. The first milestone being automated coder, second milestone being automated AI researcher, eventually you get to super intelligence. But we agree it's continuous progress. There won't be any particular sharp, bright lines. And we depict that progress happening over the course of 2027 in the scenario. First, I want to just focus on understanding Arvind, your position.

So it sounds like you're saying companies are mostly going to be properly cautious about integrating AI into their stuff. And some of them will be incautious, such as O Canada, and then we'll pay for and then the industry will sort of learn from their mistakes. And this is going to result in, A, market pressure on the AI developers to make more aligned and more controlled AIs, but then B, separately, even if they're not aligned and controlled by the developer, like, the industry will sort of control them and, like, only deploy them in ways that, like, they can't actually cause that much harm.

So I'm definitely, there's definitely a huge disagreement with me. So we've identified a crux. I think that this is like kind of true right now. Like I do agree that like right now people are a lot they're rightly cautious about integrating AI into everything and this is creating this sort of like market force. However, as you yourself have said, there's plenty of companies that aren't cautious

and that are integrating things anyway and we'll see how that plays out. The main thing that I want to say though, which is maybe a disagreement, is that I think that once you've got the army of superintelligences, like I'm imagining, let's suppose that things go exactly as AI 2027 predict, which of course isn't going to happen. But suppose it's literally exactly that way.

Then, like, in 2028, Professor Arvind will be writing op-eds about how we shouldn't trust these super intelligences with all this power and responsibility and how they shouldn't be, like, integrated into the economy. And we shouldn't be, like, letting them autonomously run factories and design new types of robots and so forth. But then you'll be shouted down by, like, the president and all these other powerful people who will be saying we have to be China. And, you know, like, if we don't do, like, look at what's happening in China.

In China, they're deploying their AI everywhere. And, like, you know, and, like, it just feels like a political fight that you're going to lose. And I don't, I'm not saying you should lose the fight. Like, I agree, you know, like, it's dangerous to deploy superintelligence everywhere. But I'm just saying, like, politically speaking, I think the point to intervene is before you have the army of superintelligencies, not after it's already been built. And the companies are already, like, using it to lobby the government and so forth. like personally, I'll probably just retire at that point. And I'll probably just like, I'll probably just like officially give up and just like go spend

some time with my family or something because I think it's basically hopeless once you've gotten to like, you know, middle of, once you're at like March of 28 in our scenario, like I feel like it's basically hopeless at that point. Whereas it sounds like you instead have quite a lot of hope and think that like even if you let things get that. Well, okay, but now this contradicts what you're saying. Yeah, tell me what you think. I'm confused about where your position on this is. Sorry, I don't know what contradiction you identified, but I think one crux of disagreement I can point to

is that for us deploying AI with adequate oversight is not this losing business proposition. I think we are innovating in ways that we're able to ensure AI control in ways that are not simply a human in a loop, rubber stamping or second-guessing every decision. A lot of people intuitively think of AI control like that. And if you do, it seems clear that the more efficient thing is to deploy AI autonomously. But for us, those are, that's not the only option. And secondly, again, this whole army of superintelligence is framing. Again, I keep coming back to most tasks are like writing. And outside of certain areas like AI research itself, which is a purely computational problem, and then maybe, you know, purely robotic tasks, most sectors of the economy, again, if you take

law and medicine and things like that, there are such inherent limitations to performance that whatever army you build is only going to give you a slight efficiency improvement. And I think it will be a losing idea for most companies to trade off huge safety risks for those slight efficiency improvements. Well, I definitely also disagree about the slightness. I think that, like, it differs some field to field. And perhaps in some fields, it'll be slight, but I think in many fields, it will be massive, and that will cause this huge profits for the AI companies. And, like, it'll be this crazy thing. Not everywhere. Like, I agree in some fields, it'll be only a slight advantage. But, yeah, I think, trying to give back to the main thing that you were saying,

oh, yeah, you were more optimistic about the control stuff on the technical level. You're basically, like, it's not just going to be, like, trying to put humans in the loop. It'll, like, actually be, it sounds like you're saying that, like, by the time we have, you know, by the time 2027 rolls around, we will have control techniques that allow us to get the benefits of these really smart super intelligences without the risks as long as they're properly applied, basically, like as long as, is that, is that you're saying? Yeah, I mean, our timelines are longer, right? We don't think it's by 2027, but we also don't think we're going to have the potential for, you know, super intelligent AI without oversight giving us massive economic efficiencies by 2027.

Right. But like hypothetically, if the timelines were shorter than you think, and we do get these very powerful AIs by the end of 2027, you would be more optimistic than me that we could deploy them in a way that's sufficiently controlled that they don't need to be aligned in order for it to be safe. Well, I mean, if it is 2027, then we're massively wrong on the timelines, right? Then we would also have to be massively wrong on the timeline of controlled research. A lot that control research is happening, we see that accelerating over the course of the next decade or so. If capability massively accelerates but control research doesn't accelerate, then yeah, then we're in a bad place.

Okay. In that case, that's another nice point of agreement is that, like, I also think that if we had another, like, you know, 15 years before we got to superintelligence by the definition that I described, I would feel much more optimistic in general because I think that things like control research and alignment research would have just generally had a lot more time to cook. And in a bunch of ways, I think the world would be off to a better start if things were happening in 2037 instead of, you know, 2027 or whatever. Yeah. Yeah, so we agree there.

Throughout the course of this discussion, we've been covered in a number of places where the two of you disagree and a few where you degree, where you agree. I'm wondering if you could, you know, maybe start to summarize this for our audiences as we move to a close. And one way to do that is to kind of illustrate, I'm going to move us beyond 2027. Let's move to something more interesting, another date, which is like 2030, let's say, five years from now, okay? And so I'm going to ask you both this question through your lens of prediction in terms of how you're viewing AI and kind of the world and how this all plays out. But basically, what does AI look like by 2030 and what does the world look like? Arvind, first you in 2030. What are we looking at?

Sure. So let's start with the economy. I think some jobs are likely to be radically transformed. perhaps will face massive job displacement. But I think for most jobs, we're going to have task-by-task, gradual automation, along with lots of innovations in how to deploy these AI systems safely. We are likely to have a ratcheting up of the geopolitical competition. But I predict and also hope that we are not going to buy in wholesale to the,

to the arms race dynamics, because look, if I can make a slight digression here, those arms race dynamics are present even without AI. And that's the reason that war has always happened, where each side is fearful of the other side. And, you know, if that's the approach we're going to take to geopolitics, that's a very risky approach, even if you take AI out of the picture, right? So whatever diplomatic methods we've been using throughout the history of, you know, war and geopolitics, to be able to tamp down those tensions and to realize that escalation is not in anyone's best interest,

we have to apply all of that to the current moment as well, perhaps even more skillfully. And again, I'm cautiously optimistic that that's something we will be able to manage. And then finally, predictions in terms of other aspects of AI-related disruption, I think challenges to the education system, challenges to people's notion of identity, challenges to what art means. A lot of these things that are not really about the labor market, those are perhaps going to happen on a shorter time scale. And right now, they have been happening to a greater extent than economic disruption has actually been happening. So I would kind of maybe flip. I don't know if Daniel agrees with this perspective or not.

but the disruptions that we're going to see quicker are not really related to economic efficiencies, but rather people adopting AI in their personal lives in various ways, whether as companions that they form social relationships with or, you know, in the education system, and then you have a crisis of what the heck does college even mean? Are we teaching our kids the meaningful things? Similarly, what does art mean? Even back in 2023, I think Hollywood had to contend with what does this mean for, you know, for movies and for art? And then there was a movement to push back against the use of AI and art. So those changes happen very quickly and those are perhaps going to accelerate by 2030.

But I did not foresee large-scale job replacement, for instance, by 2030. Daniel, how about you? What does AI look like? What does the world look like, 2030? Well, if you go to AI-227.com, which depicts in hundreds of pages of detail, this sort of thing. And to be clear, I actually would guess I'd get on, I've updated towards slightly longer timelines since about end of last year

when we were writing the core story that became Mayah 2027. So now I would aim more towards 2028 instead of 27, but like, it's still basically the same. So, so like the short answer to your question is like, I expect the world to be utterly radically transformed by 2030, probably. I'm not confident.

You can ask me what I would, what I think it would look like if that's not true, you know? And then perhaps I would agree with Arvind that like I think that the I could imagine things being taking longer than I expect. I could imagine the current sort of like wave kind of petering out and resulting in lots of downstream applications

and lots of cool apps that use large language models but but the core dynamo of AI automated AI research just sort of like never really materializing. I could imagine that happening. It's not what I'm betting on though. What does that look like? Can you get for people who haven't read 2027? I know there's a lot more detail there

and there's like multiple paths we might take but like for people who haven't gotten into that detail yet, what does 2030, when you say radically different by 2030, I mean, some people are envisioning, yeah, Daniel, I mean it's radically different after the internet. And then we got, we got phones, smartphones, everything is radically different. This goes back to the start of this whole thing, like normal technology versus not. Yeah. Which, by the way, slight tangent, like, I think that, like, it's going to be normal in some

ways and not normal in other ways, you know, like, nuance. Perhaps you agree with that. But anyhow, if things go the way that AI-20s, you know, I think. 2027 predicts, which I think they probably will by the end of this decade, if not literally in 27, maybe afterwards, maybe before, whatever. If things go roughly like that, then basically it's like, first you get the automated coders, then you get the automated AI researchers, then you get the superintelligences by the definition just described.

Then you get even more powerful AIs that are not just like a little bit better than the best humans, but like maybe only a little bit better in some domains, but like a lot better in other domains. You continue to improve them, make them more efficient, faster, etc. you've got millions of them running, et cetera. They start assisting in the creation of new fabs to produce more chips so that you kind of have. And at that point, it is starting to look like the sort of aliens thing. The analogy becomes relevant because it's sort of like you've got this new species

that's better than humans in a bunch of ways and at least economically is competing with humans, if not militarily. And it's got a population size that's growing really fast because they're producing more chips and more factories and so forth. How long does it take to get to the point where, The next milestone I would mention would be the sort of like fully autonomous AI robot economy. And what that means is that, like, yeah, there'll still be humans around. But like the collection of factories and automated mines and automated this and automated that taken as a whole is self-sustaining.

Like it doesn't, it doesn't like crucially depend on humans for any crucial components of it, such that, like, hypothetically, if all the humans were just sidelined, it could continue to grow as an economy, and it could continue to build more factories and mine more things and do more research and so forth. And my guess is that that happens, like, maybe a bit less than a year after we achieved the superintelligence milestone. Maybe it'll take longer,

but probably by 2030 it will have happened. So probably by 2030, there will be all of these special economic zones full to bursting with all of these newly built mines and factories and chip fabs and data centers. And they might, still be humans working there, but the humans will be gradually being replaced by robots of various kinds. And, like, as a whole, it'll be as if there is, like, this new species that is now,

is now just, like, superior and is sort of self-sustaining. Now, if we're lucky, that new species will itself be sort of subordinate to humans and, like, we'll be following human orders and so forth. And that's also possible. Can I just say one quick thing? Sorry, Danielle, I didn't be to interrupt. I thought you were finished. I mean, so one thing I'll see here is that I think, for a lot of AI technologists, and I'm not necessarily saying for you, Daniel, you know, all of this might happen and that it might go in the direction of kind of either utopia or dystopia, utopia where these AIs are safe and they have provided for all our material needs and we can spend all our time on leisure and then the dystopia is catastrophe. But like, that's only in the

Silicon Valley bubble for the vast majority of people. Both of these possibilities are dystopian. almost no one actually wants this. So whatever the economic arguments are, this is only going to be possible if there is a complete collapse of people's democratic ability to influence the direction of things. Because the public is simply not willing to accept either of the branches of this scenario. Yeah, let me let's talk about that. So have you heard of pause AI? Yeah, of course. Yeah.

So like, I mean, they're basically, I'm not sure if they agree with you, but they at least agree on that. They're basically like, we just shouldn't do this. Why don't we just stop all this from happening? And it sounds like you're saying at least sort of something vaguely similar, that like most people in the world, if it was put to them in a vote, would basically be like, how about we just don't build all this crazy robot stuff? Is that right you're just saying? Well, I mean, how about we don't use taxpayer money to massively subsidize AI companies

and create special economic zones, right? So there's a big asymmetry here between intervening in the default course of action and pausing things versus, affirmatively, you know, having government work towards this future by doing all the enabling things that I think you agree needs to happen before this can be realized. Yeah, so excellent. So zooming in on that a little bit, this gets back to what I was saying earlier about the like, when will I just give up?

And I'm saying, like, politically speaking, we could imagine like an alternate version of AI-227 in which the government basically is immune to the lobbying from the companies and is like, no, of course we're not going to make special economic zones for you. Yes, we of course we're going to regulate you. Blah, blah, blah, blah, blah. Like, you can imagine that sort of alternate trajectory. And then it would take more than a year to transform the economy, you know? And it would, like, all that sort of stuff would be spread out and much more gradual. However, just as a matter of, like, politics, I think it's, like, going to be really tough to, like, win that political fight. If you let it get to the point where there's already this crazy race to superintelligence between,

U.S. and Chinese companies, and they've already built superintelligence, and they're already like networked heavily with the government, and like the government has been helping them with security to beat China. Like, like, at that point, I think it's just like very likely that the government will be like, yes, make the special economic zones, yes, we'll cut the red tape, yes, we'll subsidize you. You know, like, like, I kind of am expecting the government to be captured basically by the companies at that point, possibly literally with the help of the AIs. After all, If they're super intelligent, that means they're better at lobbying than the best lobbyists, and they're better at, like, you know, persuasion, they're better, like, that certainly can't hurt, you know.

I feel like every time I interact with chat GPT-O-3, right, it's persuading me as something. I've just got a lot of good ideas. I mean, you know, it could be bottom up like that. Well, whatever. The point is that I'm hoping to intervene before then. Like, I'm hoping to, like, not let it get to the point where they've already got the superintelligences and then we're asking the government to, like, make this be a slow transition, you know? Yeah, yeah.

Yeah, I totally agree that if even before Super Intelligence, if we get to the point where national policy is oriented around the race to AGI, then, yeah, I will retire at that point. I'm glad you guys are both not retired yet and able to give us some insights today. Thank you so much. I think the audience will benefit from this. The entire AI community will as well. Maybe I want to just end with this kind of like bonus question because this has been in the back of my mind.

I can't really figure this out. I mean, both of you are genuine, informed, intelligent, earnest people, right? Like, I very much believe that about the two of you and many in the space. And what I am perplexed, this is kind of the meta question going to this debate, is like, how is there such wide variance of opinion here? Like among informed, intelligent people. On your side, Arvin, you're like, hey, this is normal tech. We've been through this before.

We've seen it. on Daniel's side, he's like, this is a new alien species that's going to be super intelligent. We've just unleashed it on the world. How is there such a variance? I don't think I've ever seen a debate like this in anything I've ever done where you have like all of these very smart informed people reasoning about this and coming up with widely different conclusions. And the different P. Dooms and P. Utopias range from like literally Yon Lacoon, 0.01% all the way to

but L.E. Z. Yucowski and beyond. It's, like, going to happen. It's 100%. I don't understand this. Have you guys been able to make sense of this? It's not so surprising for me. I was trained as an academic philosopher, and there's a saying that, like, for all P, there was a philosopher who says P

and a philosopher who says not P. So, like, it's, I think that, like, it's actually just kind of normal for human discourse for this to be the case, that there's, like, intelligent, smart people who disagree with each other. when the stakes are this high, Daniel? Why would the stakes being high change that or something? Like, it's the reason why the philosophers disagree is not because, well, they also don't think

the stakes are that high. But, like, I don't think that, like, charging them up with lots of emotion would, like, suddenly make them sing kumbaya. You know? Fair point. Yeah, if I can also take a tab at this, we actually try to say a few words about this in the concluding section of the paper. We talk about what a worldview is and why worldviews can't be so different from each other. One, I think different predictions about the future arise from different interpretations of the present. So I think that's one big factor. So fundamental questions like, is AI currently being adopted rapidly and is it faster than previous waves of tech adoption like PCs when they first came out in the 70s or whatever? People disagree radically on that, right? So even on the things that

are in principle, empirically testable today, there is a big disagreement. And that's at least disagreement that you can potentially, you know, minimize by looking more carefully at the data, talking to each other, and so forth. But then when you get to really stuff about the future, there are assumptions that I think are essential to even start to think about these questions and those assumptions are different. A bigger one is our epistemic tools are different. So what weight should we put on which kinds of historical analogies? What is the role of probability estimation, and various other epistemic tools. And then values also, I think, start to play into this.

And all of these factors kind of reinforce each other. And people are in different epistemic communities. I think a couple of observers have talked about how these are kind of West Coast versus East Coast views. And for me, you know, they're, I mean, I was in Silicon Valley for three and a half years. I was doing startups in a literal reason why I wanted both physical and intellectual distance from that community was because my modes of thinking were different and I felt more

at home here. And so we kind of segregate ourselves into physical and intellectual communities based on our assumptions, values, epistemic tools, and so forth. And so it's not, I think, too surprising that you should get divergences as a result of all of that. Well, Arvin, Daniel, thank you so much for airing these divergences publicly so we can all come to our own conclusions. Whatever happens next, it's going to be interesting. Let's just say that. We appreciate you coming on. Yeah, thanks. Good luck, guys. Take care.