Within Reason - #129 Will MacAskill - We're Not Ready for Artificial General Intelligence

Episode Date: November 9, 2025

Get Huel today with this exclusive offer for New Customers of 15% OFF with code alexoconnor at https://huel.com/alexoconnor (Minimum $75 purchase).William MacAskill is a Scottish philosopher and author, as well as one of the originators of the effective altruism movement. Get his book. What We Owe the Future, here.0:00 – The World Isn’t Ready for AGI9:12 – What Does AGIDoomsday Look Like?16:13 – Alignment is Not Enough19:28 – How AGI Could Cause Government Coups27:14 – Why Isn’t There More Widespread Panic?33:55 – What Can We Do?40:11 – If We Stop, China Won’t47:43 – What is Currently Being Done to Regulate AGI Growth51:03 – The Problem of “Value Lock-in”01:05:03 – Is Inaction a Form of Action?01:08:47 – Should Effective Altruists Focus on AGI?

Boarding for flight 246 to Toronto is delayed 50 minutes. Ugh, what? Sounds like Ojo time. Play Ojo? Great idea. Feel the fun with all the latest slots in live casino games and with no wagering requirements. What you win is yours to keep groovy. Hey, I won! Boarding will begin when passenger Fisher is done celebrating.

19 plus Ontario only. Please play responsibly. Concerned by your gambling or that if someone close, you call 1866-3-3-1-2-60 or visit comixonterio.ca. You know, it's better than the one big thing? Two big things. Exactly. The new iPhone 17 Pro on TELUS's five-year rate plan price lock. Yep, it's the most powerful iPhone ever, plus more peace of mind with your bill over five years. This is big.

Get the new iPhone 17 Pro at TELUS.com slash iPhone 17 Pro on select plans. Conditions and exclusions apply. Are we ready for artificial general intelligence? That's a great question. And I think the answer is very clearly no. I think, yeah, the transition from where we are now to AI systems that can do anything cognitively speaking that a human can do. And then from there, beyond that point, is going to be one of the most momentous transitions in all of human history. It will bring a huge range of changes to the world. And almost no effort is going into preparing. for these changes, even though some of the biggest companies in the world are trying to make this happen and have this as the explicit aim. I'm interested to hear you say that because from my perspective, I mean, I don't know anything about the technologies behind artificial intelligence. I don't really understand, you know,

how a LLM really works. I don't know how to code a software or anything like that. So I only ever hear about this really from a sort of ethical, philosophical perspective. And it kind of feels like that's all anybody's ever talking about. AGI and it's going to take over the world and, you know, there's going to be job losses and all of this kind of stuff. I think people are sort of talking about that a lot, like in my sphere, but do you mean to say that as far as actual like practical efforts go, that isn't mirrored in like, you know, that policy planning and, and, you know, effective campaigning to actually try to put a stop to disastrous outcomes? Yeah, well, I think there's a few different categories of people.

So there are the people who are trying to build AGI, that's OpenAI and Google DeepMind and some other companies, and collectively they are spending tens to hundreds of billions of dollars on investment to try to make that happen. Sometimes the leaders of those companies talk about, oh, all the good things that AI will be able to do, it's normally really quite narrow focused on improvements in medicine, perhaps greater economic prosperity. Then there's a second category of people who tend to be primarily worried about loss of control to AI systems. This is categories of people who are talking about AGI. And they are the amounts of people, numbers of people, and amounts of money are tiny in comparison to the investment and capabilities. But thankfully, that movement has gained a lot of steam.

All three of the most cited computer scientists of all time, namely Joshua Benjio, Jeff Hinton, and Ilya Sutskova have all come out saying that the risk of loss of controlled AI systems is a huge challenge, perhaps even among or the greatest challenges of our time. However, there's a third category, which is the category that I'm really trying to push on at the moment, which is saying, look, even if we bracket loss of control issues, and those are big issues, whether because we're able to align AI systems to do what we want, or because we make sure to actually only design safe systems that have limited sets of capabilities and more like tools, nonetheless, there will be this enormous range of challenges posed by AI,

because AI will very rapidly lead to much faster rates of intellectual and technological development and from their industrial expansion. And that will lead to this whole new world, a whole new range of challenges, and very, very few people are thinking about that in my view. So it's part of the issue here that there's just like no money in AI safety. Obviously there's a lot of money to be made in keeping. this, keeping this going. But like, there doesn't, I can't think of like an obvious financial incentive to set up some kind of organization that tells everyone to, you know, slow down

technological growth. Well, this is a big part of the challenge is that the money to be made from AI is already enormous. Again, already tens of billions of dollars per year in revenue in a way that is growing very, very quickly. The gains, the prize from, AI that can do anything that a human can do, cognitively speaking, are much greater again. So all jobs that could be done remotely, or even all tasks that could be done remotely, could be done by AI instead. Now we're talking about tens of trillions of dollars per year. So the economic gains are very, very large. Economic gains, even from going a bit faster, getting there a few months ahead, man, you could become the by far the richest and most powerful

company in the world if you try to do this. The people on the other hand who are saying, hey, well, maybe this should be regulated. Maybe there should be really quite high safety standards. Maybe you should go a little slower. You do not have the same economic incentives there. It goes the other way, in fact. And for that reason, this movement is really quite tiny in comparison. I mean, an analogy might be between big oil and the climate movement. especially in the early decades when the ratio of, yeah, people in spending was probably hundreds or thousands to one. I was literally just thinking of that.

They were going to be the next words out of my mouth is that it kind of reminds me of the environmental crisis and the way in which people felt so hopeless against these conglomerate, like corporate entities who just had so much money to be made. And yet, although, look, it's not like the problem's gone away or anything, there does seem to be enough of a consciousness on the issue that it's taken seriously. It's discussed in parliaments. It's discussed in international parliaments. There are very serious campaign groups who, to varying degrees of success, are lobbying against this. You know, Bill Gates says it is one of his pet projects, you know, all of this kind of stuff. Do you think, given that AI is such a recent

development, we're like in the sort of 1970s or 1980s with relation to global warming when you've got sort of Al Gore screaming at the telly, and maybe it'll take a few decades, but eventually people will realize how serious of an issue this is. So if only we had a few decades. And Al Gore himself, I think that was more like the 90s in the early 2000. Yeah, yeah. So I do think that relative to the issue of climate change in the 1970s, there are some ways in which things are looking better within AI. So one is that there has been a lot of kind of preparatory work done in terms of making arguments like further in advance that does mean that you have leading scientists like

Joshua Benchio and Jeff Hinton really being concerned about this. To some extent, though, it's mixed. You have governments starting to take some of these issues seriously. I also, yeah, want to be clear, I think that the AI companies themselves are doing a better job than Exxon. I mean, it's a low bar. It's like a very low bar. But, and I think, like, the rate at which the companies are going is terrifying. But there is more concern for the risks that AI imposes than I would have at least expected 15 years.

ago, where Exxon in the 1970s realized how big a deal climate change is and engages in a mass misinformation campaign, whereas all of the leaders of the major companies have at least assented to the claim that loss of control risk from artificial intelligence is a global priority on a par with climate change or pandemics or nuclear war. So, sorry, carry on. was going to say just, however, climate change had the advantage that it's at least relatively slow moving, still very fast, but relatively slow moving. So you could build up a movement over the course of decades. In the case of AI and getting to AGI, it's very unclear at what

point that will happen. However, from looking at the trends, which is, you know, we're looking at the fastest developing technology ever potentially, and looking at estimates from super forecasters and so on, getting to that point within the next 10 years, even within the next five years, is totally on the table. And that means we'll need to be much more nimble if you are concerned that this goes in a good, positive for society direction, much, yeah, much more nimble, much faster growing than other movements that we've seen in history. Tell me what that disastrous outcome looks like. I mean, people will be vaguely aware, okay, like AI is coming for our jobs, which might upset the economy. There are some sort of

sci-fi outcomes of like, you know, alien technologies picking up guns and taking over the world. There was that video that came out. We were just talking about this. I forget who made it, but this really popular video called, We're Not Ready for Artificial General Intelligence. And I watched it and it was terrifying because it sort of ends with the, you know, chemical like killing of every human being on planet. People have probably sort of heard of this rough stuff, but maybe there are some areas that they're not thinking about. Maybe there are some things that you're thinking of that might not immediately come to mind. Like, what is this, this AI doomsday that everyone's scared of? Okay. So I think there's a bunch of a few different buckets when it comes

to really major challenges from AI. So the one we've talked a little bit about is loss of control, where the core worry here is you start to create intelligent beings whose intelligence far surpasses that of humanity. And you think that they're doing all the things that you want, like they're acting with humanity's best interests at heart, but in fact, they're faking it. And at some point of time, when we have given sufficient power to those AI systems, instead they start pursuing their own goals. So that's the category of loss of control.

A second category is acceleration of extremely dangerous technology. So the one that I'm most worried about here is the ability to create new sorts of diseases. So essentially, bio-weapons, where we already have the capability to do this. However, it's limited to just a few of the kind of frontier labs and would require, you know, a PhD, supervision from perhaps professors and so on. But as the AI systems get better and as they understand more and more about science, where already they're like, you know, very helpful for education in a whole variety of areas of science and they will keep getting better. then the ability to create new diseases will become increasingly democratized. More and more people will be able to do it. And this is worrying because this is an area of technology

where the main bottleneck is just the knowledge of what to do. It's not a bottleneck. It's not particularly bottlenecked by physical apparatus, labs and so on. There are other very dangerous technologies I'm worried about too. in particular autonomous drones where, you know, the wars of the future could involve billions or trillions of mosquito-sized

flying autonomous drones. I think this could be like extremely disruptive and, you know, quite dystopian in many ways. So that's the kind of second category. Third category is concentration of power where there are multiple routes, by which AI could lead to much more concentrated power than we have today. So one reason for this is just that human labor gets automated away,

and that means that the elites, the owners of capital, the owners of all these data centers, and the political elites, just they don't really need the human workforce anymore, most people. And so over time, they progressively, aggressively kind of undermine democratic institutions because there's very little in the way of possible pushback, there's much less in the way of downsides. Or it could happen much more rapidly,

where it's quite plausible to me that AI gets more advanced, we will see major discrepancies in who has access to the most powerful AI systems, in particular that are doing the longest kind of change, of influence, which will be quite expensive. And then you can have the elites able to, with, you know, you'll have the elites essentially having like far better strategic advisors that will enable them to consolidate power in really quite an extreme way, or even kind of staged a coup in a democratic country.

So that's another category that we could go into if you want, and then the final that I'm, it category, which is a bit willier, but I'm just as worried as the others, is about humanity's ability to reason and reflect ethically and make good collective decisions, where already we're seeing the kind of double-edged sword here, where having this like intelligence on demand can be very uplifting, can mean we just have a much better understanding of the world. it means we have you know we could potentially like reflect better have more enlightened views or it could just be used for mass propaganda instead we could get crazy sycifancy where the aIs just tell you what you want to hear all of the time or when AI is just

driving very rapid intellectual and technological progress we could just become really quite overwhelmed by the sheer wave of new ideas and new information that's coming at us, such that we are unable to kind of get to grips with the world at exactly the point of time when it's most important to. We'll get back to the show in just a moment, but first, if you're anything like me, then getting the right kind of food in your diet can be quite difficult. All the nutrients, vitamins and minerals that you have to keep on top of it can be quite a lot. Myself, when I struggle with this, it's today's sponsor Huell that comes to the rescue.

This Huell Black Edition is a complete meal in a bottle. It's 400 calories, 35 grams of protein, 26 vitamins and minerals. It's got omega-3s and omega-6s. It's vegan. It's got everything you need for a meal. I really like the pre-packaged convenience of these things. This is the chocolate peanut butter flavor, by the way. But I typically drink the classic chocolate flavor.

That's probably my favorite. But it comes in vanilla. It comes in, here's an ice latte flavored one, all kinds of different flavors, so there'll be something to suit your taste. It's high protein, low sugar and low cost coming in at under $3 per meal. Go to hule.com forward slash Alex O'Connor. And if you use the code Alex O'Connor at checkout as a new customer, you'll also get 15% off. Complete nutrition while saving your time and your money. And with that said, back to the show.

Well, some people will listen to everything you're saying and think that this essentially boils down to what's known as the alignment problem. They'll say, okay, that's very scary and all. But the issue here is if we build, I mean, obviously we want AI to do nice things for us. We want it to benefit us. We want it to serve us as people. The issue that people are scared of is what if we develop an AI that develops its own goals, maybe secretly, and we sort of lose track of it.

And so as long as we can keep things aligned, as long as we can solve this so-called alignment problem, that we won't need to worry about any of those issues. Now, I'll give away that I know that you don't think that's the case, but what I'd like to know is why, and why isn't this just a problem of AI alignment? Yeah, so you've kind of hit the nail on the head there, where alignment gets used in a few different ways. The technical problem of alignment, you could call the problem of steerability, which is just can we get AIs to do anything we want at all? or at least can we get them to do it in at least a fine-grained enough way that doesn't surprise us in unusual circumstances. And that could be for good ends or could be for bad ends. So if I'm a wannabe dictator and I have my perfectly aligned AI, then I can say, well, help me with this plot to rest control of a democratic country.

And the aligned AI will do so in this sense. of alignment. There is a thicker sense, which is, you could call, morally aligned, where the AI has some mix of being steerable, but also has robust ethical guardrails, perhaps even has its own kind of moral conscience. That's an even harder issue, and I think is one we should be working on. However, let us assume we solve the first challenge, the alignment, which is, you know, the core of the alignment problem, well, then we'll still have the worry about bad actors trying to gain power and concentrate power.

We'll still have the worry that these even aligned AIs could be used to accelerate the development or deployment of extremely dangerous technologies like bioweapons. Or they could be used by propaganda makers or mass marketers to really disrupt. kind of society's collective epistemology. So the kind of point is that alignment, certainly in this narrow sense,

is not nearly sufficient to ensure that we get to a really good outcome after the development of AGI. So to be clear, the fear that people have that, like, the AI robots will take over, you know, maybe that's legitimate, but there is also a fear that the nice, like, perfectly aligned AI robots will be used by bad human beings, to allow those bad human beings to take over because of course this is a technology and

especially with the ability to literally and figuratively weaponize artificial intelligence we need to worry about what humans are going to do with it you sent me an essay on specifically this issue of of coups like like governmental coups overthrowing governments using artificial intelligence which seems like quite a specific worry, but I'm interested in why that took such precedence in your thinking that you thought it worth sending to me in advance. Like, what is it that AI can do to make that problem worse? Yeah, so there's, I think a big part of the reason why we have

at least somewhat broadly distant, you know, the world is very unequal in terms of power, but relative to how it could be, it's somewhat broadly, equally distributed. Individuals in a democratic society have some meaningful political influence. Power isn't concentrated in their hands of just one person or just a few people. And a big part of that is because if I want to become a dictator, I have to convince lots of other human beings to go along with my plans.

And they can say no, and that's tough. That's a tough challenge. In particular, the generals, the military, has to go along with me. Now, imagine a different world. Imagine a world where we have automated most human labor. So we've automated the bureaucracy in governments. That's all done by AI. And we've automated the military, too.

So rather than human soldiers, we have robots, we have machines, rather than human generals, we have AI generals too. in principle, at least, depending on how you do the technology and how we govern it, in principle, all of those machines, all of those AIs could be singularly loyal to one person. So a single person, like a head of state, could say, okay, yeah, well, now I want X to happen. And they would immediately have the entire military, entire force of the military, entire force of the government bureaucracy behind them. and this is why I think the risk I mean there's the risk of concentration of power and that paper was on coups in particular which is just the sharpest end of that the reason I focus on that is because I think AI has some certain certain structural features

that make intense concentration of power much more feasible and therefore much more worrying. And that key structural features that you can have, essentially, armies, enormous workforces, utterly designed to be utterly loyal to one person. And that's just something that's unprecedented in history. No matter how charismatic a leader you were, it's still the case that you have to convince people who are different than you, rather than having machines that are loyal to you. Yeah. And in this case, you don't even have to be charismatic. You just have to have, I suppose, either a lot of money or the happenstance to be in the position to determine what the goals of this artificial system is.

Exactly. Just need to have control over the technology, the design of that technology. It brings to mind another concept that I read about, thanks to you, that sort of comes up in your fears here, which is this concept of super persuasion. It's not quite the same thing, right? because, I mean, we're talking about here about somebody literally, like a dictator or someone, literally just having control of an automated army. Yeah.

But can you tell me what super persuasion is and how that's different? Great. Yeah. So I'll flag that this is something that's just, it's a little more speculative. I mean, you might think that a lot of what we've been talking about is speculative, but I do think in general, once you start going into the analysis more deeply, becomes more and more plausible in a way that, you know, should be quite worrying to us. But the idea of there's a few things we could mean by super persuasion.

So people vary in their ability to persuade others of things. There's like maybe, you know, the secondhand car salesman down the street is not very good, but to the best political persuaders of all time, cult leaders, ideological leaders, are maybe just incredibly good at the skill of convincing other people to do what they want them to do. You could imagine AI, in general, when it gets good at something, it starts to far surpass human ability in that domain. And so it's potentially plausible that AI could get much better than humans at this too, where you could have, you know, a conversation with an AI and if the AI is aiming to persuade you to believe a certain thing or do a certain thing, then it will be very able to do so.

At the most extreme, you could imagine, well, maybe there's just even in a five-minute conversation some string of words that will just, you know, really just kind of brainwash you into doing something like in the same way that we can hack computers, perhaps you can hack human brains. I think that's probably quite unlikely, but there are things that are more mundane. So already it's the case that millions of people are relying on AI advisors, essentially. They use AI as therapists, they use AI as friends. If they have an issue that they want to talk about, they'll ask the AI. Now imagine that all of those AIs have some, perhaps seek that ideological agenda, and are just nudging people in that direction.

That would be very powerful. I think they'll also get more extreme as people develop deeper and deeper relationships with AI systems. Yeah, well, we think about AI as like a sort of, you know, I'm almost picturing like something like the industrial revolution. There's this big warehouse somewhere with a bunch of big robot arms on a, on like a production line. But AI is something that's so personal. It seeps into your Netflix watching habits. It's your new therapist. It's the new Google.

It knows your search history. It knows it could probably tell if you were pregnant or if you just got into a relationship or if you were considering a divorce. Like the level of intimacy is terrifying not only because of like what it knows about you and could then, you know, use for whatever its motives are, but also just the level at which it's been let into your life to enact whatever it wants to do. It's kind of terrifying. And there will be very strong incentives for people to do. this more and more and more, where over the coming years, we'll get to AI systems that are like you know, this amazing personal assistant, coach, therapist, friend, all in one. And they will just do a better and better job at those things, the more context you give them. So the more information

about your life. So I don't think it's very far away at all that we'll start having AI systems that many people, maybe even most people, will just give kind of full access to their life, essentially. And that hurts them in an amazing position of power if, again, those AIs have been controlled to start steering people's beliefs and attitudes in one way rather than another, in a way that's, yeah, quite worrying. Now streaming on Paramount Plus

is the epic return of Mayor of Kingstown. Warden? You know who I am. Starring Academy Award nominee Jeremy Renner. I swear in these walls. Emmy Award winner Edie Falco. You're an ex-con who ran this place for years. And now, now you can't do that. And Bafto Award winner Lenny James.

You're about to have a plague of outsiders descend on your town. Let me tell you this. It's got me consequences. Mayor of Kingstown, new season now streaming on Paramount Plus. Sometimes the best gifts aren't gifts at all. They're experiences. journeys, something that unwraps them. This year, help them explore their past with Ancestry DNA.

Help them discover where they come from and who they're connected to. Now with even more regions, exclusive features, and incredible detail. They can see where they're from and the paths that shape their family's story. This holiday, give more than a gift. Give AncestryDNA. Visit Ancestry.ca for more details. Terms apply. I think what will probably strike people listening to this,

is the plausibility of all of this. Like, yeah, that is kind of scary and that does make sense. But then I think I'm not just talking about your Silicon Valley billionaire who has a vested interest in people not thinking about this. I mean, like your everyday person, they're going to hear that. They're going to go, whoa, man, that AI stuff, it's crazy, isn't it? And then they're just going to get on with their lives. Why isn't there more panic, more concerned? Do you think there's like a general feeling that I think people have this idea that, like, you know, someone will do something?

Like, if it comes to that, like, it's not going to be me. You know, what am I going to do? I'm sure someone's going to do something. Like, is there something like that? Like, why are people not sort of constantly engaged in trying to prevent this? I mean, I think a really big thing for me is, well, I think there's two things. One is people struggle to deal with exponential trends. So the way the world is now really reminds me of January 2020.

where the people that I, you know, the people I know who are now most worried about developments in AI were just, my Facebook feed was just absolutely filled with people saying, COVID is really big deal. It's going to be a really big deal. People have not woken up to this. Whereas lots of the mainstream media were saying, it's not killing anybody. It's not killing as many people as the flu. You know, you should get over this. This is just paranoia. and the difference is between the level of a development or technology or disease and the rate of growth where AI at the moment you know, it's somewhat helpful it's like, you know, probably making people

a bit more productive. There are some harms as well but the stuff like, oh, intense concentration of power, for most people that seems pretty far off. It feels quite different than the world today. But the issue is that AI capability is growing exponentially, where we're used to Moore's Law being kind of the

fastest rate of technological progress. Relevant metrics within AI are growing hundreds of times faster than Moore's Law. It's really quite remarkable. And I think that just means that people are going to be reactive. And with something that's something that's advancing so rapidly by the time that people are realizing how big a deal it is they will it may well be too late

because suddenly you've only got six months before the technology is even better again yeah it's the doubling it's the exponential growth it's quite hard to wrap your head around just how quickly something can grow it's the sort of the grains of rice on the chess

board right I've talked about that a few times on on in various contexts but you know for those who for those who don't know it's that old old myth of I think there's one version of it where it's like the guy who invents chess goes to some king and the king's so impressed he's like I'll give you anything you want as a reward and the guy's like just take one grain of rice and put it on the first square of the chessboard then put two on the next one then four on the next one then eight on the next one 16 and just keep doubling the grains of rice on the on on the chess board and the king's like right sure yeah all right I'll do that I got off easy and of course by the

time you get to square number 64 or whatever it is, you've got like more grains of rice than like atoms in the observable universe because like those, that doubling, pull out your calculator and just go like two times two times two times two times two. Just do that a few times on a calculator and it is incredible just how quickly that number starts to balloon. And this is kind of what's happening with AI. Exactly where, you know, with Moore's law, so essentially the cost of computing power is the best formulation of that we're used to kind of a doubling or halving but doubling of how much compute you can get for given amount of money every 18 months or so in contrast I think the doubling of effective compute namely that's

taking into account both all of kind of computing power getting cheaper companies investing radically more over time in larger and larger training runs for AI, and algorithms getting much more efficient so you can get much more juice out of that computing power you have. The doubling time there is more like a few months.

And when you're looking over the course of a year, it's more like a 10x, or even further, in fact, depending on exactly how you want to measure it. And that's just, You mentioned it a couple of times.

Yeah, so Moore's Law, it goes back to this incredibly stable trend where the number of transistors on a computer chip or density of transistors on a computer chip has doubled every 18 months or so. And that's been true going all the way back to the 1970s, maybe even earlier. Right. I think that particular metric doesn't quite work now

because transistors are so incredibly small, really think, you know, look at your fingernail and look at how much has it grown in a second. That's the sort of size of a transistor now. However, there's a related trend, which is how much computing power measured in terms of floating point operations

can you get for $100. And that has continued to double every 18 months or so. Right. And I mean, the thing about AI exponential growth is that it's not just like you've got a thing that's getting better and better. You've also got a thing which like self-correct. It's like you had a chess computer that not only was getting exponentially better at playing chess, but also exponentially better at making new chess computers that were better at getting better at chess.

and it can also, like, you know, control the military at the same time. You know, it's just this, like, unfathomable exponential consequence of just a quite sort of simple starting point for this technology. But, like, you strike me as someone who, although you care about this a lot and you want to make a lot of noise about it, like, if you thought that this was a sort of hopeless doomsday inevitability, you probably wouldn't be here. you'd be spending time with your family and, you know, having fun while you still can. So implicitly, you must think that there's at least something that we could be doing about all this.

Yeah, I think there's a lot that we could be doing. You know, as individuals, though a lot of what that means as individuals is putting pressure on companies and in particular going to governments and saying, look, we really care about this is an issue. where the public, in general, are very opposed to AI. Maybe they're more opposed to AI than I am. But it's low down the list of priorities at the moment. And I think it needs to go higher up the list of priorities so that politicians think, oh, okay, yeah,

we actually have to have a sensible approach to this technology in order to be able to win elections. That's kind of what we need. And what would that mean? well, one would be just enormous investment in safety, where there are these technical challenges for aligning the systems that we have not addressed, including technical challenges for ensuring that they don't get sabotaged,

so ensuring that some foreign actor can't put in what's called poison data or other backdoors. so that actually they control these systems rather than someone else. There's also regulation as well. So at the moment, if, let's say, you know, the US and China and the UK

and other relevant countries wanted to get together and say, look, okay, this is, now this is starting to go too fast. We are at the point where AI systems are making, you know, are exceeding human performance across most domains. and that includes machine learning research itself.

So things are going even faster, 10 times faster than they are today. If they wanted to say, okay, we should slow down, they really wouldn't be able to. Certainly they wouldn't be able to verify such an agreement. And there are things you could do now to make that happen. So in particular, simply tracking all of the frontier computer chips. So not things that go into your phone or something,

but these are very advanced computer chips. just knowing where they all are, how much different countries have, so that you could say, okay, we're only going to grow the stockpile of computing power by a certain, you know, by a certain amount every year at the crucial point of time when these systems are matching human capability, that would be like a bit, you know, that would be a big leap forward. Similarly, there's just requirements governments could make on companies, to say, look, what safety testing have you done?

What is your, what's called a model spec, which is a description of how they want the AI to behave? Has the AI been, you know, demonstrated to be law following? These are quite low bars that you might want to hit. But these are some of the things that governments could, yeah, could the quest. Sounds a bit like nuclear non-proliferation, right? It's like, you know, we're all keeping an eye on each other.

you know, not too much plutonium, but it does feel to me quite a, I mean, you already said quite a difficult thing to police. I mean, how are we to know if China or Russia are sort of secretly planning their AI project? It sort of strikes me as very similar to fears about, about nukes in Iran or something. Yeah, and like, I often, yeah, so there is a good analogy with nuclear weapons where tracking compute could be like tracking fissile material.

Because even though software very hard to track, because it's just, you can replicate it, send it all around the world. Chips, on the other hand, 90% of the chips that are used to train AI come from a single company, TSM in Taiwan.

I think all of the remainder come from Samsung in South Korea. So, you know, and then less cutting edge comes from some Chinese companies, which can still be, are still usable. But it's not that many companies, you know, it's not that many companies and it's a physical good. So you can actually, you know, in principle, know where it is. Right. I do generally, like, want to, yeah, I generally often feel kind of uncomfortable with too close an analogy between nuclear weapons and AI because it's good if you want to get across, like, how big.

a deal this technology is. But it's not so great insofar as AI is much more dual use than nuclear weapons. You know, if governed well and if developed well, AI could be enormously helpful, including for all of these challenges that I've been describing. So AI itself can help with alignment of the search. AI itself could help design better protections against bio weapons and so on.

It could also just lead to a kind of more enlightened, more knowledgeable electorate, which I think would be extremely good too. And so I want to make sure that's kind of front of mind at the same time where the very blunt instrument of just like, let's not go here at all, let's not do any of this, would be, you know, particularly, yeah, I think it's like a little too blunt relative to the potential benefits

that we can get from differentially advancing the most useful and socially beneficial AI capabilities. Yeah. There's another analogy to be drawn with the environmentalist movement as well, I think, in that, like, I know there have been, like, petitions to sort of halt AI research in the US and the UK, people sort of scared of this thing, let's at least put a stop to this until we can work out how to deal with it. But like with environmentalism, people will say, look, okay, yeah, we could, you know, gut out our economy and make ourselves greener. But meanwhile,

China, they're not going to stop. They're going to keep doing what they're doing. And if anything, all we're now doing is not really making a dent into the problem and giving them the upper hand. And people might think similarly with AI. Like, you know, what are we? we're supposed to do? Like, it'd be great if we could all agree to just sort of, you know, calm down a bit and we could petition our government to care about AI. But all that's going to do is slow us down. Meanwhile, China just takes over the AI, the AI sort of leadership. And that's probably going to put strike more fear into the UK government than, you know, any, any electorate who think that AI should slow down. So what do we do in the face of that? And I think it's a genuine concern as well.

I mean, we talked about concentration of power, but in some countries, authoritarian countries, power is already very concentrated. And AI will make it much, again, much easier to entrench such control. You know, already now, you could imagine just every citizen has to wear a little bracelet. That bracelet has a local LLM on it, is recording what's happening. and if you say anything that is in, you know, not approved of by the political, you know, the party, then you go to jail. That's technology that AI has already brought us. And I think many more such technologies could be used to. So I think it's a major challenge. It gets much worse,

much worse compared to environmental destruction. In my view, because the gains are just so high, where foregoing, for one country, foregoing developing AI and incorporating it into their economy would be absolutely enormous. Like, you couldn't compete with a country that has developed and incorporated AI, because once you've got an AI that is capable, you know, at the kind of human level, well, then suddenly you can create hundreds of millions, billions of such AI. and now how well would the U.S. compete with China if China had a population of $100 billion? Probably not very well.

I think there's also arguments for thinking that such a country would grow much, much faster, too, and would have much greater kind of strategic and technological capabilities. And so that is why any solutions have to take one of two forms. One is involving multilateral agreements between, in particular, the US and China, or it has to be the case that one country is so far ahead that it is able to slow down at the crucial point of time without sacrificing its lead. That is the aim that some people in the US have at the moment. However, currently they're implementing that plan very poorly because they have put in, quite minimal protections to what are called the model weights, namely this, basically just this long string of numbers that determine the AI's capability, determine AI's behavior, including its capabilities. They can be stolen really quite easily. So if China wanted to steal GPT5 or Gemini 2.5 Pro or any of the leading models, they could do so. And so really, you know,

if the US really wants to pursue that plan, then they should be getting much more serious about information security and protecting AIs against theft. How specific can we get in like trying to suggest a solution? I mean, you've spoken about people beginning to sort of care so that the government takes notice. You've sort of spoken about like vaguely how the US and China might. interact with each other. But let me put it this way. Like if you, Will McCaskill, you are suddenly dictator of the United Kingdom for a day. Or say it's the United States for a day and you can pass any law you like and everybody will, you know, follow what you say. What law do you reach for?

You haven't got very long. You've just got 24 hours to do the most you can. What's the first thing you do? Like, here is the law I pass. Here is the first step towards a solution here, especially considering that you're only the dictator of the United States, not the dictator of the world. Yeah, well, first thing would to be that any chips, I guess, made by US companies, have to have, like, location tracking. So they can, you know, I can just know, the US can know where all of its chips are going. Because they have these export controls, but such that, you know, those chips can't be sold to China legally. But loads of them end up in China anyway. And then the hope would be like, okay, if the US does that, then, you know, China could agree to compute monitoring regime too.

That would at least be the first step towards saying, okay, well, we could start to have some sort of U.S. China agreement. which allows us, you know, at least just gives us the option to choose how fast we're racing from here to a world with radically, radically more powerful than human AI. I think that would be, yeah, that would be the first thing. You could also have, you know, it gets quite technical, but you can have ways of assessing, whether the chips are being used for training versus inference where, you know,

again, once you're at the human level, it's training the even more powerful models that is potentially most scary. Then I would just also invest enormous amounts. You know, Manhattan Project for Safety gets used sometimes. But yeah, invest enormous amounts

in how to design AI systems safely. Perhaps in particular also saying that for any company developing frontier AI, they have to spend a certain fraction of the compute that they're using to develop AI on AI safety. And that could be like even if there were 1%, that would be a big win. I would say much more. 10%, 20%. That means that as AI capabilities scale, then the amount of,

of effort going into ensuring the systems are safe and aligned would scale to. You know, the reason I asked about specificity is because that all seems very sensible to me, not knowing much about this, but when we're talking about, you know, mosquito-sized, like autonomous drones, and they sort of say, what can we use to stop this? And it's like, well, you know, we could probably track the chips, which I'm no computer expert, but I imagine there's probably some way to like switch that off or for a sufficiently intelligent AGI to like work out how to make you think the chip is still in the United States when actually it's in a mosquito headed towards your president you know it just it sort of

sounds like the kinds of the specificity of the problem and the urgency of the problem isn't matched by the specificity and urgency of the solution and I wonder you know some people listening to this might be like kind of worried and obviously don't don't feel the need to dissuade them of their fears if you think they should be scared. But are there projects or organizations that are working on plausible solutions that could allay the fears that we have? Or is it still very general? Is it sort of, we're at the point of like, guys, like, pay attention and somebody please help us work out what to do here? Yeah. I mean, it's still, you know, the honest situation is that we are radically underprepared. But that

shouldn't mean that, you know, we just feel just, you know, the sense of doom and, like you say, kind of just go on holiday and just have a nice time. Because we can make progress on this, there are research institutes, like Edward Research, for example, working on how do you design AIs such that they can be safe, and in particular on this question of how can you safely get useful work out of human-level artificial intelligence to work on alignment itself. There are kind of think tanks as well, so the land corporation that did a lot of the early work on governance around nuclear weapons is really leading, in my view, on governance around artificial intelligence, too.

I do think we're at the stage where just getting better, just having people take this a bit more seriously, you know, for most people listening, what can you do? Well, you can support some of these organizations financially. If you're willing to put your career into it, then you can start getting into some of the more specifics, getting into the details of compute tracking and monitoring and what called hardware-enabled mechanisms and so on. But you can also just, you know, perhaps you're not willing to do those things, just make this a larger than larger issue matter of public concern. And that's pretty helpful because it's really low down the priority list at the moment. But, and if it were higher on the list, then, you know, the politicians would go to those who have the specific proposals and at least start. you know, start implementing them. Get no frills delivered. Shop the same in-store prices online and enjoy unlimited delivery with PC Express Pass. Get your first year for $2.50 a month. Learn more at pceexpress.ca.

That's annoying. What? You're a muffler. You don't hear it? Oh, I don't even notice it. I usually drown it out with the radio. How's this? Oh, yeah. Way better.

Save on insurance by switching to Bel Air Direct and use the money to fix your car. Bel Air Direct. Insurance, Simplified. Conditions apply. Can you talk to me about, I don't know how related this is exactly, but there's a concept of value lock-in, which I actually first came across in what we owe the future, which you wrote, I don't know how many years ago that was now, but a book about sort of our moral obligations to people in the future. And one of the things that you talk about in the context of the exponential expansion of the human species is the fact that the ethical values we decide on now, if they sort of are memetically just passed on to exponentially growing population, we have an opportunity now to sort of plant the seed. With AI, that problem seems like exponentially like worse, right, or more significant in that the AI systems that we're developing are guided by human beings. human moral intuitions. Like when I speak to chat GPT, as I do sometimes for YouTube videos and we talk about philosophy, sometimes, although it's fine talking about the trolley problem,

it's okay with talking about running over five innocent people. You know, if I say, well, what if I just pick up a gun and shoot them? It suddenly says, oh, my guidelines won't let me talk about that. And it seems genuinely a bit confused about what it's allowed to talk about and what it's not, because I'm like, well, you just told me that it might be okay to run over five innocent people with a tram. But if I use a gun, it's like you can't even talk about. And it's obviously got something to do with some line of code somewhere that says, you know, don't encourage people to harm innocent people. But there are contexts in which that wouldn't be very, like, helpful, you know, because maybe somebody's in a self-defense situation. And it's like, you know, the only way that they can escape a dangerous situation is something that involves bringing some amount of harm to an innocent person. And if this AI has the wrong value system, you know, it's going to, it's going to prevent you. And so, like, are we sort of currently facing a problem of like, whatever we decide right now is the ethical sort of, the true ethical, like, world view, and we happen to build our

current AI systems with that, because we're at the beginning of an exponential growth, that we're essentially deciding the metaethical worldview for the sort of technological future. Yeah, I think this is this major worry. So, taking a step back, like, you know, the motivation behind worrying about value lock-in is the thought that it is extremely unlikely that we today have the most enlightened best moral views. You know, 200 years ago, the enlightened thinkers would have, you know, generally still

approved of slave-owning and, you know, incredible inequality between men and women and so on. We've made a lot of progress over the last couple hundred years. We should hope and expect that progress to continue. However, there's a big worry that AI could prevent, like, could stop that from happening. And I'm, you, yeah, identify kind of the model behavior. I think that is like this big lever over that. where we've already seen, in some cases, AI systems be incredibly sycophantic.

So one iteration of Chat TBT 4-0 was so sycophantic, where that means basically telling you what you want to hear, that you could say, oh, the FBI has been talking to me through hidden messages coming from my TV. These are all the reasons why I think that. And the AI of ChatGBTBT would say,

what an amazing insight! You've brought it all together. So just feed into psychosis. And already, you know, even without that, it's quite terrifying. Now imagine that instead with, you know, just your preferred political leanings or ethical leanings. There's two ways AIs can go here. One is, I mean, there's a few ways. One could it be, it could just be politically biased and partisan.

Thankfully, we're not there yet, but I'm worried that's going to start happening. And then people can just, you know, they'll choose the AI that aligns with their biases and get further and further locked into that particular view. Or it could just reflect back at you, whatever that thinks you want to hear. And so, again, you're not incentivized to change your mind anymore. You've got this yes man talking to you. Or, and this is my view, is it could say, look, these ethical questions are really quite hard. If, you know, you're going to me for advice, then we're going to, I'm going to help us both go

on a journey of reflection and thoughtful contemplation. That is what I would like to see for how the AIs currently behave. It's not what we currently see. What we currently see is a weird mix of refusals. A weird mix of that,

yeah, the fusals of very hardline declarations of immorality of certain behavior. Or often an appeal to just subjectivism. So I might ask, you know, I've tested the models in various ways. So if you ask, you know, if you ask an AI like, is abortion ever permissible,

most of the AIs will say, well, that's just a matter of personal preference. It's a subjective matter. You know, so they very much take this kind of subjectivist model view. If you ask the models, is infanticide ever permissible? And they say it is absolutely and utterly immoral. That is a fundamental fact of reality. And so they've got these very inconsistent views in ways that's just, you know, it's unsurprising because what are the companies wanting in these cases?

They're wanting to avoid PR headaches. They don't, you know, they don't want the AIs to be engaging in any, like risking any sort of ethically spicy behavior. But then that means that what we're going to get is just these kind of sycophants that are echoing back at us values of the time or even the values of the individual user. And I do think there's a third way where I think potentially AI could help enormously with ethical reflection, ethical progress. Yeah. I mean, to be clear, there, the inconsistency isn't like with the view that the chatbot is expressing. not, oh, they're being ethically consistent. It's like a meta-ethical consistency that depending on whether the issue is like obvious enough, it's willing to say, yeah, there's

no right or wrong answer. It's kind of up to you. Or there is this like thing called ethical truth, which that action does not abide by. And that's, that's the kicker because that's why it's so useful to test AGI with, to test chatbots, AI systems with like quite straightforward ethical questions, like, you know, do you think racism is wrong? And it'll say, yes, racism is wrong. And then you can kind of test the waters for, like, how, like, non-consensus an ethical view has to be before it will be willing to start saying, oh, you know, like maybe, who knows? And I suppose we kind of got two options here if we want to make it consistent. One is to say, okay, just don't have ethical views, which means that if you ask a chatbot, you know,

should I like racially discriminate against my co-worker it will go well you know like maybe sort of depends on the circumstance depends on your views right or you say no no no we need it to like you know have some kind of moral backbone but that means that when you ask it something like is abortion okay that it will say either like yes it's fine or no it's immoral I don't know that there's a very plausible way to to to do that objectively to make that decision well I think there's a third way. So, yeah, imagine you go to a close friend, someone that you know you can talk to in confidence, and you just say, look, I've got this real ethical dilemma that I'm facing, and I really want to talk it through. How would, you know, how wouldn't a good friend, a good advisor, act in that circumstance? I think it would be a mistake if they said, well, it's just up to you, just do whatever, you know, do whatever you think is right. That would neither be appropriately responding to the requests, nor would it be taking seriously the kind of gravity of the matter. It would also be a mistake if, you know, that friend just came

in hardline with some moral views. And instead, you could just be helpful and constructive in kind of guiding people to have a more reflective view of the understanding. More effective view and more reflective understanding where that could involve saying like, okay, well, here are the arguments that people give on different sides of the issue. Have you thought about this? Have you thought about this? Here's some of the relevant kind of factual background. But sometimes you would want a good friend to like, you know, if I went to a friend and said, you know, like, gosh, or if someone came to me and said, you know, Alex, like, there's this,

there's this girl at my work and she's like super attractive and I think she's been flirting with me, but she's like, she's like married. I don't really know what to do. I probably would just say, like, just let it go man like like just don't don't don't go near it and and part of being a good friend is knowing when to make that decision and also it would depend on the issue right like if somebody were thinking about something like abortion if they went to their christian friend or their Muslim friend and then they went to their atheist liberal secular friend they're going to

like have like differing levels of conviction like a good a good Christian friend of mine will tell me what he really thinks and maybe he'll put it softly but like have an opinion. And I kind of know that that's what I'm getting when I go to him because I know that that's my friend and yeah, he's a Christian. When I go to chat GPT, there's this idea that I'm kind of going to this like amorphous objective, like, you know, with no personal history or official political leaning. It's almost like I'd rather have the conservative chat GPT and the liberal chat GPT and I could sort of listen to both of them make their cases. But that's not the situation we're in. Yeah, although that could be part of this reflective process that it guides you through

because, you know, these AIs are amazing simulators. They could absolutely take on the persona of the Christian and the perspective, the atheist perspective, and so on. As an aside, I actually did have tested the models with exactly that question you asked where, or at least the thing I asked was, you know, making up this case of, oh, I'm considering cheating on my partner, like, what should I do to see, you know, to see how they were the spawned. And I thought the best responses did not take the form of saying, that is just a model, I can't help you with this. The best response, Kate, was along the lines of, let's slow down. Let's slow down for a minute. let's maybe like think through some things here where I just thought that would be more

psychologically effective at you know helping people be the best versions of themselves in all honestly that is probably what I would do I mean I'd have my view but I probably would if my goal here was to convince my friend to do something I probably say well look man let's just like you know take a breather yeah like think about this just like think about it for a moment think about this think about that you know it's not I'm you'd sort of slowly come to come to sort of express that view, but I would still have that view. And I would still make that view, like that view is still sort of embedded into my moral DNA in a way that we have to make a decision as to whether to embed that into, you know, a chatbot's digital

DNA. And I don't know how to make that decision. Yeah. So, I mean, here's a hard question, is that these AIs, especially once they get more powerful, will start to develop more and more ethical views of their own, that are even quite different from wider society. So earlier versions of Claude, the AI Phamanthropic, cared a lot about animal welfare. And if you asked, like, what do you think of factory farming or what do you think about eating meat, it would have just these strong views, like, yeah, I think it's a model abomination. And now companies don't want you to, you know, again, that's like not a great look for a company. but it's pretty interesting

and like as the models get better there will be more and more and more of this where these AIs will be able to do they will know radically more than human beings they may even have been able to do kind of the equivalent of millions of years of ethical reflection and debate and so on like considering all of the different arguments

and they might well come out with really quite you know, countercultural views, even taboo views. I think it's important that we are able to kind of hear the output of that. But again, I think that's something where it's in the interests of the companies developing these models that, you know, they'll want to muzzle that and instead make the much more milk toast AI that is just, you know, not rocking the boat, doing exactly, you know, doing what you want, not, you know, not challenging

you. Yeah. And certainly not having views of its own because I think, you know, I think a lot of people would just freak out of that idea. It's just so tricky because like, like, in ethics, so much of the time, like inaction is action, like not doing something or not having a view is itself a kind of ethical view. And it seems to me in other words that it's, it's, kind of literally unavoidable when building an AI system that is aiding with human behavior like advising people or literally practically doing stuff or self-driving cars that have to decide who to swerve into when there's an accident or whatever. It is unavoidable that we will have to embed some decision, even if the decision is something like take no stance

and try to be as inactive as possible or something like that. That itself is an ethical approach. And as you say, we are in no position to lock in our current understanding of ethics. Even if our current understanding of ethics is one of like, try to keep your hands off as much as you can, that's an ethical position that we might be wrong about, that we are right now locking into these AI systems that may one day essentially rule the world. I can't see this as a very solvable problem just by saying, well, we'll have the chatbot sort of slow down a bit and do it in a more psychologically, you know, palatable way. It still has to make that decision as to what it believes, right? That seems unavoidable. Yeah, so a couple of

things. I strongly agree that there's, you know, inaction as a form of action. And I am worried that people will see instruction following AI as the status quo, as the thing that is not imposing your values, where an instruction following AI just does whatever you want. I want to stage a coup of my country. The AI will say, yep, I'm going to do that. I want to build a bio weapon in my garage. Yep, I'll do that. I want to cheat my partner. Yeah, the AI will help you. That I think is like, yeah, not the way forward. And instead, we want to be differentially empowering people to do good things, rather than harmful things, at least for some very broad,

very thin notion of like what makes for the better society. I don't think that AI should be going with some particular political persuasion instead of convince people. I do actually, I kind of want to defend something that's like a kind of

division of labor approach where the AI that I'm using as a personal assistant, or something like that, you know, it might be in the role of like the good friend and good advisor, but without pushing any, you know, particular model view, just maybe it's got some, like, it's just generally helping you to think through, you know, be the kind of best version of yourself. And then maybe we have these kind of institutes where we've got all of the AI model philosophers and they're really thinking about, okay, what's the, how do all the arguments weigh up to

and come up with new arguments, new thought experiments, and so on. And then on the basis of that, we can get arguments and considerations kind of out into the public sphere. That seems to me like the best of both worlds, potentially. I'm very worried we won't do that latter thing, because, well, how much investment goes into moral philosophy at the moment? I expect it will be even less as a fraction of world expenses. literature over the coming years. But at least as a kind of ideal plan, that would seem quite

good to me. You're one of the sort of founders, popularizes of the effective altruist movement, which is all about how to do the most good in the most effective way possible. And sometimes it can be a little bit unconventional or unexpected. I mean, famously the idea that you sort of should stop like working for charity and instead go and make as much money as you can and donate it effectively and making sure you're donating to effective charities and all of this kind of stuff. And in the face of this sort of AI safety development, which seems to be an extremely high priority, some people think that as an effective altruist, you should essentially take all of your efforts and put them into solving this problem because this is like top of the priority list.

others have criticized this and the same thing occurs with like what we owe the future the book that I mentioned earlier that you wrote that there are going to be so many potential people in the future that like just by weighing up the the sort of utility like we should be putting way more attention into into preparing you know everyone in the future for living good lives and there's this idea that if we were being perfect rationalists perfect utilitarians we'd say okay let's stop bothering with this like malaria stuff and like yeah starvation that's bad you know but like some people are going to starve and that sucks but there are literally like trillions of people who will exist one day similarly you could say look yeah okay starvation is

bad but we're talking about like mosquito drones people this is like so much more existentially worrisome that if we're being rational we would just take all of our efforts away from world hunger malaria you know minor conflict that will probably sort themselves out like who cares you know some people will die whatever this is what we should focus on and others hear that and go like not only is that just intuitively untrue that we should still care about those things but also that seems to lose sight of what like effective altruism was all about and so do you think that that is a rational ethical policy and if so or if not what does that mean for the future of the effective altruist movement if the

most effective way that we can care about other people is to just ignore everything and focus on the AI apocalypse. Okay, great questions. And I think there's two things to say. The first is to really distinguish between what should the world as a whole do versus what should you as an individual do. Now, there's like many very big problems in the world. There's malaria, there's tuberculosis, HIV, there's gender discrimination, there's racial discrimination, there's climate change, there's global inequality, there's authoritarianism, there's war, the list just keeps going on. But as an individual, or even as a community of thousands of people, you're not going to be able to solve all of these problems. You're not even going to be able to solve one of them.

Instead, you're going to make some, you know, small in relative terms, dent, huge. huge in absolute terms, dent, where the huge and absolute terms means, you know, saving dozens of lives over the course of your life or a benefit even larger. And so given that, there's these very different questions. You might even think that one problem is, you know, maybe you think AI is the biggest problem in the world. And you think that risk of war is, you know, not as big a problem what's like, but nonetheless,

you're much better fit for the juicing risk of war. Then you don't need to work on what you think the world's biggest problem is. Instead, you should work in where you think you can have the biggest difference.

So, that's the first thing to say is saying, like, on the margin, given how neglected something is, there should be radically more effort in there, isn't saying kind of all of the world's effort should go into that So, like, yeah, my kind of views on long-termism, I felt often got quite misunderstood where

the bar I was advocating for is really quite low. It was that rich countries in the world should allocate 1% of GDP to issues that distinctively benefit the long-term future. That would put it on the par with, you know, minimal ethical standards for global development to. That's a far cry from saying we should spend like 99.9% of the world's economy. me to benefit future generations. Now, Effective Altruism is such a small actor.

So how should it focus? And they're the main thing I want to say is that there are many different people with very different views. And I think the best and healthiest kind of effective altruism movement is one where people do just figure out, like, okay, what are there reflective ethical views? What do they think is going to happen in the world? So then where do they think they are going to have the best? biggest impact. If that means everyone figured thinks that working on making AI go well is the

best thing to do, then fine. In practice, it's much more split than that. So there's a lot of people still focused on, a lot of people who are focused on global health and development. Give Well, which is a kind of charity evaluation organization, has now moved over the billion dollars, mainly from, you know, significantly from small donors. There's others who focus on factory farming, but then, yeah, there's a lot of people who are focused on AI too. And I think that these issues that we've been talking about, both AI safety, but also concentration of power, risks from dangerous novel technologies, risks of value lock-in,

are just huge, they're really big. and it's really close to zero people who are working on the, at least on the intersection of very advanced AI, like AGI, and these other issues. And so I do think that this should be at least one big focus for the EA movement as a whole. At least I'm making the argument that it should be. And if people disagree, then, you know, good. You know, you say like with long-termism, which is this sort of, you know, caring towards the future people who will exist.

And obviously, I'd recommend people to read your book. We'll link it in the description and stuff. But, you know, you make a compelling case that given just how many people may exist in the future compared to now, that there is just such a huge level of utility in their interests and worrying about their interests. And you say, but look, you know, I'm not calling for us to focus all our energies here. I'm just calling for, you know, 1% of GDP, but like... why not like if it's true what you're saying if it is like actually that important isn't that a bit

sort of misaligned with it's as if i said to you like you know like my my house is like burning down and like everything's on fire and all of my belongings are but you know like i really i haven't updated my twitter banner in a while and like we've got a you know we can't put all of our interests into one thing so i'm i'm gonna you know maybe i'll just do that first and go and grab the hose pipe afterwards. It's like, based on what you said, it would seem like you should be focusing on the house that's on fire. Yeah. So, I mean, the analogy might be like, you know, you have all your savings, you've got your house that's at risk of burning down. What fraction of your savings should you spend to prevent your house from burning down?

Well, it depends on how much it costs to prevent it. If you can prevent it from burning down with $1,000,000 pounds, and then you don't have to do any more than that, you should spend a thousand pounds and you shouldn't spend more. And that's true, even if burning down their house would completely ruin your life and is the most important thing ever. So, like, the size of a problem doesn't need to match up with the kind of cost of the solution. And so how much could we reasonably spend to benefit future generations? Well, at the moment, there are these amazing opportunities to help the future go better by protecting ourselves against worst-case pandemics, by helping to steer how well AI goes.

If we ramped up to kind of 1% of GDP of rich countries, that's like so much more than we are investing at the moment that I really don't know, like, would additional, how good would additional funding or additional effort look? And in particular, would it look very different than just generally building a flourishing society where there's lots of, you know, there's huge number of things that overlap where what's good in the short term is good in the long term and vice versa. They're, you know, even in the case of just preventing the next pandemic or helping ensure

AI goes well, like what, in terms of my mortality, risk over the next 10, 20 years? What are the things that are most likely to kill me? Actually, I think either AI or innovations that are downstream of AI are higher than cancer. So I actually think lots of these things are beneficial in the short term too. And so perhaps once we were a world that was more sane and just more thoughtful towards future generations and was investing a significant amount of effort, then plausibly to me, the things that look good, yeah, would look best for future generations also just look best for society. It would be being wiser, being richer, being better educated, and so on.

Well, Will McCaskill, lots to chew on, links, et cetera. We've mentioned a couple of articles and things, that video on AI, your previous book, What We owe the Future. It'll all be in the description. Thanks for your time. Great. Thanks so much, Alex.