Sean Carroll's Mindscape: Science, Society, Philosophy, Culture, Arts, and Ideas - 187 | Andrew Leigh on the Politics of Looming Disasters

Episode Date: March 7, 2022

We're pretty well-calibrated when it comes to dealing with common, everyday-level setbacks. But our brains aren't naturally equipped for dealing with unlikely but world-catastrophic disasters. Yet such threats are real, both natural and human-induced. We need to collectively get better at anticipating and preparing for them, at the level of political action. Andrew Leigh is an academic and author who now serves in the Parliament of Australia. We discuss how to move the conversation about existential risks from the ivory tower to implementation in real policies. Support Mindscape on Patreon. Andrew Leigh received his Ph.D. in Public Policy from the Kennedy School of Government at Harvard University. He is a member of the Australian House of Representatives representing Fenner. He was previously a professor of economics at Australian National University, and has served as Shadow Assistant Minister for Treasury and Charities. His recent book is What's the Worst That Could Happen? Existential Risk and Extreme Politics. Web site Research web site Wikipedia Twitter

Hey everyone, it's Cal Penn. I'm inviting you to join the best-sounding book club you've ever heard with my podcast, Earsay, the Audible and I-Heart Audio Book Club. Every episode, I nerd out with amazing guests and dive into the best new audiobooks available on Audible. It's the book club for your ears. Listen to Earsay, the Audible and I-Heart Audio Club on the I-Hart Radio app or wherever you get your podcasts. If you're a QuickBooks customer looking to grow your business without the growing pains, You need the Intuit ERP. Upgrade to Intuit Enterprise Suite in a matter of hours. It's the AI Native ERP from the makers of QuickBooks. Learn more at Intuit.com slash ERP. Hello, everyone, and welcome to the Mindscape Podcast. I'm your host, Sean Carroll.

What you're hearing right now is a last-minute blurb I am appending to the beginning of the podcast to make an announcement that some of you may have seen on either Twitter or my blog the other day, namely that I am moving from Caltech to Johns Hopkins in Baltimore, Maryland. My new title will be Homewood Professor of Natural Philosophy. How cool is that? I got to say. The Homewood Professorships are special kind of positions that in principle float freely above specific departmental lines. But for practical purposes, this is a joint appointment between the physics department and the philosophy department. I will have offices in both departments. I'll take students in both departments, teach courses, the whole bit. So it's extremely exciting for me because rather

than doing philosophy-type things or even doing public-facing outreach-y type things or doing a podcast and kind of having that be in addition to my job, all that stuff is now part of my job. I am a philosopher, as well as a physicist, card-carrying right now, and all of the podcast, writing books, giving talks, kind of stuff, they want me to do that. This is the first time in my academic life, when this is actually encouraged rather than tolerated. So very, very exciting for me. We're moving over this summer. Nothing bad to say about Caltech.

It's been a great place. But this particular position with a lot of people who I already know is just exactly right for me. There's several people who we've had here on the podcast on Mindscape already at Johns Hopkins. Henry Farrell, the political scientist, Adam Reese, of course, the astronomer, Emily Real, the mathematician. So it's a very, very exciting time. You can read more if you go to the blog and I explain what it's all. about. The other thing I wanted to very quickly note is that this podcast was recorded about existential threats to humanity and the political response to them over a month ago, before we

were in the current crisis that actually makes us think about existential risks and the political response to them. So it was not precognition. It's just that some topics never go away. Okay, now we turn to our regularly scheduled podcast. You know, when I think about guests for the podcast, I have rules. I have hidden rules in my brain about what kind of person I want for the podcast and what kind I don't. I don't always make these explicit because I retain the rights to change the rules. They're my own rules. I made them up, right? One of the rules is no politicians, no working politicians or people who are currently running for office. And it's not that I have anything against politics. In fact, I'm extremely in favor of politics done right. Of course, politics, politics

can be done wrong and it can be very tiresome and people's IQs can go down when they start talking about politics. But I'm a huge believer that politics is important that we need to invest both our intellectual and organizational energies in making politics work and getting the right people into the right places and thinking carefully about what that means, who the right people are, what the right places are. But I also recognize that there's a difference between the activity of politics, and the activity of understanding the world. You know, there's an activism, intellectual divide. Both are important, but for Minescape, I'm much more interested in the intellectual side of things.

Even if we have, you know, musicians or whatever on the podcast, it's about the craft. It's about how you think about these things. It's not about people's personal histories and gossip. It's also not about individual policy proposals that people are trying to get through or individual candidates who are trying to be elected into office. So rough rule, no politicians on the podcast. And I am violating that rule today. And again, it's my rule.

I made it up, so I'm allowed to violate it. But today's guest is Andrew Lee, who is a member of parliament in Australia. And I'll explain why I violated the rules. For one thing, Andrew comes from a kind of academic-y background. He has a PhD in public policy. He was a professor of economics at Australian National University. and he has a new book out just a few months ago. And the new book is called What's the Worst That Could Happen, Existential Risk and Extreme Politics. It's a book about existential risks,

which we've talked about before in the podcast with people like Martin Rees and Nick Bostrom. But in the down-to-earth political context, what do we really do about these existential risks? The book is published by MIT Press. You know, most politicians' books are not published by MIT Press. So given the topic of the book and Andrew's background and the fact that it's a serious intellectual book, I thought it would make a good choice for the podcast. And that's what we're talking about. We're talking about existential risks. So we go through the usual list of options, right? There's climate change, bioterrorism, asteroid hitting the earth. And we try to think about both what these risks are and also what we should do about them at a fairly down-to-earth level.

What can governments do? What can people do, et cetera? And for better for worse, you know, we end up, or at least in my mind, the interesting part of the conversation was really not any policy proposal so much as a perspective proposal that, you know, people need to individually think about the world and where the world is going in a slightly different way if we are to take these kinds of existential risks seriously. I think it's a pretty easy case to be. make, that we should take them seriously. We can argue about the margins, about the details, about exactly how much resource or time or effort we should put into mitigating existential risks, but we should be able to agree that something that has a non-trivial chance of ending human life or even causing enormous disastrous consequences for most human beings, that's something we should worry about, even if it's going to happen after our individual lifetimes, even if it's going to happen 100 or 200 years down in the future. So that's what we do. We take these seriously. Andrew explains how, you know, he didn't come into office in Parliament being the existential risk guy. You know,

he has been thinking about it since arriving, and he cares now about things like super intelligent AI in a way that he didn't before. So as with anyone, you can agree or disagree about his particular diagnoses, but it's food for thought. It's making us think about these big things that are, look, I'll be honest, some of them are kind of scary. Some of them are very real. Like, one of Andrews' points is this is not a one in 100,000 chance. This is a much bigger chance than we might hope that there really is going to be something disastrous that happens in the foreseeable future. So let's think about it. That's the first step to preventing it. That's where we are for this podcast. So let's go. Julie, welcome to the Mindscape podcast. Thanks, Sean. Great to be

with you. We're talking about existential risks. So that's something a phrase I'm sure people have heard before. Why don't just so we're grounded, just so we're starting on the same page, why don't you just give us a list of some of your favorite existential risks? What are your top three to sort of worry about when you're worried about these things? Well, Sean, existential risks I think of as things which would either end humanity or fundamentally alter the trajectory of the human project. I guess those I'm most concerned about would be nuclear war, bioterrorism, and artificial intelligence gone awry. But you can think of other possibilities, unchecked climate change, even an asteroid hitting the earth, which is featured in the new Netflix film, don't look up. So there's a plethora of ways

things could go wrong. None of them probable, but the possibilities are big enough that I think it's worth us investing a bit more time and energy and making sure that don't happen? I guess one question is there is a distinction between existential risk in the sense of truly ending all of humanity or even all of life on Earth versus causing an incredible disruption that would be terrible without actually making us extinct. I mean, I don't really think of climate change as something that could literally end humanity on Earth, although it could cause tremendous disruption and poverty and hardship and so forth. Do you think that's an important distinction to make? Absolutely, because the distinction is one between future generations not

existing and future generations having a diminished quality of life or the human project taking a couple of centuries to get back on track. And I think climate change is a great example there, Sean, because it is the one where I was most hesitant initially to include it in a list of catastrophic risks. And eventually it was the work of the Harvard scholar Marty Weinstein, who passed away a number of years ago that persuaded me it was worth including. And he makes the case that the odds of a six-degree climate rise might be one in ten. The odds of a 10-degree Celsius climate rise might be one in a hundred.

And if you're talking about a 10-degree rise, you really are talking about something which takes us into the realms of catastrophe. Could you actually, I know this is not the primary topic here, but could you explain exactly how catastrophic it would be? I mean, presumably all the ice sheets melt and the water level rises, but also there's other things going on. You must have thought about this. Absolutely. You see a big rise in violence, which is strongly correlated with heat. You see huge loss of crops and potential global famines taking place.

Naturally, a range of coastal cities are immediate. wiped out by the sea level rise, but the degree of catastrophic weather events would be utterly unprecedented. There's a number of scientists who point to Venus, which millions of years back had an atmosphere not that different from the Earth's, but due to climate change is now completely uninhabitable. It's the hottest planet in the solar system, 460 degrees Celsius on the surface. And as a result of climate, climate change, Venus became utterly uninhabitable. So it's that possibility of the second Venus scenario, if you like, that caused me to put climate change in the catastrophic risk basket.

Well, let's see. There's a big difference between 10 degrees and a few hundred degrees. I'm just, I wasn't actually planning on delving into the details here, but are there atmospheric science mechanisms that would let us go totally haywire and become sort of a almost Venus-like planet? Well, I mean, so if you go to the particular Venus example, Sean, they had a runaway greenhouse effect. Evaparating water led to a steam blanket, which warmed the planet even further. Then the water vapor broke into hydrogen and oxygen and the hydrogen is literally swept away by solar winds. The process took place three and a half billion years ago and took about 10 million years to get rid of the water on Venus. Again, we're talking about very low probabilities,

but given how catastrophic it is, it seems to strengthen the case for action on sensible climate policies that avert temperature rises. No, okay, I think that's very good, very helpful. So that's like millions of years kind of thing. And this is the one place you are allowed to talk about, you know, a tiny percentage, but over millions of years. And we'll speculate about the future in the that way. And no, we won't hold you accountable for it. We're not going to say that you predict it will become Venus. Don't worry. But the worry about, you know, 10 degree climate temperature increase over 100 years, I guess there's a feeling that some people might have, that climate change is somebody else's problem, even if it's a problem, you know,

if someone is already, you know, relatively well off, if they don't live near the coast, they probably have a feeling that they'll get through it okay. And I think, I think, I think that's what you're pointing at is not going to be as easy as maybe people think. Yes, that's right. And climate change is the classic collective action problem, one in which it is very easy to take the global litterbug approach. You know, if I just throw my trash on the sidewalk, then really that's not going to make a big difference to how messy the streets are. But if everyone does it, it really does make a big difference. So just as we should pick up our trash, we should also have an interest in common pool solutions.

And, you know, you're seeing more of that movement with the Paris climate talks and Glasgow talks. But again, we're a long way off where we need to be. Right. So let me ask you, let's, with that on board, let's back up a little bit, let me ask you, has your way of thinking about issues like this changed since you've become working politician? I mean, you started out as an academic. Now you can actually have a vote in some ways that professors don't. did you learn more? Did you realize it's more important than you thought, or is it more or less

what you expected? Yeah, it's a great question because, you know, I went into politics in 2010. I was not largely concerned with catastrophic risk. Climate change was on my radar, but certainly unchecked artificial intelligence wasn't. And it was thinking in a probabilistic sense and envisaging what it could be to end the human project that really brought catastrophic risks to to my attention. Toby Ord's estimate is that there is a one in six chance that the human project ends in the next century. And if that's true and that continues to be true over the next millennium, then you can think of it as playing repeated rounds of Russian roulette for a millennium. One in six chance that you're dead in a century, but a five in six chance you're dead in a

millennium. So that then made me feel that this was an issue which was getting much too little attention. And particularly as a politician, I guess the lens I bring to the discussion around catastrophic risk is thinking about how populism makes the problem worse by causing a focus on the short term rather than the long term, firing up the temperature, destroying institutions and undermining international cooperation. So that's my particular lens on catastrophic risk. When people turn to telehealth or weight loss, they're looking for real support. That's why more people are choosing orderly meds.com. Orderly meds connects you with real doctors and access to proven GLP1 medications like semaglutide and terseptatide. No guessing, just a more supportive

experience, and all shift directly to your door in discrete packaging. Do your research, ask questions, then visit orderlymeds.com slash podcast for an exclusive offer. That's orderlymeds.com slash podcast. Individual results may vary on medical advice, eligibility required, C-Sight for details. My best skin ever at 45? Give me a theme song and a best skin care award because it feels like this, right there. That's farmhouse fresh skin, all right? I'm blowing, and everyone asks how.

The best skincare is farmhouse fresh, and the award is you, your best you. Visit farmhousefresh skincare.com and use code radio for a free starter routine with any purchase. When you first became a politician, when you started talking to your colleagues, I guess it's a parliamentary system in Australia, right? Yes. And did they just roll their eyes or do people in that position get it? You know, is there a common feeling that with different degrees of urgency, perhaps, but that we should worry about this kind of thing?

I mean, the risks of sort of ebbed and flowed when I was a kid in the 1980s. I remember having a conversation with one of my schoolmates. We must have been grade four or something. We both agreed that there was no chance that we would finish high school because the world would be destroyed by nuclear catastrophe. And that was incredibly salient at that time. In recent years, I think people have taken on the notion that pandemics can be pretty serious. And climate change has also come to the four.

But unchecked artificial intelligence, which on my estimate is the most, most worrying catastrophic risk is still regarded by many as being in the realm of science fiction. So the individual risks have been taken on sporadically by the political establishment, but less so the notion of catastrophic risk as a whole. I think that makes sense. That's what I would have guessed. That's interesting to hear. And do you feel that when you talk to them about it, they're willing to listen? or is it, I can imagine that as a working politician, they're like, how does this get me votes back home? This is not something where people's attention is really focused.

Yeah, although it's interesting, right? So you take that moment in 1998 when the Bruce Willis film Armageddon and the Steven Spielberg movie Deep Impact come out, immediately there is a response. NASA sets up its planetary defense coordination office. You have, you know, Ted Cruz famously asking in a hero. what steps do we have to be taking so we don't have to rely on sending Bruce Willis to space to save humanity and big increases in spending on planetary defence and tracking near-Earth objects. In some sense, politics has tackled that one.

Our response to asteroids isn't perfect, but we've responded in a pretty bipartisan way in most countries around the world. And yet climate change is the opposite. And here people are deeply divided and there hasn't been the same sort of unified response, at least in the United States and Australia, that we've seen to asteroid risk. I mean, you're really saying something provocative by giving credit to two movies, two Hollywood blockbusters for political action on asteroids. I mean, is that a lesson right there? A lesson for people who do care about things, that maybe we should be trying harder

to leverage popular culture to really get people worked up about this? Well, yeah, I think that's probably the exception rather than the rule, Sean, because there have been so many movies about catastrophe. Like, it's one of Hollywood's favorite films. So you think about the pandemic movies, outbreak, carriers, contagion, the bioterrorism movies, 12 monkeys, V for Vendetta, the nuclear war movies, Dr. Strangelove on the beach, artificial intelligence movies, Avengers, Age of Ultron, Terminator,

and even the climate change movies. So Waterworld, Mad Max, Fury Road, Blade Runner, 2049. So Hollywood has got us to the edge of our seats, but that hasn't always gotten people off the couch to deal with catastrophic risk. And you need to bring it together. And I think also to paint a picture, Sean, which says it is not that hard to deal with these.

issues. The analogy I often draw is about buying insurance for your home burning down. It's not a probable event, but you spend a very small share of your annual income taking out home insurance because it would be catastrophic for your household if it happened. But you also raise, that's very helpful actually. I think it's very good way of putting it's not enough to have a big scary movie. It has to be like right place, right time, et cetera. But this idea that an issue, like this, which you would think would be the paradigmatic example of bipartisan opposition. Like, you should be bipartisanly opposed to the end of humanity, I think. But in fact, in various examples, the idea of taking action on these has been leveraged for partisan purposes, one way or the other.

I mean, I worry, so since you're a working politician, tell me if I'm too worried about this, but I worry that politicians have become too good at turning any issue into a partisan issue. And the fact that one side will be for doing something automatically makes it suspect to another side, and you can get people excited about that. Yeah, I often think about that Ronald Reagan line that he once started in his dotage of saying, If only we had aliens coming to Earth, we could get the Russians and the Americans to unite around a common enemy. And maybe that's why things worked in the case of asteroid strikes, because it was the closest to what Reagan envisaged. But when it comes to nuclear disarmament, we're a long way away from that.

And it is a strong partisan divide between hooks and doves, less of a recognition that reducing the nuclear arsenal would just reduce the, a chance of a mistake beyond anything else, as well as measures like taking missiles off hair trigger alert and allowing callback systems within missiles. All of these tend to be opposed by some of the hawks in the military establishment, even although you've had Republicans like Henry Kissinger from time to time coming out and saying, look, reducing the size of the nuclear stockpile would make America safer. I don't know what to think about these things. I do have these pessimistic moments when we start talking about these things.

But let me back up and be more philosophical about it for a second. I mean, you mentioned this number one-sixth that Toby Ord, the philosopher, I think, calculates as an estimate. So first question is, you know, where does that come from? How in the world do we come up with a number like one-sixth, one-chance-in-sixth, one-chance-in-sixth within a century? Yeah, so one in six is Toby's estimate of putting together the total natural risks, asteroids, supervolcanoes, stellar explosions, and then the anthropogenic risks, which are much bigger, nuclear war, climate catastrophe, pandemics which occur so-called naturally,

pandemics that are engineered, unaligned artificial intelligence. And I would add to Toby's list, widespread. authoritarianism enabled by surveillance technologies. So you put all of those together and you get a risk that's in the ballpark of one in six, just to give you a kind of benchmark for what that means, that's about the chance that if you're 90 years old, you'll die within a year. But, okay, I mean, I think as a good Bayesian, I'm entirely on board with the idea that we should be thinking about these probabilities. But when it comes to something like the probability nuclear war, I'm just at a loss of how to actually do it. I mean, I think I would have given a

very different answer in 1970, 1980, 1990 and 2000, right? I mean, how do we get confidence in these numbers that we try to cook up? Yeah, I mean, these are heavily rule of, you know, back of the envelope numbers. What we're doing here is envisaging both the probability of the event multiplied by the chance that if it happened, it would cause catastrophe. So, you know, the research around nuclear winters is important because you want to envisage not only what's the probability that nuclear weapons are launched, but if in the event that happens, how does that then affect the planet overall? We don't have a good sense of either of those, but they're the two things we're multiplying

together. Yeah. Okay, but I mean, I guess in the case of nuclear war in particular, do you have any more details of where that number comes from? I mean, it must be someone's estimate of the chances that some politician is going to do something terrible. Yeah, the chance of politician will do something terrible or the chance that there's just a mistake being made within the system. So, you know, given that you've got tens of thousands of nuclear weapons on Hair Trigger Alert, you want to envisage problems like the one that we had in the Cuban missile crisis where you had a Soviet submarine, had depth charges being dropped near it, thinking that it was under attack, almost firing its nuclear missiles by mistake.

So the chance of a sort of an accidental misfire then leading into a cascade is what you'd envisage. But yeah, their best judgments by experts. They've got huge margins of error around them. What we're trying to do really is get a sense of rough orders of magnitude rather than really nail down the last decimal point of the probability. The point is it's bigger than you'd think. Yeah, okay, good. No, I like that answer very much.

It's of the order 10 to the minus 1, not 10 to the minus 5. And that's the important distinction, right? Precisely. And so this, but there is some, again, philosophical questions here. I mean, we're giving utility, we're being consequentialist in some way. We're weighing the value of our future lives. And this is always an interesting philosophical question. I know that economists like to discount the future a little bit because, in part,

we're not there yet, in part because we don't know. So is there some discounting that goes in? Like if you were really doing a cost-benefit analysis about, you know, how much risk we should tolerate versus future damage? So I love that you've gone to the utilitarianism point because I think it is super important. If we massively, if we discount the future at the regular sort of discount rate that you use, say 5% a year, then you get the results that future people aren't worth very much. Indeed, if you discounted a rate of 5%, then you get the result that Christopher Columbus is worth more than all the 8 billion people currently alive today. Or similarly, that your life is more valuable than 8 billion lives in 500 years' time.

Now, if you think that's absurd, then you should probably, you're having a problem with the notion of discounting human life, and I do too. I don't think future lives have any less value than current life. lives and therefore that we should put massive weight on the not millions, not billions, but trillions of lives that will occupy this planet in the billion years before the sun engulfs us and the potential that if we get things wrong, those people never exist. I mean, I want to be on your side about this, and I think I am on your side about this, but I'm never less going to push back because I don't think I could defend this with any

rigor myself because there's just so much we don't know. I mean, sure, billions of people in the future are valuable and we don't want to harm them unnecessarily or even expose them to risk if we can avoid it. On the other hand, like, how do we know that what we do now can't be fixed with something we do pretty easily 10 years from now and the billions of people in the future are fine? So even though we care about them, knowing what to do for their benefit seems very hard. Yeah, I mean, certainly there's uncertainty and there's the possibility that we come up with another solution. But in my view, that's different from discounting, Sean. It's not that those people are less valuable, but you might say they have equal value,

yet I'm choosing to take a different set of actions in anticipation of change technologies that might emerge. That's a reasonable approach in my view. But, you know, I share the view of philosopher Will McCaskill who says that there is the idea of saying that we're more valuable than future generations is almost a form of prejudice that seems to be on par with racism and sexism. He calls it presentism, the idea that we're putting our inherent moral value above those of people who'll live in the future. It is, it's presentism to say that Christopher Columbus is worth more than people.

today, it's presentism to say that we're worth more than people in hundreds of years' time. We should value their existences, which, let's hope, will be far more pleasurable than ours. They should be able to live lives of greater meaning and enjoyment, duration and health than people alive today. And so we ought to be protecting them. I certainly do want to protect them, but I also would like to develop a non-utilitarian justification for doing so. It's exactly in these cases where you're multiplying a tiny number, a tiny amount of risk by billions of people being affected that I think that utilitarianism is on shakiest ground. Are there other justifications for taking dramatic action right now? Just more, I don't know,

deontological or virtuicist sort of reasons. Like, it's just the right thing to do. Yeah, I mean, you can think about these as your descendants, if you like, if you have kids, an argument which probably is less immediately tractable to people without kids. But, you know, if you love your kids, you love your grandkids, there's no reason you shouldn't love your great-great-great-grandkids who you'll probably never meet or your great-great-great-great-great-grandkids, who you'll certainly never meet. That sort of moral obligation to one's genetic line argument could well be powerful with some. And it might just be the case that you want to envisage what it would be like at that moment at which the world is snuffed out.

And going back to that Netflix, Flick, don't look up. That moment of annihilation is one that ought to be on your minds when you're thinking, would we spend a very small share of the world's resources in significantly reducing the chance of catastrophe? Why not? Yeah, no, I think that I liked Don't Look Up. I presume that you actually saw the movie, right? Yes.

And it's gotten a lot of controversy because film people don't like it. Like as a movie, there's complaints about characterization and plot and whatever. But of course, as a scientist or politician, you're like, no, this is kind of touching a nerve in a very important way. But one of the parts I didn't find realistic is the choice of disaster. you know, an asteroid, which is coming to destroy the Earth, and we know exactly when it's going to happen. I think that for things like climate change, there's a big difference because it's gradual, right? I mean, it's coming, but it's sort of creeping up on us, and there's no threshold, there's no date at which, oh, this is when climate becomes bad. Does that kind of difference in jeopardy make it harder to make sensible policies about it?

Yeah, absolutely. I mean, don't look up is, it has a silliness about it, which, makes it great theatre but not necessarily a great depiction of reality. And the uncertainties in climate modelling are real and have been exploited by those who want to continue to make money under the current system. So likewise, artificial intelligence. We've seen attempts by those who are working in the current system to say, well, if you're in favor of dealing, of reducing artificial intelligence risks, you must be against artificial intelligence. And that's, I think, a silly position, but you certainly see it cropping up from

time to time. Tires matter. They're the only part of your vehicle that touches the road. Tread confidently with new tires from Tire Rack. Whether you're looking for expert recommendations or know exactly what you want, Tire Rack makes it easy. Fast, free shipping, free road hazard protection, convenient installation options and the best selection of Go to Tyraq.com to see their good year test results, tire ratings, and reviews, and be sure to check out all the special offers. Tirerack.com, the way tire buying should be. Okay, good. I think I got my philosophical itch scratched a little bit there. I mean, let's get down to more brass tacks here.

Let's just, if it's okay with you, walk through some of the biggest existential risks, because they're all different, right? I mean, they all demand slightly different responses. So maybe we can start with the natural ones. You know, there's a set of things like volcanoes. We're having this discussion a couple days after a big volcano, right, in the Pacific Ocean, that was big enough to be very noticeable without quite being existentially worrisome. But it reminds us. Volcanoes, asteroids, even, I don't know, if there's something that qualifies as an earthquake

or a solar flare that would be truly disastrous. So how do we plan for these things that are kind of random but really catastrophic when they happen? Well, in the case of asteroids, we've done pretty well with the planetary defense office. In the case of supervolcanoes, it's harder. We seem to be particularly poor at predicting geological events. And super volcanoes are, you know, they're planet changing. The last one was Indonesia's Toba Super Volcano, 74,000 years ago. And that is an event which could cause complete global crop failures, massive livestock deaths, huge disease spread and so on.

So, I'm sorry, the last one's the New Zealand one, volcano, 26,000 years ago. So, anyway, tens of thousands of years since we had a super volcano, but we're poor at producing. predicting them. So the odds there are one comes along in the next century are low, but it would be great if we could better predict them. Right. And do you care, do you worry about solar flares very much? I once had a very scary conversation with a lawyer who was on some committee to study this, and he said, yeah, once every thousand years there will be a solar flare big enough to knock out our entire electrical grid on Earth, and millions of people will die. And we are doing nothing to prevent something like that. Well, he can't prevent it, but to harden the grid to be able to

sustain something like that. Yeah, I mean, I've got them in the overall category of unknown and non-anthropogenic risks. So, you know, there's certainly a range of things that could happen. The fact that we have been around on humanity on the earth for a couple of hundred thousand years without being wiped out by one of these things suggests that the odds in the next century are relatively low. But again, the better we can do at forecasting what's coming to us from out of the atmosphere, the better as safer will be.

Well, but I think I don't want to add another worry to your list of existential risks, but I think this is the real worry with the solar flares is that its real target is the electrical grid. And so we never had an electrical grid before 100 years ago, right? So they could happen all the time on the scale of centuries, and there could be a 1% chance per year, which could be completely disastrous and we just wouldn't know.

Yes, although it's hard to see how it wipes out humanity, if that's the case. So given that our ancestors managed to get through without anything that we can see in the fossil record is looking catastrophic, it seems to fall in the category of bad but not catastrophic. I think that's fair, although I personally, my lifestyle would be impacted. if I didn't have electricity for a month or two. So I don't want it to happen. We certainly wouldn't be having this conversation. No, I know.

I wouldn't have a podcast. That'd be terrible. But I guess it's a paradigm, both the asteroids and the volcanoes and the solar flares. These are all paradigms for this issue of let's calculate a rate of risk per year, right? Rather than climate change, which we see happening, it's just a matter of how much it happens. But where there's a rate of something happening pretty quickly, you know, how do you, as a policymaker decide how much money to spend? Is there literally a set of rational utilitarians somewhere that is saying, okay, we need to spend this much money per year to prevent this risk happening

with a certain percentage chance? The economist, Jimmy, would love it if that was the case, Sean, but of course, as you well know, this is not the way in which the system works. The risk mitigation is better for the natural events, but we need to calibrate the probability a little bit more precisely and try and get better alignment of spending to risk. One of the values, I think, of the conversation around the value of a statistical life, which was controversial when it was first proposed, is that it did cause safety spending to be better targeted at those things where the additional money saved the most lives.

And perhaps in the same way, this discussion about catastrophic risk probability, might help tilt funding towards dealing with the most likely dangers. No, I like that. So this is sort of effective altruism for effective charity for the world, for humanity as a whole. Like just by thinking about it and talking about it, maybe we will allocate our resources a little bit more rationally. Is that the hope?

Yeah, that's right. I mean, I, like you, are a big fan of givewell.org. And one of the points that givewell make is that the difference between effective and ineffective charities isn't just two or three times, but potentially a hundred or a thousand times efficacy. So likewise, when we're looking at these catastrophic risks, we've got risks such as an asteroid impact, which is over the next hundred years, probably a one in a million probability. Then you've got an engineer, the chance that a bioterrorism event knocks out the world's population, that's a one in 30.

So these are very different probabilities, and yet we're probably not putting enough resources into making laboratories safer and making sure that terrorists don't put their hands on material that could be used to engineer the next pandemic. Well, okay, good. Let's move on to the topic then of pandemics and bioterrorism. I mean, there's both natural pandemics and man-made pandemics, I guess,

and presumably a different suite of responses or mitigation strategies are necessary for those. Yeah, absolutely. And so tracking zoonotic diseases has received considerable attention as COVID swept the world. And so we need to do better in terms of those in natural pandemics, as well as taking steps such as working out how to reduce the risk of spread at so-called wet markets. But the bigger threat in my view is terrorists getting their hands on what the Nixon administration once called a poor man's atomic bomb. And for that, we might think about strengthening the Biological Weapons Convention, which currently has a monitoring budget smaller than the budget

of the typical McDonald's restaurant, and making sure that there is better controls over so-called gain of function research in which researchers at respectable institutions look at how they're able to make bad bugs worse. There's an argument for doing that research, but the notion that we should just publish it, publish it and allow everyone to have access to these sorts of findings is, in my view, pretty dangerous. Well, what is the current status of exactly that? I mean, can biologists just publish whatever results they get along these lines, or are they somehow restricted in a sort of classified knowledge kind of way? Yeah, so, I mean, let's take one of the recent examples. There was a team of researchers at the University of Alberta and Canada who showed it

it was possible to make horsepox, a cousin of smallpox by ordering parts of DNA on the internet and reassembling them. They showed they could do it for about $100,000 in about six months. they submitted those findings to science and the journal science said, no, we don't think that publishing this, that the scientific merit outweighs so-called dual-use research of concern, which was the science editor's way of saying bad people getting the hold of the findings. But then the researchers simply sent the paper to another journal which then published it. So that's an issue that the scientific,

community is wrestling with at the moment and having overall standards that cover not only journals but researchers themselves is going to be pretty important. And I'd like it if there was as much attention being put into this as the considerable amount of attention that's being put in countries like the US and Australia to researchers who are collaborating with people in China. Good. But there's a hugely important fact lurking in the background of what you just said, which is that it sounds pretty easy to do bioterrorism, right? I mean, even if maybe not me in my garage, but a halfway decent science lab might be able to just cook it up,

no matter what restrictions I put on you if I'm trying to be in secret. Like, is that a worry? Just sort of a random individual actor with a little bit of resource. Let's give a million dollars to do some biology. Can they really make a bug that would hurt a lot of people? Look, potentially. Genetic engineering is moving very fast. It's possible to print DNA and to, for example, upload a sequence and have the DNA that shipped to you for very low costs and desktop printers are increasingly going to become available. So there has been an interesting proposal. MIT's Kevin Esvelt has made a proposal that any of these DNA printing outfits should have built into the machinery,

a system that checks essential segments of risky sequences, which then makes it difficult for people to print bugs such as the 1918 influenza strain. Yeah. So now my imagination is running wild here. So I don't know why this, I haven't thought about this before. But, you know, we're in the middle of the pandemic now, right, with COVID. And in many ways, it's like a warning pandemic, even though it's been extremely terrible and deadly. It could have been enormously worse.

You know, it has a relatively long incubation period, but it's not that fatal. And if you wanted to design a bug that would, a bug in the casual sense of virus, that would be very, very deadly. could you design it to lurk inside people without any effects for months and then turn on and become extremely fatal? That's the worry. And once you're at the level of designing DNA, then why not? Yeah, that's the risk. So you've got this spectrum of researchers talk about of deadliness and contagiousness. And typically viruses tend to be one or the other. very deadly. So you think about untreated HIV, extremely contagious, you think about measles or malaria, but there's relatively few diseases, thankfully, which are both extremely contagious and extremely deadly.

So the risk that you have a pathogen that ticks both those boxes is what people are worried about with natural pandemics, but all the more so with bioengineered pandemics. Yeah, I mean, a naturally occurring pandemic, the virus has its vested interests are not to instantly kill everyone it infects, right? Because it wants to be passed on. Yes, exactly. But there's no such restraint on artificial ones. This is what I've just now realized in talking to you, that you can design, you could imagine designing a virus that is way more deadly than anything that we would imagine naturally occurring. Yes, yes, and the rapid advances in genetic technology have huge potential in terms of disease alleviation and saving lives around the planet. We just need to make sure, Sean, that as we put those innovations in place, we're not inadvertently assisting those who would look to make the pandemic equivalent of a dirty bomb. I guess the tiny mitigating factor, is that you can't point this biohiblin of a dirty bomb very precisely, right?

Like everyone, you know, you would be in danger yourself of being adversely affected if you just set loose a terrible new disease on the world. Yes, that's right. And, you know, past attempts haven't proven successful. So, you know, you think about the arm-shrinco sect spreading anthrax in Tokyo Stubways. they did initially try and find some disease that they could unleash, but they weren't able to do it. So they used, sorry, they used sarin gas in the end. And that was because their attempts to engineer a disease hadn't worked out.

Similarly, there was a set of poisonings in Oregon that took place in 1984, Yeah, it's actually the worst biological terror attack in American history, which involved the poisoning of salad bars. That group had previously looked at spreading HIV AIDS, but hadn't worked out how to weaponise it. So these terrorist groups have tried and failed in the past. The challenge is to ensure that they keep on failing in the future. Maybe this gets into things that you're not allowed to talk about,

but I always presume that if there's some technology that could be used for better or for ill, it will be developed and it will be used by somebody. So I presume that governments in secret are developing bio-weapons, even if they plan to never use that. Is government development of these kinds of things just as big a worry as terrorist group development? Look, I don't think so in advanced countries. at least. And the decision that the Nixon administration made in putting in place the Biological Weapons Convention was that these are fundamentally weapons that are more useful to less powerful

states. And that it's strongly in America's interest to not be involved in working on bugs as a form of weaponry. So the US had a range of programs which it then shut down under the next administration following the Biological Weapons Convention. We've had some evidence that the shutdown by the Soviets wasn't as complete, and certainly there's been other incidents such as Saddam Hussein's use of chemical weapons on his own people. But largely, I think it is the case that in advanced countries, there's not secret research going on into biological weapons, because of the recognition that these are, as Nixon said, a poor man's atomic bomb.

Yeah, okay. That does make some sense. I'm not quite sure because I never know what countries are doing in secret, but I think that the motivations that you mentioned do make a lot of sense. For the terrorist groups or just for the mad scientist or whatever, the fact that we can buy our own genetic kits, DNA, writers and so forth, you mentioned that it would be sensible to at least imagine restrictions on either the technology or the publication of the technology. How specific are proposals along those lines to not let

people just buy a DNA engineering kit or not publish the results if they figure out how to re-engineer smallpox? Yeah, I think Kevin Asfeld's proposal is pretty specific and makes a lot of sense and that ought to apply not only to firms that are shipping DNA, but also to bench top DNA synthesis machines, that then becomes ensuring that people can't do it at home. The concern around research publication, I think, has crystallized into a number of quite sensible proposals, runs counter to the ethos that you and I are so familiar with

in universities, that publish your parish idea that when you've got new findings, you share them with the world. And so I can understand the discomfort that people have about keeping research silent. But in this case, I think it's strongly in the interests of humanity. Okay, but are there like bills in front of parliament or so forth, or how advanced is this effort to think this through? No, it's still in the realm of sort of concrete policy proposal. but I certainly hope that it'll it'll come into, it'll crystallize into clear codes of practice

and legislation in the coming years. When people turn to telehealth or weight loss, they're looking for real support. That's why more people are choosing orderly meds.com. Orderly meds connects you with real doctors and access to proven GLP1 medications like semi-glutide and terseptatide. No guessing, just a more supportive experience. And all shift directly to your door in discrete packaging. Do your research.

Ask questions. Then visit orderlymeds.com slash podcast for an exclusive offer. That's orderlymeds.com slash podcast. Individual results may vary not medical advice, eligibility required, C-Sight for details. My best skin ever at 45? Give me a theme song and a best skin care award because it feels like this, right?

That's farmhouse fresh skin, all right? I'm blowing, and everyone asks how. The best skin care is farmhouse. House Fresh, and the award is you, your best you. Visit Farmhousefreshskincare.com and use code radio for a free starter routine with any purchase. Okay. And then, you know, just to wrap up the pandemics, there's still the naturally occurring pandemics that we have to worry about. You know, I'm a little depressed, as I think a lot of people are, at how badly we as a species have responded to this particular pandemic.

Do you think that will be better next time? Are there obvious right things to do? Or do you think that going forward pandemics are going to be political footballs, as we say here in the United States? Yeah, I certainly hope that we're going to get better in terms of making sure that pandemics don't leak out of labs. You know, there's a theory this one leaked out of a lab. I think that's probably wrong. But it's certainly the case that the last person to die of smallpox caught it from a lab leak rather than from naturally occurring smallpox. So we do need to be careful around lab leaks and making sure that BSL4 facilities have better safeguards around them would make a lot of sense.

We could also do more in terms of having detection facilities. One of the ways in which we picked up on COVID early was this program for monitoring emerging diseases, so-called Promed, in which doctors were just posting findings that they found, and ProMed was collating them together. Their, ProMed's global budget is $1 million a year, which is about what it costs to build a suburban playground. The idea that that's the best we can do as a planet seemed nuts to me. I'd be increasing Promed's budget substantially

because I think they're one of our best early detection weapons against pandemics, whether they're natural or anything. And what about things like, we did a pretty good job of developing a vaccine. We did a much less good job of making it widely available, especially worldwide, and we did a terrible job at convincing people to take it, I would argue. Do you see Pat's forward to doing better next time? Yeah, I mean, as a politician, I've been thinking a lot about how you get ahead of disinformation. And certainly a lot of the stuff I've read suggests that in some ways you want to inoculate,

people against the hoaxes before they come. But once a hoaxes taken root in people's mind, it's quite hard to dislodge it. So among Indigenous Australian communities, for example, the hoaxes got there in many cases before the government with the idea that the vaccine was being given to those communities first because it was being tested on them. And once that idea had taken root in communities, it's become quite hard to vaccinate indigenous communities. So as the overall Australian vaccination rate is good, the rate in indigenous communities is bad. And that's just a microcosm of the overall challenge of dealing with

disinformation. We've got to get out there before the bad actors do and warn people of the character of the disinformation messages that are to come. Look out for these stories. This is what people will tell you and this is why you shouldn't. And is that something we, I mean, this is an obvious answer to this question. Presumably we should be doing that before the pandemic hits, right? I mean, there should be an ongoing kind of program. I don't even know what it would look like, though, I mean, to tell people who are naturally skeptical, someday there's going to be a pandemic and we're going to want to vaccinate you, please don't reject it. Yeah, right. I mean, part of it is scientific literacy. Part of it also is making sure that we're rigorously

testing the anti-disinformation messages. So there's a couple of good papers. I've been reading recently, randomized trials, just testing different strategies, because it is one of those areas, Sean, where your gut doesn't tell you very much. No. Where, in fact, we now know that repeating the hoax can cause it to sink deeper into people's minds and that sometimes it's better not to mention the lie at all. Yeah, so this is research in psychology? What would you even call it?

What is the research we have to do to understand this better? Yeah, right. It's sort of a blend. The researchers that I've seen are a combination of public health and social psychology. And they come up with useful findings. The notion of the fact sandwich comes out of that. So if you have to mention the lie, then start with the truth, mention the lie, mention the truth again. And so at least you're giving a double dose of truth for every time you mention the lie. That fact sandwich idea has come out of clever randomized trials on messaging. It's one part of dealing with disinformation, but it's going to be increasingly important if populism continues to maintain the hold it's got in many advanced countries. Well, yeah, I mean, I think we will get there, but maybe it's okay to take things out of order. You know, it's not just disinformation, right? There's a motivation, a political motivation on some sides to take the, let's say, the anti-vaccination stance or something equivalent. And that seems like a tougher nut to crack in terms of prevention. Yeah, that's right. If you want to build a powerful support base, then finding an issue such as conspiracy theory can be really useful.

You go back to the way in which the Nazis used conspiracy theories about Jewish people to fuel their rise. Conspiracy theories about African-Americans have been used through by various American populists. You see in India the use of conspiracy theories about Muslims to fuel the sort of Hindutva movement, which has taken over the ruling BJP. That sort of weaponising of conspiracy theories to target a small, to spread fear about a small group in the population, is a tried and true tactic for populists around the world. Yes. And how do we stop?

that to ask you a completely unfair question. Yeah, I mean, I think we, first of all, need to call it out, and calling out racism turns out to be a pretty effective way of combating the political weaponization of racism, recognizing that the rise of populism does in part have its roots in economic problems. The loss of good middle class jobs, the hollowing out of manufacturing has been a source that the populist anger has tapped into. And recognizing, too, that you can't fight fire with fire, and that if you're looking at an alternative political approach, then that needs to be the kind of calm, stoic approach,

which characterises leaders, such as Nelson Mandela, rather than a sort of angry approach, which doesn't seem to be effective against populism. Well, this is, I think, and this is a big picture message of your book, I think. Correct me if I'm wrong or tweak it as you will. But the real lesson of thinking carefully about these existential risks is not just, oh, against risk number one, we do this, against risk number two, we do that, but rather we really kind of have to think hard about our whole. whole approach to politics and even life to really adapt to the fact that we face existential

risks now in a way that maybe we didn't previously in human history. Yeah, that's right. And strengthening democracy is an important way of reducing the hold that populism has gained on our politics. When I look at the U.S. Constitution, I regret that Jefferson's ambition of an update every generation has essentially been dropped for the last two generations, which means that you have a democracy that's not as democratic as it should be. So I've talked about a number of democratic reforms that I think would make sense in the United States context, you know, holding elections on weekends or holidays, reforming the electoral college, encouraging active citizenship, the kind of detailed community engagement,

rather than simply engaging on social media as a substitute for real political action. But we also need to realize that good politics involves acting with calmness and wisdom rather than trying to beat populists at their own game. No, that's a great message, but it does, what you just said shifts my attention a little bit because you know a lot more about American politics than I do about Australian politics. Before I forget, can you just like say a little bit about what it's like to kind of be in this Anglo-American English-speaking tradition but be in a different country in Australia? Do you have to like kind of keep up with political and social movements in both where you are and in the U.S. just because the US is so influential? Oh, look, I'm married to an American.

I spent four years of my life in the U.S. I'm as interested in American politics as the politics of any other country except my own. So it's a pleasure rather than a duty to follow American politics. But I do see so much in the American democratic experience that suggests that the beacon of democracy that America was two centuries ago is now shining less brightly. And I look at the changes that

be involved as essentially part of that responsibility that democracies have to keep on improving their systems, not to sit back and say, this is ideal, we'll never do anything more. I suppose the most radical proposal I would have is to treat voting as a civic duty, just as Americans are required to fill in the census, just as Americans are required to serve on juries. I think having compulsory voting would significantly improve turnout and make turnout more representative of the population as a whole. And am I right that you have that in Australia or something like it? Yeah, we do. And, you know, we don't have 100% turnout, but we have much higher turnout than most advanced countries do, largely because, not because people are fearful of being fined, but simply because the fine, which is, you know, less than one hour's average wage is something which spurs a civic duty and makes people feel it's election day, I will go and vote rather than, you know, will I, or won't I go to the polls today. But this goes back to this idea that we talked about very briefly before, that politicians have become very good at weaponizing potentially controversial issues.

So even if one political party in the US got behind that, there's no chance the other one would. So I'm very skeptical that things like that are going to actually happen. Right, right. And so this requires leaders who are willing to make decisions which are in the interests of the polity as a whole rather than simply in the interest of their narrow political party. You know, we've got past examples of that, leaderships of both parties that have chosen to make decisions in the national interest. But like you, I worry that we're seeing less and less of it.

And the increasingly tight hold that Donald Trump now exerts over the Republican Party is a real concern, and as is the fact that almost three quarters of Republican voters, believe that Donald Trump won the 2020 election. I think while you're in that environment, it's quite hard to get the requisite changes that ensure that America's democratic ideals are realized. Well, I agree. And I also perceive that the diminution of a devotion to democracy does occur on both sides of the political spectrum. Even if it's one side's fault, I'm not trying to sort of both sides, say everyone is at fault, et cetera, both sides here in the United States

have lost any motivation for working along with the other side. And I don't care whose fault it is. That's a bad situation for a democracy to be in. Yeah, yeah, I agree. And, you know, one of the most influential books to me in recent years has been Etan Hirsch's book, Politics Is for Power, which makes the case that increasingly Americans are treating politics more like they treat their local sporting teams, cheer and jeer from the sidelines, but don't think that you can actually affect the game, rather than going out on the field and trying to make a difference. And Aetan talks about the importance of getting involved in your local community and trying to think of yourself very much as a player in the political

spectrum. And that also means that at a local level, because party labels are much less salient, you're much more likely to be working with people rather than just shouting at them. Interesting. I'll have to check that out. That's a good recommendation. But okay, I mean, this is probably the most important thing we're talking about, but I don't want to forget that we're going down a list of possible existential risks, because they're all different. I mean, we've mentioned climate change a lot. But what is your take on the current amount of progress we're making. I mean, we keep trying to have international agreements and it's hard to make them happen and then people violate them. It seems to be kind of a recipe for cynicism a little bit.

Yeah, I'm certainly concerned that what came out of Glasgow is inadequate for what the planet needs. You know, you look at climate action tracker, which looks at the number of nations that have implemented climate policies consistent with a two degree of warming target. It finds only a handful have done so. Even the European Union, it says, are only coming close. Many countries are insufficient or critically insufficient, including yours and mine. So, yeah, we do need to do an awful lot more. And what's striking about it is that a lot of it involves installing energy sources, which are zero marginal cost. So ultimately, a lot of this will pay for itself in cheaper energy. Yeah, it seems like there is good news on the technology front, right? I mean, solar and other

renewables have gotten cheaper a lot faster than people thought they would. Absolutely. And so the gains from installing there are substantial. And what's important too is then to think about this not just at a national level but also at a community level. So if you've got a coal-fired power station which is slated for closure, then you can take advantage of the electricity connections coming into that plant in order to use that as a site for a wind farm or a solar farm and have the construction jobs that go along with that at the same time, as well as some maintenance jobs in the future. No, I like that. That's a very clever idea.

And then, of course, we have the final risk I want to dwell on is the AI risk. You've mentioned that already. This is the one that you said you hadn't really been conscious of when you came into your job, but you've read up on it and now you're kind of worried. Yeah, absolutely. And again, we've got two questions. Will it happen and how bad will it be if it happens? There's differences among artificial intelligence researchers at the point of time at which artificial intelligence will exceed human ability. The median guess in one recent survey was 20161, but almost no one working in the area says it's impossible. computers will never outperform humans in the sorts of tasks we envisage, whether that's, you know,

writing a best-selling book or driving a truck or solving mathematical problems. So then once computers go past us, what happens? Well, presumably they accelerate past us at a pretty rapid rate. So, you know, you look at the performance of chess computers. If you put a chess computer up against a human, the chess computer now wins 99% of the time. And in Go, the probability the machine wins is 99.995%. So, you know, this is essentially saying that now computers playing these games are as likely to beat us as the world heavyweight boxing champion would be to beat me in a boxing match.

So once they accelerate past us, what do we know about their values? Well, we hope that they share our values, but I don't think that's locked in by any means. And the possibility that super intelligent machines have a set of values that are either antithetical to human existence or more likely just orthogonal to it is a real one. And we need to be very careful how we develop these computers, that they don't somehow damage our prospects as a species. quite substantially. I mean, I guess I have some skepticism about the rate of progress of AI in truly human

general intelligence kind of task. I think it's very different than Go or chess. But for the worries that you have, it doesn't have to be, right? I mean, it doesn't have to be human-like intelligence. As long as we are ceding power in some way to these algorithms, we could get in trouble. But so my question is, what exactly would be the scenario that we're worried? about. I mean, so imagine that AI becomes very, very smart. What is it going to do that will harm us? How specific could we be about that? Short answer is we don't know, but you know, you can take the

Nick Bostrom example of a supercomputer which decides that it wants to build as many paperclips as possible. It doesn't want to hurt us, but our buildings in our cars turn out to be good raw materials for paperclip building, and so it massively destroys humanity's prospects. the result. Or you can imagine that we try and encode our values, but we do so the wrong way. So we say to the computer, we want you to maximize human happiness and it puts our brains in vats, feeding us drugs to maximize our pleasure centers. We say that we want it to find a cure for cancer, so it increases the underlying rate of cancer, it increases the underlying rate of cancer so that it can improve detection.

These sorts of problems called perverse instantiation or a King Midas problem do trouble artificial intelligence researchers and suggest that we want to think about encoding computers which have three qualities that they're observant, humble and altruistic. So we're not locking in a particular moral code, but we're asking these supercomputers to watch us, us to act in our interests and to recognise that there might be a lot of complexities about human society and they want to be learning from that in order to help us.

That all sounds great in principle. The trouble is if you've got an AI race, particularly one that's being conducted through the lens of global competition between superpowers, then you might end up with the first super intelligence not sharing our values. I guess, yeah, I have a lot of questions about this. I know this is not exactly your expertise, but, you know, so let me say the dumbest, most naive question I have. Probably the answers to this one are pretty easy.

But if I have a AI algorithm running on my computer that just goes bad, breaks bad, and starts doing bad things, I can unplug my computer. So I think that the scenarios that we're envisioning here are imagining not just an AI, gone bad, but some kind of embodied AI that is almost human-like in its capacities and so forth. And what I worry is that we're being too anthropocentric. We're imagining AIs that are kind of human-like, but the real danger will come from AIs that have very different capacities that we haven't really thought about enough. Yeah. So the reason we can't just switch it off at the wall, I think, is because if it's smarter than us, then it'll want us self-improve to acquire

resources and resist being shut down, which means it'll do everything it can to try and avert a situation in which you just turn it off. And there's a lovely analogy for this, Sean. There's an artificial intelligence agent which was designed to maximize its score in Tetris, that game with a bricks drop down. Love it. Now, as you know, Tetris can't be won because ultimately the last brick comes into place. And so this game had a strategy which involved getting to the last moment and then pausing the game. And many people have noticed that the behavior of that artificial intelligence agent is not that different from what you would envisage from a superintelligence

which was resisting being turned off. So we've seen it already in the lab to some extent. And defying shutdowns is going to be one of the things that a super intelligence puts a lot of resources into achieving. Okay, no, I like that example very, very much. But I do think that it actually highlights one of the distinctions here, because I can imagine an AI that is way more intelligent than I am at almost everything, much better at not only chess and go, but symphonies and fiction writing,

but has no self-preservation instincts at all, right? I mean, it seems like the angle here is that we should worry about giving AI self-preservation instincts. Yeah, I'm not sure that I can envisage an AI which has any substantial desire, but doesn't care about being turned off. I would have thought pretty much any desire you begin with ought to then effectively encode a desire not to be turned off. You know, ultimately this comes down to whether Asimov's third law is necessary or unnecessary, you know, is three, is threat is laws don't injure humans obey orders and protect yourself some people say protect yourself doesn't need to be in there because the other the other two effectively encoded right yeah but yeah okay

I mean I do work again this is just vague worries I haven't thought about this in any systematic way but I worry that all of our experiences from biological organisms that grew up through evolution rather than being designed. So for us, intelligence and self-preservation instincts just go hand-in-hand. It's very natural, but they needn't in the AI context. That's my vague worry, but I don't want to dwell on it. What I do want to dwell on is just to just to, you know, wrap things up with some final thoughts. More thoughts on the global strategy or the human scale strategy here. I mean, one thing that I was interested that you poo-pooed as a strategy is the Elon Musk idea of backing up the biosphere of, you know, if we spread human beings out to other planets, we're less likely

to blow up ours and therefore end humanity. And you didn't seem that much in favor of that one, or at least you weren't that impressed with that suggestion. Right. I mean, largely it's because I'm a cost-benefit guy. And when I look at the cost of that strategy, it seems to be massive relative to the costs of strategies such as better coordinating AI races between existing teams. There's also the massive loss that would come from the destruction of planet Earth and all that we've built here. So, you know, I think we could do better in getting clear global guidelines on ethical AI. Interestingly, there was a proposal in 2018 from Justin Trudeau and Emmanuel Macron for an international panel on artificial intelligence. And they had the international panel on climate change as a model.

but the Trump administration didn't support it, in part because they thought it would impede the development of artificial intelligence, to which I'd respond, well, it's only impeding the development of bad artificial intelligence. Let's get the guardrails in place before we build the highway. If you build the superintelligence first and then try and think about its ethical rules, you could find that you've left it too late. Okay, that does make sense. I like the cost-benefit angle there. But speaking of which, the final thing is this kind of vague utopian but nevertheless attractive

idea you have of thinking differently about these kinds of questions. I mean, your book ends talking about wisdom and stoicism, which is not how most politicians' book ends. Actually, maybe they do. know. I don't read many books by politicians these days. But how do we make people wiser and more stoic? That sounds like a big global project that I wouldn't even know how to start doing that. Yeah, I'm not sure that I have a good strategy for populizing stoicism. I suspect Ryan Holliday is probably better on that than me. But it is the philosophical approach that in my view is the right strategy to respond to populism. That idea that we need the values of courage, prudence,

justice and moderation, that we should be rewarding people who are being bold in service of truth, that we should be celebrating a love of wisdom, that we should be encouraging a fairness in the treatment of people, and also that living a calm. and a disciplined life rather than a shouty or a chaotic existence is to be celebrated. And, you know, it's not as though we haven't seen examples of this rising to the top. You look at Marcus Aurelius and the life of Epicetus, not to mention the success of those such as Nelson Mandela in his not only in Robin Island, but in leading his own country, Gandhi as well. It's a tradition with quite a rich lineage, and one that I believe is the right strategy in an age in which we want to ensure that the population not only survives,

but that a whole millions of future generations come after us. Okay, so, but how do we do it? I mean, I'm totally in favor of this. What I worry is that you list some virtues and everyone goes, yes, those are great virtues. But then you operationalize them and people go, oh, no, I don't mean that. I didn't mean that we were going to let immigrants in or whatever it is, right? I mean, how do you make that connection between values that we're all willing to get behind and acting them out in the way that we're hoping people do?

Well, for me, as a politician, it's about resisting the urge. to go for the jugular and to make personal attacks that aren't necessary, as well as recognizing that when the temperature is turned up, it's generally not going to advantage those who care about the long term. So all of these things can be pretty tempting, and there's certainly plenty of those who say that left-wing populism is the answer to right-wing populism. So there's a sense in which this is a kind of,

of personal project for any politician who wants to make a difference to reducing catastrophic risk, whether they're on the left or the right. And you can identify plenty of those who've adhered to stoic values on the right over the years. But it's also something where you'd envisage celebrating a different kind of media engagement, for example. The shouting heads media engagement should be looked on more as a spectacle of amusement rather than being a serious way of engaging in politics. Yeah, I mean, maybe at the end of the day, the best thing we can do is be exemplars of these virtues that we want other people to have, I guess.

Precisely. All right, that's something to aim for. I like it. I like leaving the podcast with a goal or something for people to think about and try to get better at. So Andrew Lee, thanks so much for being on the Mindscape podcast. It's been a treat, Sean. Thank you. What if you could have even more and more and more help to pursue your goals?

At LPL Financial, we offer more ways for advisors and their clients to thrive. So what if you could? Paid advertisement. Investing involves risk, including potential loss of principal, LPL Financial LLC member FINRA, SIPC.