The Great Simplification with Nate Hagens - Existential Risks: The Biggest Threats to Life as We Know It with Luke Kemp

Episode Date: December 4, 2024

(Conversation recorded on October 22nd, 2024)     The human system as we know it today – which powers our economies, global supply chains, and social contracts – is a fragile network based on innumerable complex components. Yet we rarely stop to recognize its many vulnerabilities, instead taking for granted that it will continue to securely operate indefinitely. But if we take a more careful look, how can we assess the risks of major catastrophic events that could destroy life as we know it?    Today, Nate is joined by Luke Kemp, a researcher whose work is focused on existential risks (or X-risks), which encompass threats of human extinction, societal collapse, and dystopian futures. How can we begin to understand the likelihood and gravity of these ruinous events, and what kinds of responses from people and governments could further undermine social cohesion and resilience? What roles do human biases, hierarchical power structures, and the development of technologies, like artificial intelligence and geoengineering, play in X-risks? How can we collaborate across industries to protect our modern systems through effective risk management strategies? And in what ways do our institutions need to become more inclusive to better democratize decision-making processes, leading to safer futures for humanity?   About Luke Kemp:  Luke is a Research Affiliate at the Centre for the Study of Existential Risk (CSER) and Darwin College at the University of Cambridge. His research focuses on understanding the history and future of extreme global risk. Luke has advised the WHO and multiple international institutions, and his work has been covered by media outlets such as the BBC, New York Times, and the New Yorker. He holds both a Doctorate in International Relations and a Bachelor of Interdisciplinary Studies with first class honours from the Australian National University (ANU). His first book on the deep history and future of societal collapse (titled Goliath's Curse) will be published with Penguin in June 2025.   Show Notes and More Watch this video episode on YouTube   ---   Support The Institute for the Study of Energy and Our Future Join our Substack newsletter Join our Discord channel and connect with other listeners  

It is tragic that essentially no US election has been decided by the candidate's policy on nuclear weapons. The most devastating thing that could ever be done is the US launching a nuclear strike. And yet that can be decided by a single person, the president. And yet that never factors into elections whatsoever. You're listening to the great simplification. I'm Nate Hagan's. On this show, we describe how energy, the economy, the environment and human behavior all fit together and what it might mean for our future.

By sharing insights from global thinkers, we hope to inform and inspire more humans to play emergent roles in the coming great simplification. Joining me today is existential risk and societal collapse researcher Luke Kemp. Boy, I bet that's a great way to introduce yourself at parties. Luke is a research affiliate at the Center for the Study of Existential Risk at University of Cambridge. His research focuses on understanding the history and the future of extreme global risks. His first book called Goliath's Curse is on the deep history and future of societal collapse and will be available in June of 2025. But today, Luke joins me to unpack first before his book

the primary existential risks that humans face globally, such as nuclear war, environmental catastrophe and an engineered pandemic. We also discuss how we're able to study such questions and what the potential paths forward are for reducing either the probability or the severity of such events. If you enjoy this podcast and you're interested in more content that covers the critical issues humanity faces, please subscribe or follow us on your favorite platform. It's one of the biggest ways you can support this organization and help us continue our work. please welcome for a very interesting conversation, Luke Kemp.

Luke Kemp, welcome to the program. Thank you for having me, Nate. I've seen your work. I've read your papers. I've seen your name in the category of existential risks and collapse of civilizations for quite a few years now. We're going to talk about existential risks today, but you're a young fellow, quite a bit younger than me. How did you get interested in all this stuff at such a deep level at such a young age? I started with climate change. I was researching both climate science and policy in my undergraduate.

And then it was lucky enough to be selected for a student delegation, which went to Copenhagen for the Copenhagen Climate Summit back in 2009. And it was there I saw how politically distant we were from actually having any kind of workable solution. A combination of that in reading Mark Linus's six degrees. kind of gave me this crisis moment of realizing both we're so far away from solving this, and additionally it could get so bad. And then after being focused on climate change, I started teaching climate change science policy. And the students were always dreadfully depressed by the end of the course, as I should expect.

And we always tried to get them at least one of two uplifting lectures, usually on both social solutions, movements, protest movements, and technical solutions. In this case, geoengineering. And that's when I started reading into all the ways that geoengineering could go wrong. And that was kind of my gateway drug into thinking about all the ways in which the world could be destroyed apart from climate change. And I quickly realized that it's a Pokemon problem. You've got to catch them all. If you achieve that Herculane task of decarbonizing the economy of 2050, but you have a nuclear war in 2015, you're still pretty screwed.

So broadly, these are termed X risks or existential risks. You consider climate change one of those and maybe you could define X risks and list out some other ones. So the term existential risk usually refers to a basket of three different things. Human extinction, permanent collapse of global society, and last but not least, some kind of lock-in event of a societal dystopia. And usually people are referring to something like a mass surveillance global totalitarian dictatorian dictatorship. leadership that expands the globe. Now, I think about risk a little bit differently to how some my colleagues will. So they'll often refer to existential risks. And what they're really referring to there are threats or hazards. A risk is used composed of four determinants, the threat, the vulnerability,

the response, and the exposure. And this is adopted by the IPCC when its definition of risk as well. But to think about it this way, if you have a tsunami about to hit you, the tsunami itself and its magnitude is the threat. Whether or not you have the infrastructure to cope of it is the vulnerability. The fact that you, for instance, are actually in the place that's exposed to it if it is a quality exposure. And last one or least, your response obviously matters in terms of how many people are going to die, be injured. A good example, this is the 2004 Bali tsunami, where essentially both people already had the infrastructure. It was highly vulnerable to a tsunami. But importantly, when the shoreline started to recede, which is what happens prior to tsunami, people actually

went out to inspect. They were interested in what was happening. And of course, that left them both more exposed, but the response as well was bad. So all these things matter. And I think the question of, is climate change an existential risk, is kind of misleading in a way? Because all the other conditions, the world in which climate change occurs really matter. If you have three degrees occur in a world which is, my marked by pretty good global multilateral governance. You have high trust in public institutions. You have pretty good adaptive technology, and we're handling most of the planetary boundaries well. Potentially we can navigate that. If you have three degrees occurring in a world which is driven

by conflict, dangerous runaway technology, massive distrust in public institutions and each other, suddenly three degrees seems far, far more ominous. So when I think about existential risks, I think about more existential risk. What's the level of risk from a particular socio-technical system that is likely to result in human extinction, permanent global societal collapse, or kind of like societal dystopian lock-in? What are some of the other big categories other than climate change of X risks? You're the first person I've had on to discuss this, believe it or not, three years on. I'm surprised. Okay. Well, I feel honored in that case. So, existential threats, people usually put them into things like climate change, artificial general

intelligence or superintelligence, nuclear weapons, an engineer pandemic, and then there's usually a big cluster which people put into what's called natural risk sometimes, so things like asteroids, comets, super volcanoes, or just large-scale volcanic corruptions, etc. it's notable here that there's no real clear way of defarting or like selecting which ones which people just kind of settle upon certain threats they think could do a huge amount of damage and even the natural ones again aren't really natural because it depends upon how vulnerable human society is and what about things like uh bi risk bioterrorism things like that is that in this category this umbrella category yeah yeah so that would come into an engineered pandemics

usually. Okay. And are there people and organizations dedicated to this around the world looking at X risk, like existential risk, is that a, in the lexicon and everyone knows what that is and there's teams of people working on it? Yeah, absolutely. So I'm an affiliate here with the Center of Study of Excentral Risk at the University of Cambridge, also classically known as Caesar. And surprisingly, no, I don't have any colleagues called Augustus. I did have a colleague called Julie. surprisingly enough. There's a couple of others out there. So they used to be the FHAI, the Future of Humanity Institute at the University of Oxford. That got closed down earlier this year. There's ones which are focused more upon advocacy like the Future of Life Institute,

which I think mainly operates out of Boston and the US. There's also the Stanford Excessual Risk Initiative, CERI, led by Paul Edwards and Steve Ruby at Stanford University. and there's now a recent one read by Daniel Holtz at Chicago University called the X-risk Lab. So there's a few ones out there and there's a few popping up

at least one that's gone down as well. But there's certainly a kind of sub-filled, if you will. There's still not an entirely agreement on even how you define an existential risk, for instance. As you've noticed, I defined it a little bit differently to how some others will. And the field

at least initially had a real kind of technetopian bent to it So a fun little factoid here is that the original definition of existential risk, which was coined by the philosopher Nick Bostrom from University of Oxford in 2002, was initially that it was any risk that prevents you from reaching, or prevents humanity, from reaching what's called technomaturity, technological maturity. And he defined technological maturity as the maximum level of economic productivity and control of nature. So in short, if you don't have any kind of galactic spanning civilization that's harvesting suns, anything that prevents you from reaching that point is an existential risk, which isn't probably what most people are referring to when they think about eccentricer risk. But that was kind of the legacy and the genesis of the term was a lot of people who came more from

transhumanism and technootopianism thinking about big things like AI and how it could stop us reaching their kind of utopian future. But I think it's become much more diverse and widespread since then. There is quite a bit of a schism between the animist and mechanistic worldview and what you just described as the technological maturity of humans versus what others might describe, like last week's guest on the podcast, Bill Plotkin, as ecological maturity. I mean, the conclusions and the filters and the challenges are quite,

a bit different, though I suppose climate change and some of these other things apply to both categories. Absolutely. I think a lot of this comes down to a matter of values as well. You know, when I spoke about existential risks as being permanent global collapse, human extinction, or some kind of societal dystopia, those are all pretty big values judgments. You know, most people converge on human extinction being a bad thing. I think most people, but probably a few less, would converge on global collapse being a bad

thing. Socile dystopia is a complete value statement, basically. And for Bostrom and many of the other originators of the field, they really had a big value around utilitarians in particular. That you can kind of boil down value just to more or less human experience or the very least conscious experience. You can put a number on that. And if you increase that number, it's a good thing. And that's obviously very different to people who place value on things like ecosystems or even in more abstract concepts like justice or democracy. Well, sticking with that theme, how is the rise of techno-optimism, especially recently with AI and all the power and wealth amassing under the tech industries affected the likelihood

of some of these X-risks? I think we'd say it's the primary driver, if I think. When most people talk about existential risk, everyone seems to more or less agree that the biggest ones here tend to be a whole bunch of runaway technologies, whereas the combustion engine, nuclear weapons, nuclear warheads, or of course, potentially AI systems. I actually prefer to take the lead of my colleagues, Zoe Kramer, and talk about automated cognition rather than AI. Important distinction there being that it's not necessarily intelligent and it's definitely not artificial, but what we're doing is automated different parts of human cognition. So when I use AI

from now and I'll be thinking about that in terms of automated cognition rather than artificial intelligence per se. And artificial intelligence, of course, like big data was just a term that silicon values to kind of pipe itself to a large extent. But yeah, I mean, technology is key behind why people think that we're living in a particularly dangerous error. And not only is a key to the actual development of these hazards, but it's also, I think, crucial as to why we're failing in addressing them as well. As one example, think about AI. Two years ago, we had a letter released saying that we should think about AI as being a human extinction threat, just like nuclear weapons or engineered pandemics. And this was signed by a range of AI luminaries, ranging from

people like Dariy Amodi, Sam Altman. And those two of course leaders are some of the biggest companies, Demas Hasibus as well. So those are the leaders of deep mind and froptic at Open AI, the biggest developers of AI right now. The funny thing is that despite seeing this as an extinction of a little threat, they've actually hampered regulation to address it. So, oh, this we had the vetoing by Gavin Newsom, the Governor of California of Bill SB 1047, which is going to be one of the first

bills to actually actively address the risk, the catastrophic risk may I. It's kind of strange when the one hand tech companies are saying, this is really dangerous, this could literally kill everyone. Every single person you know

could die due to the technology we're developing. On the other hand, here's a fairly small level piece of regulation which's going to help reduce that risk and they will be against it. So this is pretty mammatic of, I think, the pattern which happens, which is the developers of technology obviously increase the risk by developing this technology in the first place,

which once again, nuclear weapons or combustion engine, and at the same time, they actually actively lobby to produce regulation and prevent us from mitigating them. Does it seem to you that on all of these longer-term risk to society, like climate change and the change in the ocean, or resource depletion or biodiversity crisis, that there's like kind of a free rider dynamic where people who care about and sacrifice their time

and resources to work on these longer-term threats, get out-competed by people who ignore or deny or minimize those threats because it's not a popular narrative. I mean, we see in the, this podcast is recorded on a, October 22nd. We have an election here in two weeks. And all the, um, you know, the surveys are the economy is number one and in the voters' minds and climate change is way down the list. Um, that's like a human thing. We care about the present moment. So are the existential risks and the researchers like yourself working on them, are you, are we just immediately at a disadvantage to

people who, uh, poo-poo or dismiss these risks. To a certain extent, yes. You know, if I chose to say, for instance, work with the military, I could automatically earn a lot more money and have a little more job security. Um, if I was working in the fossil fuel industry, once again, a lot more job security and a lot more money than if I'm working on eccentric risks, even at the very prestigious university. So in general, yeah, you're probably going to take a hit doing this. And you see this even of climate change, right? If you choose to do the right thing and actually become a protester,

there's a good chance you'll go to jail and at the very least you're not going to get paid very well for it. On the other hand, again, if you're working for a fossil fuel company, whether that be as a lawyer or even a miner, you can get very handsomely rewarded. I have some friends in high school who work in some of the Australian mines and they get paid. Essentially, I get paid in three years in the space of half a year, more or less. So unfortunately, morality doesn't always pay. And, yeah, I think there is this a simple matter here of being out-competed by those who aren't so risk-adverse. So let me ask a related question of all these organizations you mentioned where you're working at Cambridge. You mentioned one in Boston and Stanford.

Do you find a commonality and the personality and temperament of people that are working on existential risk? Because I know people looking at limits to growth and overshoot and oil depletion. My friend Tom Murphy, I do the math podcast, did this survey and found they were all, I think, one, I mean, predominantly a one type of Myers-Briggs personality type. I think it was I-N-T-J. Is there any commonality either in ethos or temperament or personality leanings that you've found? To begin with the field was definitely dominated by people who identify as transhumanist. What is transhumanism? Yeah, transhumanism is essentially the idea that in order to have a good future, we need to actively upgrade the human body through both cybernetics as well with biological means, whether that's by engineering ourselves, no longer have disease and live forever, or potentially even cooking ourselves up to the cloud. So it started with the technological maturity threshold test that anything that wouldn't allow us to colonize the outer space was considered an ex-risk. Precisely. It was, yeah, kind of a combination of transhumanism, utilitarianism, and what people

would call long-termism as well. I think the field is much more diverse now, and I actually have trouble thinking of one kind of encompassing ideology or characteristic across my colleagues. They're actually all really quite different, both in political ideology, values, disciplinary background, etc. The only potential commonality I can see is a colleague of my mine once quiped that everyone here in the field of existential risk has an uncharacteristically high base rate of happiness, just because working on these things, you kind of need to be naturally quite happy. Otherwise, it's going to weigh you down pretty quickly if you're just naturally more

of a pessimistic and probably less happy person. That's pretty interesting. You mean it self-selects for that, because if you started as an unhappy person, you would quit the work in 18 months or such. Yeah, you'd have to quit the work early or you potentially might not want to do in the first place. Yeah, yeah. There's some self-reflection that I will do off camera on that question. So a major part of my work, Luke, emphasizes how human biases affect the way that we perceive

and act in the world. For example, I've had Chuck Watson on the podcast to talk about risk homeostasis, where we repeatedly at risk happens, but nothing bad happens, and therefore we psychologically adjust our behaviors in the direction towards taking more risk. Another would be recency bias where things that we have just recently been seeing or occurring become front in our mind and make it more likely that we're seeing those sorts of things. Do you see any of these or other biases affecting the ability of researchers in the field to accurately study existential risks. There has been some literature on this, including a 2006 piece by L.E.R. Z. U.S. Kelski,

who highlights 10 different biases that could potentially exacerbate existential risk. And also a piece by Jonathan Vivina at Duke University, who talks about the tragedy of the uncommon, that the situations that we deal with existential risk is so uncommon, that they activate certain biases that make them more difficult to deal with. And the usual ones that come up are things like, I think you've already mentioned one, availability heuristic or the availability bias, that essentially more recent easily recalled events or risks are more likely to be prioritized in your mind, while ones which haven't happened before or happened a very long time ago are less likely to be prioritized. And what we're dealing with here with things like nuclear war, for instance, either completely unprecedented and sometimes even just outside the scope of comprehension to a certain extent. So naturally they're much more difficult to prioritize. Despite biases being both relevant and important,

I believe that at worst, there can be a distraction and at best that's probably an exaggerated factor here. When we look at, say, for instance, people's response to things like genetically modified crops or nuclear power, people have usually suggested that if anything, they're being too risk-adverse. They're taking tail risk and taking potentially large,

speculative risks far too seriously. And even when you actually look at the survey data of how people react to and want action on particularly big risks, it's pretty clear people actually want to have usually pretty drastic mitigation actions. There's strong majorities across every surveyed countries in favor of stronger climate action and often in favor of pretty strong carbon prices as well. There's one survey of 63 countries on lethal autonomous weapons, so usually drones that can kind of select their own targets and kill them about human and division.

intervention. And I think it was what, 61% all were in favour of having a ban on lethal autonomous weapons. A survey of six countries, including Israel and the US, found strong majorities for complete disarmament within a forceful international treaty. And Europeans in particular very strongly pro-nuclear abolition. So it doesn't seem like, to me, biases of really playing a strong factor here. And if anything, we kind of mistreat the public to a certain extent where on the one hand, they are kind of like biased and irrational in taking some risks too seriously, like with nuclear power, for instance. On the other hand, we think they're just way too biased to actually take any big risks seriously, like with nuclear weapons. It's kind

of like the old, you've obviously familiar with Schrodinger's cat. Now, once they're a great meme of Schroding as immigrant, they're simultaneously going to sue your job and also too lazy to work. And I feel like we take a similar approach when it comes to the public to a certain extent. And when I actually think about things like climate change, it's pretty clear that biases haven't been the key factor as to all we're not acting. It's the fossil fuel industry. Let me ask you this. Are your audiences whose job it is to mitigate X risk, presumably government agencies and potentially military and think tanks within the government and the broad. public to some degree. But the people that are leading those efforts, aren't they generally,

and I'm naive here, I'm just asking the question, kind of left brain siloed reductionist mindsets that help create these X risks in the first place? Generally, yes. Isn't that a strange paradox? Yeah, and I think it explains a lot in terms of the incentives. I think for most of my colleagues, it's not so much they're necessarily left-brained or anything like that. It's, I think, largely matter for incentives that both getting a grant is far hard to do if you're pitching a really complex topic or if you're, say, for instance, espousing eco-centered values. It's also just much hard to do in terms of methodology.

And you're also less likely to be able to communicate to those audiences, particularly, once again, high-level policy makers, people working in the AI industry, or even people working in the hospital industry. So all the incentives usually point towards you taking a very particular kind of approach. And what about the cause and effect? I mean, how did these risks come about? I mean, from an ecological perspective, I could say that ecological overshoot is originally the cause of all these risks coupled with a somewhat schizophrenic species who is out of context from our evolutionary ancestral environment. But more broadly, you could give maybe a better scientific answer of the cause and effect of X-risk.

So I think this is the biggest blind spot in the field of existential risk. I'd like the fact you talk about energy blindness, and I think in some ways we're cause blindness. Or we have cause blindness, I should say. So as mentioned, we have these threats, AI, climate change, nuclear weapons, engineer pandemics, etc. I think if you want to think about causes, you can take a step back and say, who are the actors creating these risks? and they tend to be pretty clearly concentrated. If you're looking at nuclear weapons, 89% of stockpiles held by two countries, the US and Russia.

If you're looking at climate change, what is it, roughly 10 countries, if you're counting the EU as one country, responsible for over 80% of global emissions. So this is just the power law of energy use, yes? Roughly when it comes to climate change, yes. But for instance, when it comes to nuclear weapons, there's no clear reason as to why two countries should have most of the stockpile. And yet also, same thing happens with AI, for instance.

You know, if you're looking at, if you have a big fear about building super intelligence on Silicon, then there's only really three big companies you should be worried about right now. Anthropic, open AI, deep mind. Across the realm of risk, it's all pretty highly concentrated amongst a number of actors who are usually characterized by large levels of secrecy and enormous levels of economic power. So that's the level of actors. We can think about threats, actors. You can then say, okay, we have these actors.

We have the military industrial complex, big tech, the fossil fuel industry. But why are they acting in that way? What's the structural incentives driving them to destroy the planet in pursuit of profit? This gets to the economic superorganism, does it not? I think eventually, yeah. I think you go from basically talking about actors to talking about driving mechanisms, which is where we start to think about things like Tragedy of the Commons, arms races, races to the bottoms, etc.

And I think then underneath all of that is probably what you're gesticulating towards with superanorganism, which I think is the root causes. What's the actual base human psychological factors and environmental conditions that over the long course of history have led us to this point? And frankly, I haven't heard many good answers on that, and there's very few people exploring it. I mean, within the actual research centers I've worked in, I'm pretty much the only person who's thinking about that particular approach.

Well, I mean, there's there's a behavioral suite of who we are as an evolved hominid, but then if you take that and combine it with energy surplus and combine it with hierarchy, and you play that game in a lot of iterations, you end up building agricultural surplus and then fossil surplus and then monetary digital claims on reality and now AI. And it's this funnel to the top. on obtaining power and the power is shared by increasingly few who have immense power. And so we are downstream. We as individual organelles of the superorganism, you just said that a lot of people in these

countries, including Israel and the United States, voted against having this drone technology and voted for climate action. But do our votes and our opinions and our. ethos and our morality even count relative to the power structure that is pursuing the things that you've discussed? Open question. The answer is no, not really. Yeah, I think I'd roughly agree of your characterization here. So I'm working in a book currently called Goliath's Curse, which is essentially about the history and future of societal collapse. But what I'm also trying to do in that book is to do exactly this, to trace back to the root causes of existential risk. And I

located in a couple of psychological factors, which I won't get too in depth here, but essentially I think it comes down to a combination of things like status competition, I think is an absolutely critical one. The Dark Triad, so Machiavellianism, narcissism, psychopathy. The authoritarian impulse, I call it. So when people feel threatened, they're much more likely to, as the trope goes, reach for a strong man leader. And we actually have reams of good psychological evidence for this now. And power corrupts. Again, it's actually surprisingly strong body of evidence that power makes people more corrupted every time. And I think once you combine those certain ecological and environmental circumstances,

which talk about the book, particularly what I call looted, essentially resources that can be easily stolen and hoarded. So wheat, rice, grain is pretty much exactly that. You start to get these big dominance hierarchies, clusters of power in which a few rule over the many with the use of violence. And I think that's where you start to get all these things. like Traged at the Commons, race to the bottom, arms races, what I would call Goliath traps, and that eventually leads to where we are.

Let me ask you a really speculative question. I don't expect you to have an answer, but I would like you to speculate. So I have read, and maybe you found this research for Goliath's curse, that there was either a great flood or there was a comet or something 5,000 years ago that hit the ocean, and there was a response. And way back in the day, we lived, many human settlements were on the coasts because we got our sustenance from fishing and seafood and such. But if there was such a calamitous event, I'm not talking about the biblical flood, but there's some evidence of some calamities back in the day. if there was such a calamity, not in one area, but all over the place, would it have engendered

some of the natural evolutionary selection for strong man and dark triad type personalities where until that point, before that X risk happened, there was more equality and sharing. But in a crisis, a strongman, comes in. So first of all, what's your comment on that? And second of all, if we have one of these risks manifest in coming decades that doesn't result in human extinction, but that is pretty bad, does it actually favor those sort of humans in a selective way? Totally speculative question, but this is your field. So I'm asking your opinion. I doubt it was either a comment or a flood.

When we think about what people would call the basins of civilization or the original states, they pop up in five different areas of the world. You have the Zia dynasty in China. You have Uruk and Mesopotamia. You have the first dynasty in Egypt. You also have, usually it's Tijuanauku referred to in the Andes, and you additionally have Monta-Aban in Meso America. There's no clear particular event which unites all those.

They happen on very different timescales. Montauban happens, I think, 3,000 years after you get Uruk in Mesopotamia. So that doesn't suggest there was one kind of coordinated comet strike or flood or anything like that. What is true is that we have what's called the Younger Drys period. So we have what's called the Holocene, obviously. The Holocene is when you get this onset of warm conditions after last glacial age. And the Younger Drys is basically, we start to exit and get a warmer conditions. and suddenly we get a coal snap again, and then it starts to warm again.

What seems to have happened, the kind of leading theory here, is that you get a big melting of an ice sheet in basically North America, and that spills a huge amount of water, obviously, into the oceans, and that causes, as you'd expect, large-scale sea-level rise, and that's probably what unites these myths of a flood across a whole bunch of different cultures, particularly in the Near East, is you do actually have this big pulse of sea-level rise in particular areas of the world. And people don't realize this. And I keep forgetting it myself, we're over 400 feet higher in sea level today than we were like 20,000 years ago. I mean, it's precisely. Yeah, it's a lot.

Yeah, precisely. I do not think that would be the key instigator as to why you start to get dominance, hierarchy, states, etc. That's, you know, roughly 11,000 years ago. You don't actually get the first signs of inequality really emerging until a couple of thousand years after that, you don't start to get the first signs of states until roughly 3,000 BC, 5,000 years ago, and I've worked 6,000 years after the Holocene. So it's probably agricultural surplus. Yeah, I think I push a little back on my cultural surplus, just a little bit. I think agricultural surplus is key, but as one example, Papua New Guinea had agriculture the same

time as Egypt did. But they had tarot, yams and bananas. Still technically corrupt. You could still actually get a surplus. They did have a surplus. And at one stage, actually in the 16th century, and it would have been the 18th century, they got also sweet potatoes. And sweet potatoes, even more productive. That had a huge boom population, agricultural productivity. They had an even bigger surplus. Never developed a state. It's not just having a surplus. It's the type of resource you have. You need to have a resource that can be stolen and hoarded quite easily. So it was one example. there's groups that go along both the northwest coast of the US, basically along Canada all the way down to California.

And additionally, when the first European explorers arrived into the US, they found hunter-gatherers in modern-day Florida, he'd basically built mini kingdoms. They had a king, they had a full-time military, they did human sacrifice, they had kind of built-up settlements, etc. And that was largely based upon fish. If you can ever build fish courts, or you have large, huge bits of, sorry, runs of salmon that come for a river and you can catch

them on mass and smoke them, you start to get a resource that can be hoarded. You start to get a surplus, more importantly, that can be hoarded. And that seems to be the ingredient for dominance hierarchy rather than a surplus per se. That makes a ton of sense. Thank you. That crystallizes some things in my mind. I don't want to get too far off the topic because I know this is in your upcoming book, but it's the, it's, it's the lootability, well, because it's, it's stored power in a way. It smooths out your optimal, optimal forging dynamic if you have a lootable or storable surplus. It's not the surplus per se. It's, it's how dense and loutable it becomes. Precisely. And I think once you get looted resources and you start to get bigger populations as well,

and you also have things like monopolizable weapons. So usually weapons that a few control much more effectively, that's when you start to get the emergence of these Goliaths. And if you look at, say, for instance, China or Mesopotamia, you don't get the first states until you get bronze-held weapons, bronze swords in particular. And then suddenly states start to emerge. Who would have predicted that catching and smoking salmon would have been the speed bump

on route to technological maturity into outer space. The world works in strange ways. I look forward to your book, and hopefully you're referencing this podcast in the reference section. I don't worry. I definitely get the super organism gets a mention in there as well. So let's get back to X risks. There's a lot of them. You've mentioned some of the larger categories.

how do they interact with each other and does, if you combine them all together, doesn't it multiply to be an X-risk singularity of sorts? Or how do you think about all that? The answer is we don't know very well how they combine together. There's been very little research on this, shocking a little, I'd say. Even when it comes to pretty obvious ones, like the interaction between nuclear weapons and climate change, just next to no research on that. We, of course, have knowledge that there's a shared of a rewardy resource between two nuclear powers, Pakistan and India.

No one's really tried to systematically explore how climate change could result in nuclear conflicts, let alone put a number on it. But they have done the reverse. They have looked at if there is a strategic exchange what the levels are on reducing global temperature. I mean, we would have a destroyed biosphere in other ways, but we have a cooler plan. planet then. Yeah, precisely. Nuclear winter, as I call it. So basically, when you destroy cities, they start to put up a huge amount of soot into the atmosphere, and the soot kind of acts like a sponge and starts to absorb incoming solar radiation. And that, of course, drops a global temperatures. And we have a pretty good idea of by how much as well as how much that's going to

affect crop yields. But funnily enough, even with nuclear winter modeling, they don't actually take in account climate change very well insofar as they're not actually after looking at warming scenarios. So there's a pretty big kind of juncture or split between the people who do modeling for nuclear winter and the people who do modeling for climate change. And they don't talk to it very much. Really? See, that's another thing I'm curious about is there's so many different X risks. And I have found in my work that there's so many experts in the world who are very bright and pro-future oriented, but they don't talk to people in other disciplines. Is that the same in the X-risk community? There are people working on AI. There are people working on climate change.

There are people working on bio-risk and pandemics. Or is it more integrated? People do tend to specialize, but I think we're a bit more integrated than you see in most of academic departments. So at Caesar, for instance, I have colleagues working on AI. We have ones working on synthetic biology. We have ones who focus mainly on philosophy. It's a pretty big mixture. But by having them all in one department, people do tend to talk. We have collaborative projects, etc. Awesome. That's impressive. And there's a few people who I think do try to take this much larger view. You know, for me, my main background was originally climate change,

but I really try to actually work across areas as much as I can now. Lots of people in AI think that AI will actually be a net positive for drawing down carbon. I'm just wondering what your view on the interrelationship of those two X-risk are. Yeah, so I think AI actually. is an accelerant. When we think about all the different things we're worried about, say, for instance, climate change, nuclear weapons, there's already pretty clear ways in which people want to apply AI

to essentially accelerate and expand the risk. Fossil fuel companies are already employing different AI-focused companies that will use machine learning to better locate and extract different deposits of oil, gas, and coal. Likewise, if you're looking at nuclear weapons, there's already been a few different scholars in the US proposed in the idea of having a system which essentially has more or less an automated response to any incoming strike. So there's already a system called the dead hand system in Russia, where essentially

if the headquarters stops broadcasting a signal, a missiles fired over the top of Russia. And that missile sends out a whole bunch of radio signals that basically activate all the other intercontinental ballistic missiles, all the other nuclear warheads in Russia, which then get fired off. The idea is, if you take out Russian headquarters and they don't have time to respond, so it basically happens in a blink before they even have time to launch it in response, you still make sure you hit you basically respond as much as possible. So presumably the United States and NATO are aware of that and Russia is aware that the United States and NATO are aware of that.

So like that sounds terrifying to me. Like what happens if they have a power outage or some other thing that's unrelated to them being taken out by a missile? Yeah, I mean, I presume that centers would have almost certainly their own individual backup energy supply. But of course, if you have something like a solar flare or a Carrington event, an electric magnetic storm. See, that's what I mean. Like, how do these X risks overlap with each other? I mean, presumably, as climate continues to warm, it's going to make all these other risks more acute, I would think.

Agreed. For me personally, one thing I want to do in my future research is essentially create a kind of doomsday model or map where you start to try to map out the interconnections between these different things and think through what are the key leverage points to actually reduce most of them at the same time, as well as how do you build resilience to particular pathways towards catastrophe? But yeah, I think the key thing of AI is that it's essentially going to accelerate all these things, whether it's automating the responsive nuclear weapons, accelerating the extraction of fossil fuels, or just making the entire global economy much more tightly coupled

much quicker. All of that just increases overall levels of risk. So you mentioned this earlier. I assume you are such a personality that is generally happy. But the more that you dive into this, the more you're like, holy crap. Have you had many of those moments? Like you were aware of climate change, but now you're unpacking the onion and seeing all the different layers. Yeah, absolutely. I think, as mentioned, I had a moment of crisis when I first started to realize both how bad climate change could get and also how far we were away from political result managing this. I think I had another one when I first actually encountered the very notion of extantial risk and thinking, oh, wow, there's things like nuclear weapons. And then I think the latest one I've had was in the process of writing this book, just realizing how I think this is all. part of a much longer historical trajectory. This is at least a 5,000-year-old process.

You're going to have to essentially reverse in order to get out the conundrum of extant risk. But it seems to me that you and I are alive during the time that this 5,000-year trajectory is going to, all the cards are going to be on the table. We could be wrong, and every generation feels that, but it sure feels that way to me. Yeah. So, I mean, people obviously have numerous terms of this. Global Polic Crisis. My friends of the Cascade Institute obviously used that one quite frequently.

there's the idea of the meta-crisis, existential risk, I guess, is one version as well. I like to think of the term endgame then. In an endgame of chess, it's usually when you're making the last few moves before either that comes to a draw or checkmate. And I suspect we're entering an endgame here as well,

where the next few centuries, probably the next two or three, we're going to reach a point where either the system self-terminates or you have radical reformed in some way. I think the next few decades, if not the next few years, because, I mean, something that's probably not well understood in the X-risk community is the extreme monetary overshoot with respect to our underlying ecology and energy stores.

I mean, you could argue that we still, we've always had 10 years of oil resource reserves available, but we have 10 times the amount of monetary claims versus those reserves that we did 20 or 30 years. years ago. And I think the debt overhang in the United States and in the world triggers all kinds of others of these scenarios, which is why I refer to as a great simplification, which doesn't mean human extinction, not by a long shot, but it does mean an end to our current cultural and institutional arrangements. So is debt in the financial system included in some of these scenarios that you work on or only indirectly? As mentioned right now, no one's really connecting these things together in general. I mean, even when you think about planetary boundaries,

there's not a lot of work that actually connects the different planetary boundaries. There's some nascent work, particularly some of my colleagues at the Possum Institute, who are working on tipping cascades. So if you have a tipping event in one part of client system, say, for instance, the AMOC being reversed, or slowing, I should say. So the AMOC slowing, how does it potentially tip other things? And one of the key ones that's already been discussed is the idea of the subpoil, or Jaya, basically reversing that impact in the AMOC and then potentially AMOC impacts other things.

But in short, we don't even know how this works very well in the climate system, let alone across other plate of rate boundaries, let alone across extension risks in general. So the answer is essentially no. I would love to consider that more. I think it's important an underlying factor. So we're 40-some minutes in, and now I have tons of questions, Luke. So this should be like the most important work on our planet right now. There needs to be more people, more collaboration between people and groups, more funding for this. But I sometimes wonder if, here's a couple questions. One is, in the same way that our voting and our incentives and our, I mean, our own ethos and our own morality as organelles of the superorganism don't do much to change the power

structure is all the best information in the world on X risk going to change the power structure and the metabolism of those billionaires and trillionaires to be that are trying to get AI and more conquest and dark triad and all that. Is it just more information that won't be used or is it potentially going to be used? And here's a part two question. in the, you know, I think you wrote a paper looking at the potential of catastrophic risk on climate change, intervening with solar geoengineering with stratospheric aerosol injection. So the response to some of these existential risk in turn might create another existential risk.

That's a big question. So I'll pause here for your speculation. We could have podcasts and each of those two questions individually. Take your time. On the first, information by itself won't do anything. But information can be power, of course. And for both myself and for most of my colleagues, we're not really content with this just being simple naval gazing.

We don't want to make sure our research has some kind of impact on the world. I think the two biggest impacts I can think of that can be emergent from doing this research well. is first of identifying leverage points. As mentioned, if this is risk emerging from a single system, you want to know where the best bang for your buck is. Where's points in the system where you can have a policy or an intervention? They're going to reduce multiple risks at the same time,

and are basically going to be the most effective way to use your resources. And in particular, if we are going up against adversaries and industries who are far more powerful, you need to know what the leverage points are. You don't really have the comfort to rely upon wasting resources. Secondly, is, I think, essentially providing, if you will, ammunition or the very least resources for people who are already working on things like this. Advocacy groups, for instance.

I see a lot of this is simply being scaling up of existing approaches and the expansion as well. Like, you know, we already have, say, for instance, the divestment movement in climate change. And that's been remarkably successful. some regards. Obviously, still has a long way to go. But we don't have divestment when it comes to a whole bunch of things, like with AI, for instance. We do have the don't bank on the bomb campaign, which is related to nuclear weapons. But I'd love to see a movement which really pushes for divesting from all sources of eccentric risk. You know, make sure sovereign wealth funds and pension schemes don't invest in any kind of industry or actor that is contributing to existential risk in a

significant way. But climate change is contributing to existential risk and fossil fuels contribute to climate change, but fossil fuels also power our civilization and expectations. Yeah, I mean, most of these things are what people refer to as dual use. But that being said, you know, people want energy, they don't want fossil fuels necessarily. And even if we want to have, you know, really high energy return investment, that comes to fossil fuels, even then, the fossil fuel industry clearly has done the wrong things repeatedly here. They had evidence that climate change was going to occur back in the 1980s. They intentionally buried that evidence.

And this just runs the spectrum of how risk producers work. Likewise, when it comes to what are called novel entities, so things like micropastics, PFS, etc. 3M and DuPont, the two major conglomerate chemical manufacturers of novel entities, they had a whole bunch of very good evidence back in the 1970s of how PFS can potentially be cancer-inducing. and carcinogenic. And once again, they buried the evidence. But not to defend any of those organizations, but is it the fault of the people in the organizations? Is it the fault of the

organizations themselves? Or is it the fault of the structure and the hierarchy and the incentives that were created before? And I mean, to me, that's the real leverage point. But it's so locked in that there's no way to change it until we experience one of these X risks. I mean, Or, I mean, it's too bad that we couldn't like when there's a lot of snow in the Swiss Alps, they have those avalanche guns that they shoot off and they create little mini avalanches instead of the real big one. It's too bad we couldn't have some metaphorical equivalent of that on some of these X risks. Ideally, yes, unfortunately we do not. Right, exactly. Who do you blame?

I mean, ideally all three. I'm a big fan of hate the game and the player. In this case, we do still need to hold people accountable. It also just sets into horrible precedent if fossil fuel executives and generals know they can essentially do whatever they want. And it's okay because ultimately it's not really their fault. They're working under structural incentives. No, you know, people need to be held responsible, particularly when they're doing things that can plausibly and with foresight destroy the world. But yes, ultimately, we do also want to work on the structural incentives.

We want to work on the root causes as well. So I would think about this in terms of if you have this chain of causation for extantial risk from threats, producers, drivers, root causes, you probably want to have interventions at every point. And some of them are going to overlap as well. So to me, one of the key ones for root causes would be if this is really bad dominance hierarchies and the way they select for certain types of individuals and certain types of behaviours, then you need to start democratising more institutions. And that becomes something you can do both with the producers.

It becomes something you can do to actually address the underlying drivers. And there's a whole bunch of different interventions you can do which can support that as well. So I don't think it's as simple as just picking one area. I think it's you need to have a suite of different interventions. What were your four categories, the threat, and then what were the others? Threat produces, drivers, root causes. Yeah. And so the threat of climate change is going to be experienced.

differently all around the world. There are some countries, some regions that this is truly existential in coming 50 years. In other places, like the United States and Canada, it won't be great, but it won't be terrible. So when you think about it, I mean, it truly is existential risk to some places and it is just a risk to others. So how do you navigate that? Yeah, this is where obviously the element of exposure comes up when you think about risk. And as you mentioned, for Kiribati or Tuvalu, this is going to be existential within this entry for them. How do you think about it?

I mean, you can think about it in terms of justice. Obviously, it's deeply and jarringly unfair that the communities who have contributed least to the problem are the ones who are going to suffer the consequences most quickly. You can also think about it in terms of, as mentioned, causation. The ones who are exposed right now by really just bad luck are going to be the ones who also just have the least ability to address at this stage. As you mentioned, the US is going to be one of the ones that's most buffered from the consequences, but they're also the ones who can probably change the trajectory of the carbon pulse right now the most. I'm not putting you on the spot because there's no real answer to that question. So get back to the double whammy question that I threw at you.

My worry slash prediction is that eventually humans, government leaders around the world will have no choice but to recognize the severity and urgency of global heating. And we will do things. We will have to do things. And one of the things on the menu is the aerosol injection. But then we have something called termination shock, which you might want to explain. but how do you then weigh the risks between different X risks? Your view is going to be familiar with stratospheric aerosol injection and the notion of geoengineering in general.

Excellent. That saves me a lot of time. So as mentioned, if you have stratospheric aerosol injection, there are potentially going to be new risks you create. Myself and my colleague Aaron Tang back in Australia, we did a paper which tried to explore what are potential catastrophic risk contributors from stratospheric aerosol injection? The two major ones we came up with were, as you mentioned, termination shock, but the second one that's least discussed actually is political attribution.

Once you put in place a stratospheric aerosol injection system, and particularly if it's done just by a single country or by a small group of countries, every single extreme weather event that happens thereafter, there's going to be a question mark as to whether that was caused by it. And of course, even if you have a good attribution system, misinformation, disinformation, it's going to be a pretty good number of people who are going to doubt whether or not it was eventually it was in some way caused by the system. The scope for having political fallout and conflict due to that is enormous and difficult to address. The second one you already mentioned, termination shock.

So termination shock is essentially, right now we're experiencing warming climb as emissions climb. If you use stratospheric aerosol injection, you're not actually change in the level of missions the atmosphere. You're more or less putting a temporary salve on it. And if the system ever gets knocked off either intentionally, you're deactivating a due to political conflict or unintentionally, say for instance you have a nuclear war or a large geomatic storm which knocks the system offline, then rather than the warming creeping up slowly over centuries, so you get, say, three degrees over two centuries, it essentially

comes back all very quickly. Aerosols have approximately an eight-month half-life, which means within about two years, most of it's already gone. Well, it's just a dumb idea then, because that implies that civilization will last indefinitely to keep the aerosol injections going. I mean, that's just a non-starter. For most proponents of aerosol injection, they would say, they see this as being a temporary fix. It essentially is buying time. So you can either switch to normal energy, and or have large-scale negative emissions. That sounds like a very human thing to say. They are human, after all.

I think the way of thinking about this is what you're doing is you're not actually reducing the risk per se. You're changing the risk distribution. The average actually might actually better, depending upon what your view of future existential risk is, but you're definitely inflating the tail. So if something does go really wrong, once again, like a nuclear war and engineer pandemic,

Like anything that's large enough to knock the system offline to keep it off for a while, suddenly the risk becomes much larger because you also get termination shock. Rather than getting three degrees at a space of centuries, you're getting the space of a decade or so. So I think it changes the distribution, and it becomes a matter of pick your poison. Recently, I don't know how up on the TGS episodes you are, but I had Corey Bradshaw from Australia who said at three degrees Celsius, we will lose 50% of species on Earth. I had Johann Rockstrom on.

I had Stefan Romsdorf on recently Leon Simons, all who are very, very worried about climate change. You in your past academic research and your personal interest have studied catastrophic climate change and looked at the scenarios and discovered that there has been a profound lack of research within the climate community on some of these scenarios. Why do you think mainstream climate research have kept away from studying these sorts of scenarios? There's a few reasons. First of all, at international agreements like the 2015 Paris Climate Agreement,

channel scientific and policy attention towards lower level warming scenarios. So 1.5 and 2 degrees in particular. Those are, after all, the goals of the Paris Agreement. And if you're trying to get a grant, then tying it to the Paris Agreement and the mitigation goals is a good idea. Secondly, the climate community has an unfortunate culture of erring on the side of least drama. Due to the merchants of doubt, due to the fossil fuel industry, essentially torpedoing anyone who wants to talk about higher level warming or extreme scenarios, people just naturally try to be very, very conservative when it comes to climate change now, particularly in the research community, they don't want to be branded as an alarmist.

Last one of least is the research community itself insofar as the IPCC, you know, works by consensus, and that by nature is a very conservative process. It favours the lowest common denominator outcomes. And simply doing the research, looking at extreme scenarios, is just way harder. You know, if you're looking at, say, six degrees of warming, suddenly you have to go into kind of geological speculation. as to how that looks in the Earth. 1.5 or 2 degrees is much closer to the current ecological conditions and hence easier to forecast.

And then likewise, even if you're looking beyond warming and just thinking, what are the cascade effects, which is something we don't do very well right now, climate change at all? We essentially just kind of tally up individual hazards and make a very rough guesstimate as to how bad it could be. We don't think about usually even compound risks. So, you know, say, for instance, a hurricane knocks out your way, your aircon and then you have a heat wave at the same time.

And we definitely do very little when it comes to risk cascades with one risk, tricking another. So this is very, very interesting to me because I'm a curious person and I want to understand things. But climate science itself is so unbelievably complex with the geology and the glaciers and the air and astronomy impacts and the geomagnetic field and water vapor and the positive feedbacks. And it's unbelievably complex, as you know.

But that's just one piece of our complex social dynamics of humans and energy and cognitive biases and economics and debt and geopolitics. It's incredibly complex. And then if you add the compound risk factors. like you just mentioned one example. Like we're just at the 11th hour figuring out how to conceptualize these things. And to me, this is our time. This is our species level conversation to realize what is at stake on our watch.

And you said that because their jobs depend on it, they tend to, the mainstream scientists in these areas tend to be conservative. in order to get grants and such. So I would offer that on the flip side of that, there's there's an opportunity and also a risk for platforms like this, for instance. I don't have an employer. My job is to speak truth to power and interview people like you to provide insights

and provide color and suggestions on these things. But there's a risk too that podcasts, not this one, but others sensationalize these risks and only focus on the extreme scenarios, not only climate, but all kinds of other things because there's a popcorn dopamine, misery loves company sort of dynamic that happens with humans. And so then that actually doesn't help the credibility of people like you who are trying to work on mitigating existential risks. Again, that was a mouthful, I'll let you respond in any way you want.

To me, this is precisely why we need to have good research into these things. Because if you don't, bullshit fills the void. We had one letter in response to this article, Climid Endgame, which essentially said, talking about these things is bad, you can encourage mental health problems, etc. And one of our responses was, well, if you don't talk about it in a reasonable scientific manner, if you don't study it soberly, then other people are going to, this is going to do a much worse job, but it's going to simply speculate wildly and usually gesticulate towards the very worst doomsday scenarios. I find myself not just simply telling people climate change could be really bad,

we don't know how bad it could be, and we really should consider it to be a key contributor to existential risk. I also had numerous conversations of activist friends of mine who will basically say the world is definitely going to end in one to two decades, and I'll say, no, that is not sense. We have no clear evidence in that. We had a really bad spot. It could. This is my problem. Hypothetically, sure. Yeah. I mean, I think some people ask, am I a dumer? And I say no. And I think the difference between me and a dumer is a level of certainty. I think doom is certainly under the the probability distribution in my mind. But it's a certain subsistency. of it, whereas a doomer is it's 100% of certainty.

So I think it's a matter of openness and nuance and possibility because there are some potentially catastrophic scenarios in the coming decades for our species, for our world. But they're possible. They're not probable and they're certainly not definite. Yeah. In a recent article published, I believe last year in Wise Climate Change, myself and Joe Davidson had a paper where we talk about this distinction between what we call risk realism and dumerism, or in particular, climate dumerism and climate risk realism. And doomrism is essentially when you think the risk of annihilation is inevitable. The worst case scenario is just

going to happen, and it's usually very short term as well. It's going to happen the next few decades. you know, go and bung it in the hatches, buy some beans, get some guns and ammunition, etc. Climate risk realism is essentially saying there's a possibility that we have extreme scenarios which could result in catastrophic outcomes. And because those outcomes are so consequential, we should pay much more attention to them, much more than we're paying right now. That's not Dumerism. That's just being reasonable. That's what any good risk manager would do. You know, if you're in insurance, for instance, you want to know about the worst case scenarios. That's key critical part of your job. And any kind of decision-making uncertainty usually involves

knowing the worst-case scenario. But unfortunately right now in our current kind of discourse and atmosphere, people always lump those two together. You know, directly after publishing this article, Climate Engelma, we were very careful to phrase this in probabilistic terms. We still had people like, you know, the climatologist, Michael Mann, come out saying, no, don't talk about this stuff. You dooms, et cetera, et cetera. It's pretty toxic, actually, to automatically look. lump in people who want to talk about risk management, we've doing this. The two incredibly separate different camps. Yeah, I agree with that. It's a very difficult landscape to navigate, and I'm not only talking about climate. There's a lot of other issues. So you mentioned

conceptually how to do insurance. I imagine that the insurance industry in the world is not too happy with X-risk and to the point that they almost might have a Dr. Strange-Lovian reaction to it in that, oh, if all this comes to pass, then it doesn't matter. Therefore, they're inclined to just ignore some of these things? Or what are your thoughts on that? So in the wake of publishing endgame, actually, the most receptive audience probably had were people working in the actuary in insurance businesses. Okay. Just by nature, it's kind of speaking their language, and most of them are just kind of like, wait, you are doing this already in climate science, you're not taking a risk management perspective.

That's at the lower levels. I'm not sure how people, say, for instance, Munich R.E or Swiss R.E., the biggest reinsurers, how people the top levels really think about this. I just don't know. But in general, I think most people who are trained in the kind of risk management disciplines, they actually tend to get this stuff much better. Well, shouldn't some of those people, especially if they've amassed some savings during their life, retire from insurance and apply their skills towards the X-risk community growing and becoming more impactful? Ideally, yes. Just as a side note, we should definitely look back to the question of responses as well, because I think there's actually a really good mini-conversation to be had there. On responses to X-risk? Yeah, and importantly, how can our responses potentially make the situation work?

cause new risks, because we had a bit of a conversation about stratospheric aerosol injection in particular, but I think there's a bit more around that. Yeah, go. No, no, no, no, no. Go for it. Because I was going to ask you that. Please continue with your thoughts on that, Luke. So apart from stratospheric aerosol injection, I think when we think about extensional risk in general,

it was a whole bunch of ways in which you can go wrong. And I'm particularly cognizant of the fact that simply by studying existential risk, you can cause dangers. I had an article published a BBC a while back called The Stomp Reflex. This is what we refer to as information hazard, that you can't say something because then it becomes a self-fulfilling prophecy. Yeah, so Nick Bostrom had the notion of information hazard, which is essentially any kind of information that can do harm, essentially. So, you know, a classic example here is if you release the genomic sequence for a virus and there's technology to potentially recreate that virus, the information by itself is potentially dangerous. You know, you don't want people having access to the genomic sequence of smallpox,

which is technically already out there. But anyway. Thanks for just sharing that information hazard. But go on. I mean, the thing is, if you are a highly dedicated terrorist who wants to destroy the world, the barrier and roadblock at this stage is not going to be knowing whether the smallpox virus is out there. They already know, basically. I think we get a bit too caught up in this.

much more, I think, about resources and capabilities. But anyway, what I'm talking about here when I prefer to stomp reflex is slightly different. It's... Stomp, S-T-O-M-P? Yeah, so I think stomping down, stump reflex. So I already mentioned the idea with the authoritarian impulse, this psychological phenomena where when people feel threatened by an external threat, whether that's a pathogen, an invader, even potentially foreigners, they're more likely to accept.

or foreign-herent structures. And likewise, we see this phenomena when it comes to emergency responses, say, for instance, the way we even respond to COVID-19, it tends to almost always have one characteristic pattern. It always goes top-down. You always give a lot of power to people on top of the hierarchy. You essentially give far less to those below,

and you start to impose draconian measures. And additionally, you often start to have less transparency at the highest levels like within government. The strange thing here is that this goes actually against most of the findings in disaster risk management. In disaster risk management, there's been a really critical, consistent finding that the notion of mass panic is largely a myth. Whether it's a tsunami, a hurricane, a terrorist attack like 9-11, or an earthquake like the great earthquake of San Francisco or the earthquake which struck Mexico City, in every case, people tend to respond with an incredible amount of pro-sociality, altruism, and just kind of natural ingenuity as well.

You know, setting up makeshift camps, providing food for those in need, helping excavate those in trouble, etc. What we actually have good evidence of is elite panic. That often, when elites respond to big problems, they either use it for their own political machinations and motivations. So look at, for instance, what happened after 9-11. We got two walls worth of a couple of trouble. millions of dollars and a huge mass surveillance system. And they can just basically mess up by,

you know, sending in the troops, for instance, and making things much, much worse than it needs to be. So it's interesting that the evidence base kind of points us towards people who respond pretty pro-socialally, the bottom up way, but we need to be really careful the way we use top-down structures. And yet, our response is exactly the fucking opposite. Every single time we give a huge amount of power and less transparency to those atop the hierarchy. And suddenly, everyone in below, they need to be watched really close. We need more surveillance. We need more police and more military boots on the ground. It just doesn't make sense, but it is what I call a stomp reflex that essentially people in power tend to respond by stomping down those below

as soon as there's a crisis. And the authoritarian impulse is that people tend to strangely accept it a lot of the time. And this is what I'm really worried about when it comes to existential risk, is that if you start talking about things like bioengineered pandemics and that a small group might be able to synthesize a virus and do massive harm. Then suddenly the automatic response from a lot of people in power, particularly military, is going to be mass surveillance. Just put everyone in surveillance, basically. Have much stricter export controls.

And there's a whole bunch of things which might be good here, but I don't think mass surveillance is one of them. And we actually don't have much good evidence that it works. But already in my field have had people advocating for this. Nick Bostrom, so probably the biggest kind of founder and most influential scolomfield had an article called the Vulnerable World Hypothesis, where essentially I hypothesizes that if you have weapons that can be very easily created. So he uses a kind of funny example of imagine you put sand in a microwave and that creates a nuclear

explosion. Imagine if it was that easy to create a nuclear weapon. Then the only way to really address it would be to have mass surveillance and preemptive policing. It gives us really weird scenario, people walking around with these what he calls, ironically, of course, freedom tags, which have just a whole bunch of sensors and cameras around them. And as soon as you start to, you know, use a bit of code, which is potentially too close

to creating a very intelligent algorithm, or you start to synthesize a virus, then automatically you send out drones to assassinate the person. Very sci-fi. It's sci-fi, but it's based on a credible living human researcher. Yeah, very.

well respected on. I think he's got it wrong insofar as I don't think you can start of a weird Ford experiment and start to make a very large, severe policy recommendations on that basis. But the thing is, this is exactly the kind of idea, which is going to catch on very well amongst, I think, elite level thinkers in the military, for instance, in the security apparatus. This gets back to our incentives and the lootables that created the hierarchy and then fossil fuels and then digital money and now AI, it's turbocharging this hierarchy. And if you just read the tea leaves, I mean, I don't know what X risk is going to manifest, if any, but there's a lot of little things, including the election and geopolitics and the wars

in Ukraine and Israel. How likely is this universal surveillance and things like that? I mean, does it become a surveillance singularity that is. coming our way. When we think about the future, there's two ways of doing it basically. There's forecasting and foresight. There's a key difference between the two. Forecasting is making a bounded numerical estimate about a future event. So, you know, there's a 61% chance that Trump wins the next election, for instance. That's a prediction. That is a forecast. Foresight is trying to paint generally plausible pictures about a future scenario. Quite different. They have different

methods and tools use for them. Most people are usually talking about forecasts and they ask these kind of questions. They're kind of asking for, you know, what's the probability you have? You know, one in six, for instance, is the very famous one that Toby Ord gave. And there's lots of these out there in the community right now. So Jarrod Darmine gave a 49% chance of the world basically collapsing in some kind of way by 2,100. Toby Ord, obviously in his book, The Precipice gives a one in six chance. There was a survey done by Nick Bostrum and Anders Samburg, which gave a, I think, an average of 19% Nick Bosch from himself said he wouldn't go any lower than 25%.

Martin Rees here at the University of Cambridge gave a flip of a coin, tails you lose and go extinct, heads, you potentially have techno-utopia, you weigh 50% chance. So there's lots of different estimates out here. The thing is that I wouldn't put any stock on either of them. When it comes to forecasting, it tends to work best for having really clearly defined simple problems. Like, for instance, who's going to win the next election? Or who's going to win the next election?

Additionally, what's been shown to work with forecasting is essentially having large groups of people, usually from diverse backgrounds, who are particularly preternaturally good at doing forecasts, what are called super forecasters. These are people who are really good at probabilistic thinking, and they're good at holding multiple different, hypotheses in mind. They usually do lots of research as well. And essentially, you get a large group of them together. You have them sham formation, deliberate, and then you kind of take the average and you extremize it a little bit. That's been shown in numerous tournaments to be the most effective way doing forecasting. And even then, even with super forecasters, they tend to have decay. They become far less accurate after roughly a year or two. In short, if you're looking at making

numerical estimates about really complex big topics with long time horizons. Honestly, we don't really have that much an idea. Yes and no, because I think we're pretty confident that the thermal inertia of fossil fuels and land use change is going to end up in a warmer world. I am quite confident. I have no idea about civilizational collapse or anything like that, but I'm quite confident that in coming decades, the average human is going to have to use less energy and materials on average.

I feel pretty confident about that. I could be wrong. There could be some new energy invention with AI. But if there is, it's going to be a larger draw on our ecosystem services, which is going to feed back into the climate. So there are things that we can say with high degrees of confidence, yes? Yeah. So this is much more foresight-based, but right?

So you're not saying there's an 11% chance that everyone's going to die by either 2009. Yeah, I have no idea. But totally, I think, you know, we can very easily say the world is going to get warmer and it's likely going to be somewhere between, say, two degrees to four degrees by the end of the century. You know, we have more than enough events to point towards those kind of things. And of course, there could be stratospheric aerosol injection before then, but nonetheless, we can paint some pretty plortful fictions. And likewise, you know, I've already said early in the podcast, I think our current system in the way it produces risk is likely going to self-terminate or be radically reorganized within the coming centuries. Pretty confident in that.

I wouldn't put, I have a really high confidence kind of numbers to it at all. And I think this where it gets dangerous. When people hear numbers, they anchor it really heavily. Well, it's not only numbers, but it's also time, because you just said centuries. And when you use the word centuries instead of decades, it's almost like as unbelievable as a leprechaun, because it's so far in the future to not be emotionally plausible to me or my children. But if you say decades, someone I know, if not myself, will be alive during those times and experience it. It's a different emotional valence when you use the word decades versus centuries. Absolutely. And in this case, it's probably the academic of me unwisely kicking in where I usually use centuries just because I have a fair bit of uncertainty

in terms of how long the timeline he will be. I think there's a pretty good chance this actually does operate more on this time span of decades. So my next question is, what do we do, Luke? Leave the easy ones to last night. Well, I mean, so there's many factors that make individuals and communities and a nation more resilient to external forces and potential ex-risk. So let's start there. What does your research suggest there? So one of the strongest piece of evidence we have, both from disaster risk management, so looking at modern-day disasters, but also looking at some archaeological case studies, is having more inclusive institutions tends to make societies more resilient.

How do you find inclusive institution? essentially it's more or less just a synonym for democracy, having more people involved decision-making, having a wider distribution of decision-making power. So there was some research done by a guy called Peter Peregrine at Yale University. He's an everologist, and he tried to look at what's called the late antique little ice age, which is basically this period which roughly coincides to the demographic collapse of Rome, and it's a pulse of cooling caused by volcanoes across the northern hemisphere. And he took, oh, I think it was 22 different societies across the world.

So Hunter Gavrexon, Ontario, for instance, Montereban, one of the first states in Meso America, as well as Rome. And he tried to both create an index for how much they changed socially. So was there a population collapse? Did they reorganize himself politically? Did they have disease, assassinations, etc.? And also how much cooling they encountered. And the two key interesting findings he had was one, the cooling didn't always match the level of social change. Some places experienced very little cooling, but had massive political change.

Other ones experienced a lot of cooling, but were surprisingly resilient, like the Hunter Gavras in Ontario, for instance. And the second was that the biggest predictive factor was inclusive institutions. Basically, those who had much stronger dominance hierarchy and kind of less democracy, more glorification of leaders and more centralization of authority and power, tended to respond less favorably to climate change. And this actually echoes a huge body of literature in disaster risk management. In disaster risk management, one of these strongest findings is basically having greater democracy and having greater involvement of local people tends to result in better

responses to disasters. And there's a few good reasons for this. One is that people on the ground are less buffered from oncoming disaster. And hence, they're more likely to take it seriously, they're more likely to know contextually what is the actions going to work to actually reduce the risk. A second one is that in more inquisive institutions, you know, when people are actually doing things like voting, being involved with civic institutions, whether it's a sporting club or an actual local council, they're more likely to build social capital, what I prefer to just call social

interconnectedness. Basically, people have lots of friends who they can fall back on a time of crisis, and that's usually the thing that actually helps, is not having guns or beans, but actually having lots of friends and social networks. And last one or least, I mentioned super forecasters and the fact that, you know, having super forecasters, big diverse groups, exchanging information, and kind of taking the average, that tends to result in better judgments. And this is actually incredibly common, whether it's in foresight, for instance, the study of collective intelligence, even within computer science, or the study of deliberate democracy, is that having bigger, more diverse groups almost always results in better groups. or the study of deliberate democracy is that having bigger, more diverse groups almost always results in better group judgments than having an individual make a decision. So I think on top of that, there's simply democracies are going on average make better decisions and autocracies will.

Thank you. Double-clicking on your middle point where you said that social connections are really important. social connections are non-lutable, right? It's a form of capital that you can't loot like smoke salmon or beans. Have you thought about it that way? Actually, surprisingly not. That's actually a little prompt. I mean, there is some evidence that social connections are partially hereditary.

You know, obviously, if you're born to a billionaire, you're likely to have a whole bunch of social connections that your peers from the law, from lower classes we all have. But nonetheless, they can't be easily looted per se. Well, there's only so many self-jection. They're hoardable, though. They're hoardable but non-lutable. I'd agree with that. Yeah.

I think, I mean, the way I phrase this within the book is that most forms of power can be hoarded in some way. But it's not until you get economic power in particular that it's highly fungible. You know, once you have economic power, you can start to pay for your own private military, for instance, you can start to pay for lobbying, or even in the case of Donald Trump, going for a president yourself. That's what really changes in the whole scene is you start to get a form of power that cannot just be concentrated by itself, looted resources, but can be exchanged for all the other

assaults of power we know of in the world. Information power, control of information, control of people, control of the violence, and control of decision making. So for people working at institutions or in government or just people listening to this show, what recommendations do you have for them along the lines of X-risk? What do you tell people to do who are listening to this podcast and learning about existential risk as you've laid out here? First of all, make it a political matter. It is tragic that essentially no US election has been decided by the candidate's policy

on nuclear weapons. And yet in many ways, that is by far the most consequential thing a president can ever do. There's actually a good argument to make that right now the US is not, at least not fully a democracy per se. It's a thermonuclear monarchy. The most devastating thing that could ever be done is the US launching a nuclear strike. And yet that can be decided by a single person, the president. No democratic deliberation or approval required. And yet that never factors into elections whatsoever.

And likewise, if you're really worried about climate change, that should be one of the deciding factors for your vote. In general, people, when casting a vote or thinking that politics, existential risks should be one of the things that's forefront in their mind. And because of that, I'd say, you know, think politically. Think about who you're voting for and think about who's going to do the most to reduce catastrophic risk. Have conversations of people as well. I've been both kind of horrified and also surprised by how few of my friends even know the basics of how nuclear winter work, for instance. You know, if you're going to be living in a country like the UK or the US, which have its nuclear stockpiles, you should know what nuclear winter is. That's just a basic duty as a citizen. Apart from that, there are individual actions you can take as well. As mentioned, you know, with there's things like divestment, you can make sure that any money you have invested, whether it be in a pension or in the stock market, is not going towards key producers of risk. Right now, there's no one who's doing work on actually making sure a portfolio is kind of completely divested from risk per se. We have ESG. environmental, social and governance ones.

We have obviously client divestment, been nothing on in general X-risk. That would be cool to see in the future. I'm skeptical on that one because there's direct X-risk that people aren't investing in. But if they divest from that, then anything they put it, I mean, like Home Depot and Walmart aren't creating X-risk, but a lot of the things they sell ultimately lead to the superorganism and global heating and excess emissions and consumption, because everything is. compounded and intertwined. Absolutely.

But I'd see this more as reducing, if you will, X-risk intensity or even the Karmadence, if you're in instance, your portfolio, rather than just simply saying, I'm not going to be contributing to X-risk at all because that's, as you mentioned, more or less impossible. If the entire system, if this is a system's level from another, is producing catastrophic risk,

well, you can't really divest from the entire system, unfortunately. But nonetheless, the important thing here is to send a market signal that the key produces, people have called the 80s, agents of Doom previously are getting a marking signal, there's no longer support for them, basically. That's always been the idea of divestment basically is not to actually financial wound them, but to take away the social license so they're more prone to being more vulnerable to being lobbied, regulated, etc. I'd see it's acting in a similar way. So what, in your opinion, Luke, what research is vital that isn't being done in terms of X risks?

and what is your forecast for the future of your industry on what needs to happen in a perfect world? And are there any initiatives that you find particularly promising you'd like to highlight? So four things come to mind. I've already discussed a few of them. First of all, is actually mapping the interconnections of existential risk, having what I've previously called integrated catastrophe assessments. A second is having better risk-risk calculations. So basically trying to think through, if you do some kind of reaction or policy response to an existential risk, what are going to be the risks that's creating? And how do you potentially bounce and make a decision?

A third, I think, is looking at the root causes. As mentioned, there's very few people thinking about this causally. People almost always just focused upon an individual hazard, whether that be creating a safer version of automated. cognition or trying to reduce nuclear weapons stockpiles. And those are useful interventions. I'm not saying don't do that, but the root causes, I think, are the main game and the most neglected one at this stage. And last but at least is, I think we need to have both better research, but also just a lot more soul searching and fall forward about how we do our research and who we work with. A very large number of my colleagues work very closely with deep mind,

entropic, open AI. I had a number of work in the military as well. And there's a big question of is this potentially doing more harm than good sometimes? Is working with the key risk producers, potentially giving them both legitimacy, but also potentially corrupting the researchers to a certain extent. We've seen this when it comes to food research, research and security, etc. It's really hard to separate your research and your ideas away from the people who are funding you. Yeah, I think it's, I think there's a third risk you didn't mention that's the biggest of all, which is you're creating an antidote. You're creating antibodies in Anthropic and the other places because you're studying the risks that they're providing to society and you're sharing those with them and therefore they're mitigating those things themselves so that they're not recognized as risk. I would think that would be the biggest.

Yeah, potentially as well. Yeah. Yeah. So unexpectedly, well, expectedly, I have way more questions than I was able to get to and we're nearly out of time. So I'm going to have to have you back. Absolutely. Before I do that, you've watched my podcast. I do have a few personal questions outside of X risks, but just living during this age of the great simplification and metacrisis or whatever your label is, as a human. young human, what personal advice do you have to the listeners of the show who have listened to a lot of my episodes and now this on Exorc's and things that they didn't know, which are quite frightening. What sort of advice do you have to individual humans? I'd say first of all, take a Hippocratic oversawks. Try to minimize the amount of harm you're going to do with your career and with your life as well when it comes to the same game. I think. there's a great temptation right now to become, same if, for instance, machine learning expert, but there's also going to be a much bad chance that you end up, of course, getting employed

in big tech, for instance, and trying to find ways that your career as well as your personal life doesn't severely contribute to external risk, I think is pretty critical. Apart from that is, if you want to go all the way, take what would be called a Ophah of My Menides, which is essentially to actually try to live your life and use your career in such a way there's going to reduce risk. And that doesn't mean becoming a full-time activist, for instance, you know, there are ways where it be working in the civil service or, of course, actually work in the research community of X-risk to reduce risk in a whole bunch of ways. So those would be the two big ones, I think. Take a Hippocratic and a Maimonides oath to reduce extantial risk.

What was the second one? I don't know that name. The Oph of Maimonides. So it was essentially also a Greek oath. In this case, what it explicitly says is essentially like make things better rather than simply to do no. What do you care most about in the world, Luke? So I thought we're just having like a normal easy conversation and just like throwing these things at me. Okay, what do I value most? Consciousness, sentence, the actual ability to experience.

If, I was to go deep than that, because I think this is true for most people, you know, even if someone says something like love, it's ultimately the ability to actually feel and experience. love, it is consciousness. I mean, that's the very basic fabric of reality. But one thing I don't think is mentioned enough, particularly in my own field, is justice. I think the key to actually getting out of this endgame into reducing extantial risk is justice. And I think when you think about existential risk, when you think about this global system producing it, it is fundamentally built on

injustice. It's, you know, it's not just the executive making decision to log, the government to increase fossil fuel subsidies. It's also the miners who are often working in incredibly poor conditions and getting underpaid, not so much in Australia, but elsewhere. And likewise, even if you can look at say nuclear weapons, again, there's miners working at the front, digging up plutonium, often in terrible conditions. And its indigenous community has been displaced and often killed every day to allow for more land that this global glyph can eat up. So for me, justice is absolutely. critical, not just the human experience, but to existential risk.

And not just the human experience, but the non-human experience. What about the millions of other species and trillions of other animals and organisms on the planet that have no say in our existential risk mitigation or cause? Yeah, precisely. And for me, this is actually one of the most important reasons to reduce eccentric risk. A lot of people, this ends up just being a numbers game that reduce Excentral Risk is important because there's lots of people, and, oh, go, it's a good thing to do. And more importantly, there'll be lots of people in the future. It was a very famous Ford experiment provided by Derek Parfit of, imagine you have a 1% chance of nuclear war, a 99% chance of nuclear war, a 99% chance of nuclear war.

Oh, no, sorry, I'm getting you totally wrong. I'll just replay all that. a nuclear war that kills 50% the population, a nuclear war that kills 99% of population, or a nuclear war that kills everyone. Where's the biggest difference there? And Parfit somewhat counter-intuitively said, the biggest difference is between 99% and killing everyone, because if you kill everyone, you're also killing all generations to come. And of course, there could be, particularly you go off Earth, billions of those. And look, there's a value to that. There's a value of thinking about this in a kind of cold numerical fashion. But to me, one of the reasons I want to use

extra risk is because it's kind of part of a redemption story for humanity. For the last 5,000 years, we've been a bulldozer, not just to each other, but to every other form of life on this planet. And we don't get the chance to become good stewards or to redeem ourselves for the past unless we, of course, continue and unless we transform. If you could wave a magic, wand and there was no personal recourse to your reputation or safety, what is one thing you would do to change human and planetary futures? Democracy democratize everything. And I want to provide one example for this. So I'm guessing most of your viewers will be familiar of the Trinity Project, the creation of the first atomic weapon in the United States. So in the sands of New Mexico,

they're creating an atomic bomb. And there's a calculation, by a physicist called Leosy Lizard, where he essentially calculates that there's a very, very small infantestimal, but nonetheless, there is a chance that if you ignite the bomb, it'll actually ignite the entire atmosphere with it, killing every single life form on the planet. It will incinerate the entire biosphere. Yeah, they talked about that in the movie Oppenheimer. Precisely. Yeah.

And just, it's low probability, but it can't be ruled out. And, of course, they still go ahead. and do it. And they're not only that, they actually take bets in a very sardonic way on whether or not this will incinerate the entire world. Now, imagine alternative universe here, where the decision to use the atomic weapon, to basically have the trinity test, is not taken by a bunch of physicists and policy makers in the US, but instead they decide to do a jury. They randomly select a number of citizens, plumbers and nurses from across the US. They have them sit down, and, and they have them sit down, And they're briefed by the physicists.

Leo Silsid says, we can't rule out the possibility. This will destroy life on Earth. Importantly, the intelligence agency analysts also come to them and say, we know that the Nazis are no longer going to build their own bomb. They weren't taking the right course in the first place, and they've actually given up on a base stage. So this is no longer even a race to prevent a thermonuclear Third Reich. This is essentially just about US power,

and potentially, in the very best case, end in the war early. Imagine these nine people, a dozen people, maybe even more. They're given that information and told to make a decision on whether or not to ignite the Trinity test. What decision do you think they make? The plumbers and nurses? Yeah. I would say that they would say don't go forward. Yeah.

They're going to say fuck no. I've yet to meet anyone who doesn't usually either laugh and incredulity or just immediately ends up. of course they're not going to. It's just, it's a stupid thing to do in the first place. But I think this goes for a lot of existential risks. Once you actually put it in the hands of regular people who just have kind of sober access to the information, and more importantly, aren't going to benefit from its creation, money of these things become blindingly simple and obvious. It's by democratizing. What you just described, that is actually the existential risk, is the fact that we are in a situation of downward causation and rationality,

reasonableness, science, ecological stewardship, all the things have no truck in the power hierarchy and everything that's happening at the high levels that you're discussing. Yeah, at the very cause in many ways that you've really about the dominance hierarchy we're stuck in. And unless you find a way of democratizing and spreading power, then I still think in the long term you screwed. Well, not just me, but everything. I don't know. When I say you, I mean, all forms of life.

Yeah, well, you're not the first to say that. Dude, this has been a really great and interesting conversation. And I'm sorry that I'm not going to go for a bike ride in the two hours preceding a call again in the morning. No worries. I mean, I've left with so many questions. If you were to come back on the show, what is one topic relevant to the great simplification that you would be willing to take a deep dive on and unpack your expertise? decent scholarship. Societal collapse.

I have a book coming out next June called Galives Curse, The History and Future Societyoclapse, and it'll touch upon some of the territory we've recovered, but also much more, particularly from a historical lens. And I think we can have a really good conversation around that. And it also may have some stuff from complexity and simplification that would be interesting too. Let's do it.

Let's record it in April or so and at time it with your book release. Excellent. Sounds good to me. Great to meet you, Luke. To be continued. Likewise, Nate. Looking forward to it. If you enjoyed or learned from this episode of The Great Simplification, please follow us on your favorite podcast platform. You can also visit ThegreatSimplification.com for references and show notes from today's conversation. And to connect with fellow listeners of this podcast, check out our Discord channel. This show is hosted by me, Nate Hagan's, edited by No Troublemakers Media, and produced by Misty.

Stinnett, Leslie Batlutz, Brady Hyann, and Lizzie Siriani.