Making Sense with Sam Harris - #292 — How Much Does the Future Matter?

Episode Date: August 14, 2022

Sam Harris speaks with William MacAskill about his new book, What We Owe the Future. They discuss the philosophy of effective altruism (EA), longtermism, existential risk, criticism of EA, problems with expected-value reasoning, doing good vs feeling good, why it's hard to care about future people, how the future gives meaning to the present, why this moment in history is unusual, the pace of economic and technological growth, bad political incentives, value lock-in, the well-being of conscious creatures as the foundation of ethics, the risk of unaligned AI, how bad we are at predicting technological change, and other topics. If the Making Sense podcast logo in your player is BLACK, you can SUBSCRIBE to gain access to all full-length episodes at samharris.org/subscribe.   Learning how to train your mind is the single greatest investment you can make in life. That’s why Sam Harris created the Waking Up app. From rational mindfulness practice to lessons on some of life’s most important topics, join Sam as he demystifies the practice of meditation and explores the theory behind it.

Welcome to the Making Sense Podcast. This is Sam Harris. Well, it's been a big week for news. I think I'm going to wait for a few more shoes to drop before touching any of it. Earlier in the week, Trump's home at Mar-a-Lago. Is that really his home? home at Mar-a-Lago. Is that really his home? That golf club that looks like a dictator's palace in what he might call a shithole country? It was raided by, quote, raided by the FBI in search of misappropriated documents that may or may not contain secrets about nuclear weapons or nuclear policy or something nuclear. Again, I think I'll wait for more information to surface before I comment on this, but needless to say, it would not surprise me if Trump had done something terrifyingly inept and self-serving and even nefarious. So let's just

wait and see what happens there. I share many people's concern that the execution of this search warrant was inflammatory enough that if the facts in the end don't seem to merit it, it will be yet another thing that bolsters Trump's political stature in the personality cult that has subsumed the Republican Party, and that would be a bad thing. Once again, this is an asymmetrical war of ideas, and any misstep that seems to suggest that our institutions have become politicized, in this case the FBI, run, we should note, by a Trump appointee, Christopher Wray, not the fittest object for allegations of anti-Trump partisanship, really, really. But any apparent bias here makes any further discussion of facts more or less impossible.

So anyway, let's wait for the dust to settle. Let's see what was actually in those boxes, and maybe there'll be more to say. And then yesterday, the author Salman Rushdie was attacked on stage by a man wielding a knife under conditions where he obviously did not have enough security. And it sounds like he is on life support now. So that obviously is terrible. And I share everyone's horror on his account. I said as much on Twitter, not so many hours ago, where I wrote, like many of you I'm thinking about Salman Rushdie now. The threat he's lived under for so long, which was so horrifically realized today, was the product not merely of the hatred and zeal of religious fanatics, but of the cowardice and confusion of secularists. Everyone in arts and

letters should have stood shoulder to shoulder with Salman in 1989, thereby distributing the risk. And the fact that so few did is a moral scandal that still casts its shadow over the present. its shadow over the present. And now I now notice in my at-mentions that I am being attacked by many people for being soft on Islam and for using the phrase religious fanatics out of an apparent unwillingness to name Islam and its theology as the culpable party here. Islam and its theology as the culpable party here. All I can say is I'm surprised and somewhat amused by this. I mean, these are people who clearly don't know my history. I have been accused of Islamophobia even more than I've been accused of having Trump derangement syndrome. It might surprise some of you to learn. Anyway, my conscience on that score is quite clear, and I invite those of you who doubt my bona fides as a critic of Islam to do some googling.

I even wrote an op-ed with Salman in defense of our mutual friend Ayaan Hirsi Ali, I think in 2007. That's on my blog somewhere. I believe the title is Ayaan Hirsi Ali Abandoned to Fanatics. It was published in the Los Angeles Times, but I think it's behind a paywall there. Anyway, I don't actually know Salman, despite the fact that we wrote that op-ed together. That was accomplished by email.

We've never met in person, which surprises me. We've always had several good friends in common, Ayan being one, Hitch being another. Salman was one of Hitch's best friends. I was merely a friend. Anyway, thinking about him, I was merely a friend. Anyway, thinking about him, and I would just say, by way of spelling out further the point I was making on Twitter, my use of the phrase religious fanatics was not at all an instance of my mincing words

with respect to Islam, although I can see how someone might have thought that if they were unfamiliar with my work, I was just assuming enough familiarity with what I have said and written on this topic that it simply never occurred to me that I could be accused of pandering to Islamic extremists here or to the confused liberal secularists

who would support such pandering. I just assumed my reputation preceded me here. Anyway, the real point I was making here was that the problem has always been larger than mere Islamic extremism and its intolerance to free speech. This extremism has been given an immense amount of oxygen by confused and cowardly and, in some cases, well-intentioned people who worry that any criticism of Islam will give energy to racism and xenophobia. Salman was attacked by such people from the moment the Ayatollah pronounced his fatwa. And the list of people who rushed to heap shame upon their heads by criticizing a novelist forced into hiding out of deference for the sensitivities of religious lunatics,

is an awful list of names. And so it is with more recent atrocities, like the Charlie Hebdo massacre. The people who explicitly or implicitly took the side of the jihadist murderers in criticizing the butchered cartoonists for their alleged insensitivity, these are people who are still in good standing in many cases, and I'm not aware that any have apologized for having got those

moments of moral and political emergency so disastrously wrong. I'm doing my best not to name names here, but anyone interested can see who took the side of religious barbarians there. And there have been so many moments like that since the Fatwa on Salman was first pronounced. And many of you who are living by podcast and sub-stack newsletter might take a look, because some of your modern free speech heroes were people who thought that just maybe those cartoonists had it coming to them. Anyway, this is an awful thing. I hope Salman recovers, and whatever happens there, if there's more to say, I'll do a podcast on it. I'll do a podcast on it.

Okay. Today I'm speaking with Will McCaskill. Will is an associate professor in philosophy at the University of Oxford and a TED speaker, a past Forbes 30 Under 30 social entrepreneur. He also co-founded the Center for Effective Altruism, which has raised over a billion dollars for charities. And he is the author, most recently, of a wonderful book titled What We Owe the Future. And that is in part the topic of today's discussion. I'll just read you the blurb I wrote for that book, because it'll give you a sense of how

important I think Will's work is. No living philosopher has had a greater impact upon my ethics than Will McCaskill. In What We Owe the Future, McCaskill has transformed my thinking once again by patiently dismantling the lazy intuitions that rendered me morally blind to the interests of future generations. This is an altogether thrilling and necessary book. And that pretty much says it. The book is also blurbed by several friends,

Paul Bloom, Tim Urban, Stephen Fry. As you'll hear, Will has been getting a lot of press recently. There was a New Yorker profile that came out a couple of days ago. Also the cover of Time magazine was devoted to him and the effective altruism movement. And all of this is great to see. So Will and I get into the book. We talk about effective altruism in general and its emphasis on long-termism and existential risk. We talk about criticisms of the philosophy and some of the problems with expected value reasoning. We talk about the difference between doing good and feeling good, why it's so hard to care about

future people, how the future gives meaning to the present, why this moment in history is unusual, which relates to the pace of economic and technological growth. We discussed bad political incentives, value lock-in, the well-being of conscious creatures as a foundation for ethics, the risk of unaligned artificial intelligence, how we're bad at predicting technological change, and many other topics.

No paywall for this one, as I think it's so important. As always, if you want to support what I'm doing here, you can subscribe to the podcast at samharris.org. And now I bring you Will McCaskill. I am back once again with the great Will McCaskill. Will, thanks for joining me again. Thanks so much. It's great to be back on. So you are having a bit of a wild ride. You have a new book, and that book is What We Owe the Future. And I got a very early look at that, and it's really wonderful.

I don't think we'll cover all of it here, but we'll definitely get into it. don't think we'll cover all of it here, but we'll definitely get into it. But before we do that, I want to talk about effective altruism more generally, and in particular, talk about some of the pushback I've noticed of late against it. And some of this echoes reservations I've had with the movement, and some of which we've discussed, and some of it strikes me as, frankly, crazy. So yeah, let's just get into that. But before we do, how are you doing? How's life? I mean, things are very good overall. I definitely feel a little bit shell-shocked by the wave of attention I'm currently through. And, you know, that's, I don't know, I'm alternating between excitement and anxiety. But overall, it's very good indeed. And I'm happy these ideas are getting

more their time. Am I right in thinking that I just saw EA and a long interview with you on the cover of Time magazine? That's exactly right. So that was the news of today that came out this morning. And I think it's a beautiful piece by Naina Bajikal. And yeah, I'd encourage anyone to read it if they want to get a sense of what the EA movement is like from the inside. bend more of it here. There's a lot we've already said about this, but I think we have to assume that many people in the audience have not heard the many hours you and I have gone round and round on these issues already. So at the top here, can you define a few terms? I think you should define effective altruism, certainly what you mean or you think we should mean by that phrase, and also long-termism and existential risk, because those will be coming up a lot here. Sure. So effective altruism is a philosophy and a community that's about trying to ask the question,

how can we do as much good as possible with our time and with our money, and then taking action to put the results of that or answers to that question into practice to try to make the world better. Long-termism is a philosophical perspective that says at least one of the key priorities of our time is to make the long-term future of humanity go well. That means just taking seriously just the sheer scale of the future, how high the stakes could be in shaping it. The fact that there really might be events that occur in our lifetimes that could have an enormous impact for good or ill in our own lifetimes and taking seriously that we can help steer those events in a more positive direction,

helping set humanity on a better path to improve the lives of our grandkids and their grandkids and their grandkids and so on. Then the final term, existential risks, are a category of issues that I and others have increasingly focused on or started thinking and worrying about. And these are risks that could be enormously impactful, not just in the present, but in fact for the whole future. So some of these risks are risks that threaten extinction, such as the use of our next generation of bioweapons in World War III with viruses of unprecedented lethality and infectiousness that could really result in the collapse of civilization or the extinction of everyone alive today in the worst

case. Or very rapid advances in artificial intelligence where that could result in, as you've argued, the AIs being just far more capable than human beings, such that the AI systems merely see humans as an obstacle to their goals. And humans are disempowered in light of that, or in fact, again, just made extinct or eliminated by such systems. Or existential risks can lead to permanent loss of value or future value, not just via extinction. So a perpetual dystopia that was, you know, perhaps it's a little better than extinction, but not much better, with a lock-in of fascist values.

That would count as an existential catastrophe as well. And increasingly, I've been getting to know the leading scientists who think that the scale of such existential catastrophe, it's actually not small. It's not like, in any way like this, one in a million chance of an asteroid strike in the next century, but much more like the risk of dying in a car crash in one's lifetime, where these risks are certainly in the next century, but much more like the risk of dying in a car crash in one's lifetime, where these risks are certainly in the multiple percentage points, maybe even tens of percentage points, just in our lifetime alone. So in the definition of effective altruism, I think one could be forgiven for wondering what is novel about simply wanting to do as much good as one can,

right? I mean, how is that a new contribution to the way we think about doing good in the world? Hasn't everyone who has attempted to do good wanted to do as much good as they could get their hands around? What do you view as being novel in this movement around that? their hands around. What do you view as being novel in this movement around that? So I'm certainly not claiming that effective altruism is some radically new idea. I think you can see its intellectual roots going back decades. I actually argue you could see it going back two and a half thousand years to the first consequentialist model philosophers, the Mohists. the first consequentialist model philosophers, the Mohists. But I actually think that this mindset of trying to do as much good as possible is at least somewhat unusual, where in general, when people try to do good, and this was certainly true for me before I started thinking about this,

they are looking at what causes have been made salient to them, where in my case, that was extreme poverty. I remember as a teenager reading about how many people died of AIDS each year, just thinking, oh my God, this is an atrocity. How can we not be taking more action on this? But what I wasn't doing at the time was sitting back, kind of taking a step back and thinking of all the things, all the many, many problems in the world, what are the things, what's the thing that I should be most focusing on if I want to have the biggest impact I can? What's the best evidence? What are the best, you know, what are the best evidence saying about this? What are the best arguments saying about

this? And really thinking that through. And so I actually think that mode of reasoning is somewhat unusual, and that's what's distinctive about effective altruism. Yeah, so let's deal with some of this pushback before moving on. I'm sure there's been much more that I haven't noticed, but in the last few days I noticed a Wall Street Journal article that really did not have much substance to it. It seemed to have been written by someone who just doesn't like paying taxes and didn't seem to like Sam Bankman Freed's politics. I didn't really detect any deep arguments against effective altruism there. I guess his one salient point was that creating economic value, i.e. doing business, is generally the best way to help people

at scale and reduce unnecessary suffering. And I guess I think we could easily concede that insofar as that's true, and wherever it happens to be true, it would be very easy for us to say, well then, sure, let business solve our problems. And so there's, you know, and I don't think you would resist that conclusion wherever it's true. I mean, one could certainly argue that someone like Elon Musk, by, you know, building desirable electric cars, has done, if not more good, a different species of good than many forms of environmental activism have to move us away from a climate change catastrophe. So if that's true, let's bracket that, and it's hard to know what exactly is true there, but I certainly have an intuition that there's a place for business,

and even a majority place for business, affect our climate outcome and that it might be greater than charity. He seemed to think that was a real, you know, some kind of coup de gras against effective altruism and even philanthropy in general. But I think that blow just passes right by us, doesn't it? Yeah. So the thing that I'll say that's the grain of truth in that criticism is that in some circumstances, markets can be extremely good ways of allocating resources. So take the problem of people having access to the internet and to computing technology in the United States, at least for people who are middle class or above, then, you know, businesses are dealing with that, like, at least relatively well. There's great incentives to give this product that

are benefiting people, people are willing to pay for it, and there aren't kind of major externalities. However, as has been well known in economics for I think 100 years now, there are very many cases where markets don't produce the ideal social outcome, which they call externalities. So take the case of climate change, which is, you know, an absolute paradigm of this example, where if someone mines coal and burns it, that can be good for them because they're creating energy. But it has this negative effect on third parties that were not kind of privy to that decision. They had no influence over it because it warms the climate and it makes those third parties worse off. And so the standard solution there, and this is endorsed

by economists on the right as well as the left, it's just uncontroversial, is that you want some amount of government involvement there, at least in the form of taxation. So putting a tax on carbon. There's other things you can do as well, but that would be one way, in order to guide market behavior to better outcomes. However, I would go further than this as well and say, you know, there are not only cases of market failure, there are also cases of democratic failure as well, where if you consider the extreme poor who are living in a country with poor political institutions, perhaps even a dictatorship, they do not have democratic say over the policies that rich countries like the UK and the US are enacting. And so they will not be given adequate representation,

neither in the market, nor in political decision making. Or if we look at non-human animals, they have no say at all. Or if we look at future people, those who will be born not now, but in the generations to come, they have no say over, they have no participation in the market. So they have no influence there. They have no say in political decision-making either. And so it's no surprise, I think, that their interests are systematically neglected. And we have no guarantee at all that kind of business as usual for the world would lead to the best outcomes possible for them. Yeah, yeah. Okay, so then there was this article I noticed in this online journal, maybe it's also a print journal, called Current Affairs, which I hesitate to mention because

the editor over there strikes me as someone who is truly mentally unwell, given some of the insane things he's written about me and the length at which he's written them. But this article, not written by him, written by somebody else, raised some points that I think are valid, as well as many that weren't. And the valid ones are around the issue of how we think about expected value and, you know, kind of a probability calculus over various possibilities in the future. And this is really integral to your new book and let's talk about the concept of prioritizing, to whatever degree, the well-being of people who don't yet exist. When people say, well, we should prioritize the future to some significant degree, it's very easy for them to do this by reference to their children, degree, it's very easy for them to do this by reference to their children, you know, young people who, you know, currently exist and have no control over anything. That's easy to see. But when you're talking about the hypothetical billions and trillions of a future humanity

that may thrive or not even exist, depending on how we play our cards at this moment, I think people's moral intuitions begin to fall apart. And the most extreme case of the problem here is, if you try to run any kind of utility function based on a straightforward calculation of probabilities, if you say, well, if I could just reduce the risk that we will be wiped out by some catastrophic event, let's say a global pandemic, if I could reduce that by even one in a million, given the fact that you have trillions upon trillions of potential lives on that side of the balance, well, then that's more important than anything I could do now, including saving the lives of identifiable people, even thousands of identifiable people.

And then that's where people feel like something has simply gone wrong here in how we're thinking about human well-being. So perhaps you can address that. Sure. I'm so glad you brought it up because this is a flaming that often gets mentioned. And I think it's really unfortunate. And I want to push against it in the strongest possible terms, where if it were the case that the argument for concern and serious concern about how we might impact future generations rested on these vanishingly small probabilities of enormous amounts of value, then I wouldn't be writing this book. I would be doing other things.

And I don't think it does. And the reason this framing came up is because in academic articles, Nick Bostrom, who I believe you've had on, he sketched out what are the implications of a very particular moral view, uh, expectational total utilitarianism. And he said, well, if you're taking this view seriously, then that's the conclusion you get. What he wasn't saying is that we should endorse that conclusion. In fact, he, and essentially everyone I know, at least in practice outside of the ivory tower, take that as a reductio rather than an implication. So Nick himself is not a consequentialist even, and wrote other papers on this idea of Pascal's mugging. So it's similar like Pascal's wager,

but just without infinite stakes, where these tiny probabilities of huge amounts of value, look, it just, it doesn't seem like you just do the math and multiply them. Can you actually describe the Pascal's mugging example? Sure, I'm happy to. And so the idea is just, you know, you've had a nice time at the pub, you walk outside, a man comes up to you and says, give me your wallet. You say no. And he says, okay, fair. If you give me the contents of your wallet, let's say that's $100, I tomorrow will come back and I will give you any finite amount of well-being. So just whatever amount of well-being you ask for, I will give you it. And you say, no, I don't,

I just think I'm certain that that's not going to happen. And the man says, come on, you're not certain, are you? Look at my eyes. I've got this pale complexion. I look a bit, you know, it's possible that I'm this, you know, traveler from the future or this alien with like amazing powers, or I'm some sort of god it's possible maybe it's a one in a billion billion billion billion that I will be able to do this but it's at least possible and you take that one in a billion billion billion billion multiply it by a sufficiently large amount of well-being for you that in what decision theorists call expected value which is where you take the probability of an

outcome and the value of that outcome and multiply them together to get the expected value, that expected value is greater than that 100 pounds for you. And then maybe you say, oh yeah, that's okay. I guess that seems correct. You hand over the 100 pounds. And now what Bostrom thinks, and what almost everyone who works in this thinks, is that that's absurd. That if you've got a model, if you've got a decision theory, it's actually not about morality, it's about your theory of rationality, or your decision theory. But if you've got a decision theory that says you should give the wallet to the mugger, then something has gone badly wrong.

And one way that I like to illustrate this, and this applies to the, and this problem gets called the fanaticism problem. And it's a really interesting area of decision theory because I actually think it's the strongest argument against a theory that is otherwise very good, which is expected value theory. But seeing one way of this going wrong is if this argument is used for giving your money to the mugger or for, you know, looking for future amounts of value, even if it were just one in a billion chance that you could change it, well, why are you going after only finite amounts of value? There's a long religious tradition that says you can get infinite amounts of value via heaven. And so really the upshot, if you're taking expected value theory seriously, is to become a missionary to try and ensure that you get into as many

heavens as possible, according to as many different forms of religious belief as possible, and try and convince others to the same, to do the same. And that's where expected value theory, if you're really going with the math and doing it, takes you. And I know of zero people who have done that. Everyone says, look, okay, something, we don't know what, I think this is an incredibly hard area within decision theory. We don't know what the right answer is, but it's not that. And the argument for concern about the long-term future is the fact that these risks are more like the risks of dying in a car crash, and much less like the one-in-a-billion-billion Pascal's mugger kind of risk. Well, admittedly, this is a problem I have not thought much about, but I have a couple

of intuitions here kindling as I listen to you describe it. One is that, obviously, expected value becomes eminently rational within the domain of normal probabilities. I mean, it works in poker, it works in any kind of decision analysis where you're just trying to figure out what you should do under uncertainty, and you're dealing with probabilities of, you know, one-third or 10%. That is how we organize our intuitions about the future in the most systematic way. And I guess there are even corner conditions within the domain of normal probabilities where you basically, this is not a repeated game. You've got one shot to take this risk. And it may be that, you know, risking your life, whatever you put on the other side of

the scale doesn't make sense, except we don't live that way. We risk our lives every time we get into a car or get on a plane or ride a bike or do it. We've all made these trade-offs, whether we've actually done the decision analysis or not and consciously assigned probabilities or not. We just have this vague sense that certain types of risks are acceptable. And I guess my other intuition is that this is somewhat analogous to the problems disgorged by population ethics, where when we talk about things like the repugnant conclusion, and you're talking about very large numbers of very small things, or very small utility, you seem to get these grotesque outcomes ethically. Where do you think this is breaking

down? Or why do you think it is breaking down? So the standard response within decision theory would be to have what's, so it all gets technical quickly. And I apologize if I don't do a good job of explaining the ideas. But the standard response is to have what's called a bounded value function. And it's easiest to see this when it comes to money, let's say. So if the mugger in Pascal's Mugging was saying, okay, I will take £100 off you today, and I will come back tomorrow with any finite amount of money you like that you can spend on yourself.

Let's just assume you're completely self-interested to keep it simple. I could say, look, my probability of you coming back is so low that there is no amount of money that you could give me such that the probability times that amount of money would add up to more expected well-being for me than £100 now. So let's say I think it's one in a billion billion that the mugger will come back. Okay, sure. The mugger could give me, you know, a thousand billion billion dollars. And that would mean more, I would have greater expected financial return. However, I wouldn't have greater expected well-being because the more money I have,

the less it impacts my well-being. And there's some amount, there's only so much that money can buy for me. At some point, it starts making this like vanishingly small difference such that there's a level of well-being such that I can't get like a thousand times greater well-being with any amount more money. And so that's very clear when it comes to financial returns. But you might think the same thing happens with well-being. So supposing I have the best possible life, just like peaks of bliss and enjoyment and amazing relationships and creative insight over and over. And that's for some like very, very long lived life. Perhaps you just imagine that. Now you might think, oh, it's just not

possible to have a life that's a thousand times better than that life we've just imagined. And in fact, I think that's how people do in fact reason. So even if you'd say to someone, look, you can have a 50-50 chance, you either die now or you get to live for as long as you like and choose when you pass away. I think most people don't take that bet. And that suggests that we actually think that perhaps the kind of,

I mean, there's extra issues with kind of, you know, very long lives, but we can reformulate the thought experiment. Either way, I think people just do think there's just an upper bound to how good a life could be. And as soon as you've accepted that, then in the self-interest case, you avoid, you can very readily avoid this kind of Pascal's mugging or Pascal's wager situation. And perhaps you can make the same move when it comes to moral value as well, where you're ultimately saying there's a limit to how good the world or the universe can be. And as soon as you've got that, then you avoid Pascal's mugging, you avoid Pascal's wager.

However, you do get into other problems. and so that's why it's so difficult. Yeah, I mean, the thing that I think is especially novel about what you've been doing and about EA in general, which I always tend to emphasize in this conversation, I think I've brought this up with you many times before, but its novelty has not faded for me. And here's the non-novel part about which someone could be somewhat cynical. There's just this basic recognition that some charities are much more effective than others, and we should want to know which are which and prioritize our resources accordingly. I mean, that's the effective part of effective altruism. That's great, and that's not what has captivated me about this. I think the thing that is most interesting to me is by prioritizing effectiveness in

an unsentimental way, what you've done for me is you've uncoupled the ethics of what we're doing from positive psychology. Really just divorcing the question of how I can do the most good from the question of what I find to be most rewarding about doing good in the world. I'm always tempted to figure out how to marry them more and more, because I think that would be the most motivating and, by definition, the most rewarding way to live. But I'm truly agnostic as to how married they might be. And when I survey my own experience, I notice they're really not married at all. For instance, over at Waking Up, we launched this campaign, which we've called 100 Days of Giving. And so for 100 days, we're giving

$10,000 a day to various charities, and we're having subscribers over at Waking Up decide which charities among the list that you and friends have helped me vet. So every single day, we're giving $10,000 away to some great cause, and we're about 30 days into this campaign. cause. And we're about 30 days into this campaign. And honestly, I think for most of those 30 days, I have not thought about it at all. Right. I mean, literally, like it has not even been a 10 second memory on any given day. And, you know, I consider that on some level a feature rather than a bug. But it is striking to me that in terms of my own

personal well-being and my own sense of connection to the well-being of others, I might actually be more rewarded. In fact, I think I'm almost guaranteed to be more rewarded by simply holding the door open for someone walking into a coffee shop later this afternoon and just having a moment of, you know, a smile, right? Like that absolutely trivial moment of good feeling will be more salient to me than the $10,000 we're giving today and tomorrow and the next day and onward, you know, for months. So I'm just wondering what you do with that. I mean, it's a bug and a feature. It's a feature because I really do think

this is how you do the most good. You automate these things, you decide rationally what you want to do, and you put that in place, and then it's no longer vulnerable to the vagaries of your own moral intuitions or you getting captivated by some other bright, shiny object

and forgetting about all the good you intended to do. But I do hold out some hope that we can actually draw our feelings of well-being more and more directly from the actual good we're accomplishing in the world. And I just think there's more to think about on the front of how to accomplish that. Yeah, you're exactly right. And it's an old chestnut, but it's true that our psychology, human psychology, did not evolve to respond to the modern world. It evolved for the ancestral environment, our community, the people that we could affect amounted to the dozens of people, perhaps. Certainly the idea that you could impact someone on the other side of the world or in generations to come, that was not true in the

way it's true today. And so we have a radical misalignment between our the moral sentiments that we have evolved and the moral reality that we face today and so one thing either there's this project that effective altruism as a community has tried to embark on to some extent which is just trying to align those two things a little bit better and it's tough because we're not fundamentally aligned. But the ideal circumstance is where you are part of a community where you can get the reward and support and reassurance for doing the thing that actually helps other people the most, rather than the one where you get that immediate kind of warm glow. And that's not to say you shouldn't do those things too. I certainly do. And I think it's part of living a good, well-rounded life. But it's just a fact that there are people

in very poor countries who are dying of easily preventable diseases. We can take action to help them. There are animals suffering greatly on factory farms. We can take action to alleviate that suffering. There are risks that are maybe going to affect the entire fate of the planet, as well as killing billions of people in the present generation. Ah, like that is not something that my monkey brain is equipped to deal with. And so, you know, we're just, we've got to try as best we can. And that's something, at least for me, where having a community of people who share this idea, who I can talk these ideas through with, and who can, if needed, if I find these things hard, provide support and reassurance, that's at least a partial solution to this problem to get my monkey brain to care more

about the things that really matter. Well, so are there any other problems that you've noticed or that have been raised by EA's critics that you think we should touch on before we jump into the book and this discussion of long-termism and existential risk? Sure. I think the number one thing that is in some sense a problem, but I think is just kind of where the debate should be, is just what follows from long-termism, where the issue, the thoughts that I want to promote and get out into the world are that future people count. There could be enormous numbers of them. And we really can and in fact are impacting how their lives go. And then there's the second question of, well, what do we do about

that? What's the top priorities for the world today? Is that the standard environmentalist answer of mitigate climate change, reduce resource depletion, reduce species loss? Is it the classic existential risk answer of reduce the risk of misaligned AI, reduce the risk of a worst case pandemic? Is it the idea of just promoting better values? Is it something that I think you're sympathetic to and others like Tyler Coe and Apatik Collison are sympathetic to, which is just generally make the world saner and more well-functioning because, you know, predicting the future is so hard. Or is it perhaps about just trying to put the next generation into a situation such that they can make a better and more informed decision and have at least the option to decide

kind of how their future will go? These are extremely hard questions. decide kind of how their future will go. These are extremely hard questions. And, you know, I actively want to see an enormous amount of debate on what follows from that. You know, perhaps a thousand times as much work as has gone into it, I would love to see discussing these issues. And there's an enormous, these are hard, there's an enormous amount to say. And so that's where I think the kind of real meat of debate lies. Okay, so let's talk about the book, because you spoke a couple of minutes ago about how we are misaligned in psychological terms by virtue of imperfect evolution to think about problems at scale, and that's certainly true, but we have made impressive progress with respect to scaling our concern about our fellow human beings

over physical distance. Distance with respect to space, we kind of understand ethically now. I mean, Peter Singer has done a lot of work on this topic, urging us to expand the circle of our moral concern. And I think something like 100% of the people listening to us will understand that you can't be a good person and say that you only care about people who live within a thousand miles of your home, right? I mean, that's just not compatible with life in the 21st century. Although there's some discounting function, no doubt, and I think normal psychology would cause us to care more about family and

friends than about perfect strangers, and certainly perfect strangers in faraway cultures for which we have no affinity, etc. And we can, I think we've discussed this in previous conversations, just what might be normative there, whether we actually would want perfect indifference and dispassion with respect to all those variables, or if there's some hierarchy of concern that's still appropriate. And I think it probably is. But the topic of today's conversation and of your book is just how abysmally we have failed to generate strong intuitions about the moral responsibility we have and the moral opportunity we have to safeguard the future. So perhaps you can, with that set up, you can give me the elevator synopsis of the new book.

Sure. So like I said, this book is about long-termism, about just taking seriously the future that's ahead of us, taking seriously the moral value that future generations have, and then also taking seriously the fact that what we do as a society really will impact the very long term. And I think there are two categories of things that impact the very long term, where there's things that could just end civilization altogether. And this concern really got on society's radar with the advent of nuclear weapons. I think an all-out nuclear war, I think unlikely to kill literally everybody, but it could result in complete, you know, it would result in hundreds of millions, maybe even billions dead, unimaginable tragedy, which I think would make the world worse, not just for those people, but for the extremely long time, and could result in

the collapse of civilization too. At the more extreme level, there's worries about, in particular, engineered pandemics. And this is a risk that we've been worrying about for many years now, This is a risk that we've been worrying about for many years now, where pandemics occur occasionally and are extremely bad. And so society doesn't prepare for them in the way they should. I think the risk is going to get much worse over time, because not only do we have to contend with natural pandemics, we will also have to contend with pandemics from viruses that have been enhanced to have more destructive properties, or even just created de novo, such as in a bioweapons program. pandemics from viruses that have been enhanced to have more destructive properties, or even just created de novo, such as in a bioweapons program. This is a very scary prospect, and current estimates put the risk of all-out catastrophe, 95% dead, something like 1%, essentially. A second kind of way, set of events that could impact the very long term,

way, set of events that could impact the very long term are things that don't impact the length of civilization. That is, it wouldn't kill us off early, but does affect the value of that civilization, kind of civilization's quality of life, as it were. And in What We Are the Future, I focus in particular on values, where we're used to values changing an enormous amount over time. But that's actually something that I think is contingent about our world. And I think there are ways, especially via technological and political developments, that that change could slow. And in the worst case, our future could be one of perpetual totalitarian dictatorship. That would be very scary indeed. That would be a loss of almost all value that we could have and an existential risk itself, even if that civilization were very long lived. And so what can we do? And things vary in terms

of how tractable they are, like how much there is actually to do. But there are many things I think we can do. So on the ensuring we have a future side of things, I actually think that there's very concrete actions we can take to reduce the risk of the next pandemic, such as technology to monitor wastewater for new pathogens, such as certain, there's this new form of lighting called Far UVC that I'm extremely excited about. If we can get the cost down enough,

then we can just be sterilizing rooms in an ongoing way, in a way that's not harmful for human health. Or on the values kind of side of things, well, we can be pushing for people to be more altruistic, more reasonable in the way they think about model matters, perhaps more impartial to taking into seriously the interests of all sentient beings, whether they're human or animal or other, or whether they're now or in the future. And we can be kind of guarding against the lie, like certain, you know, authoritarian tendencies that I think are very common, and we have seen in history over and over again, and that I think are quite scary from a long-term perspective. And then lastly, we can just start reasoning more about this. I said earlier that,

you know, I don't know, I'm not like super confident about what we should be doing, but certainly we can be doing more investigation, building a movement that takes seriously, oh yeah, like there's this enormous potential future ahead of us, and we can do an enormous amount of good to make it better. Yeah, I want to talk about value lock-in and some of the issues there. But before we get there, I think we should linger on what makes thinking about the future so difficult. I think you say somewhere in the book that, both with respect to distance in space and distance in time, I think you say something like we mistake distance for unreality, right? Like the fact that something is far away from us

makes part of our brain suspect that it may not quite exist. And this is, again, I think this has changed fairly categorically with respect to space. You just can't really imagine that the people suffering over in Yemen don't exist. But it's much harder to assimilate this new intuition with respect to time, because in this case, we are talking about people who don't actually exist. There'll be some number of thousands of people, probably 3,000 to 5,000 people or so, born over the course of our having this conversation, globally speaking. I think the math is somewhere around there. And so it's easy to argue, okay, these are people who are coming into this world for which they have no responsibility. And we, the present generation or generations, are entirely responsible for the quality of the world they inherit and the quality of the lives they begin to lead. And I would say we don't, collectively know, collectively, we don't think nearly enough about that responsibility. But when you're talking about people who may or may not be born

in some distant future, we don't think about these people at all. And I would, I suspect that most people wouldn't see any problem with that because it's just, when you think about what a crime it would be for us to foreclose the future in some way, you know, if we all die in our sleep tonight painlessly and there is no human future, what's bad about that? joy and happiness and creativity that exists in potentia on that side of the ledger. But there are no victims deprived of all of that goodness because they don't exist, right? So this is the very essence of a victimless crime. And I think you and I have touched that issue in a previous conversation. But it's just hard to think about the far future, much less prioritize it. And I'm just wondering if there's something we can do to shore up our intuitions here. I guess the first point I would make is that the only reason why these people are hypothetical

is because you, the listener, are holding out some possibility that they may not exist, when you ask yourself, what would cause subsequent future generations of people not to exist, perhaps there's some other version, but the only thing I can imagine is some catastrophe of the sort that you have begun sketching out, right? Some pandemic, some asteroid impact, some nuclear war, some combination of those things that destroy everything for everyone, which is to say some absolutely horrific cataclysm, which we should be highly motivated to avoid, right? So if we successfully avoid the worst possible outcome for ourselves and our children and our grandchildren, the future people we're talking about are guaranteed to exist. It's only time shifted. It's not, in fact, in question. And the

only thing that would put it in question is some horrific outcome that we are responsible for foreseeing and avoiding. And really, there's no one else to do the job. Absolutely. I mean, so, And really, there's no one else to do the job. Absolutely. I mean, so yeah, there's this general issue where why do we care about people on the other side of the planet much more now than we did, let's say, 100 years ago? And it's clear that a big part of it is that now we can see people on the other side of the world. We get information about them in a way that we never used to, where literally there can

be a live stream video of someone in Nigeria or Pakistan or India or any other country around the world. So that, you know, those people's lives become vivid to us in a way that's just not possible for future generations. And that makes this challenge just so deep, in fact, to get people to care. If there were news from the future, you know, the newspaper of tomorrow, I think we wouldn't be in such a bad state. You know, we could report on what the world is like in the year 2100 or 2200. And I think people would care. But that's not the news we get. It's all abstract. And that's really tough. The thing I'll say is, so, you know, you mentioned the idea of

some catastrophe just painlessly kills all of us in our sleep. And, you know, what would that be a tragedy? And I mean, I think yes. And I argue in the book at length that it would be a tragedy, both because I think that we should expect the future to be on balance good, and secondly because I think the loss of lives that are at least sufficiently good, and certainly the loss of all the moral and artistic and technological progress that we might see in the future, that those are great losses. But I want to emphasize the case for long-termism does not rely on that. You could think that everyone dying painlessly in their sleep would not be a bad thing, morally speaking. Nonetheless, a future of perpetual totalitarian dictatorship really would be compared to a wonderful, free, thriving, and liberal future

where different people can pursue their visions of the good. And we've eradicated issues that plague today like injustice and poverty. I just think you really don't need to engage in much moral philosophy to say that if we have a choice between one or the other, then it's better and much better to think about the thriving, flourishing future than this dystopian one. Yeah, no, that's a great point. Forget about the thought experiments. It should be pretty easy to acknowledge that there's a difference between having to live under the thousand-year Reich and, about the significance of the future? I mean, I guess I'm wondering if you've thought at all about how just, because again, this is, it's surprising that I think we think about this almost not at all, and yet our assumptions about

it do a lot of psychological work for us. I mean, like I really, you know, I mean, this is my job to have conversations like this, and I'm surprised at how little I think about the far future, right? In any kind of significant way. And yet, if I imagine how I would feel if I knew that it wouldn't exist, right? I mean, if we just figured out that, oh, actually, the whole story ends in 80 years, right? So we're unlikely to be around to suffer much, or we're going to be out of the game pretty soon. And people haven't had kids, don't need to worry about them. And so the basic picture is just the career of the species is radically foreclosed at some point inside of a century. What would that do to everything? And how would that sap the meaning from what we do? And it's just an interesting question, which I haven't thought

much about. But do you have any intuitions about that? It's a great thought experiment, and it's represented in this film, Children of Men, where the global population becomes sterile. And it's discussed at length in the writings of philosopher Samuel Scheffler in a book called Death and the Afterlife. And he argues that if it were the case that just, yeah, there's no world at all, let's say, past our generation or even our children's generation, it would sap enormous amounts of, in fact, perhaps most meaning from our lives. This is actually a point made by John Stuart Mill in the end of the 19th century, where he briefly had this phase of being concerned by future generations, arguing actually that it means they should be keeping coal in the ground, which I think wasn't quite the right, you know, it's maybe made sense given their views at the time or something,

that their views were incorrect. But he has this beautiful speech to Parliament, and he has this wonderful line, why should we care about posterity? After all, what has posterity ever done for us? And he goes on to argue that actually, like, posterity has done an enormous amount for us because it gives our projects meaning. Where, you know, we build, we plant trees that future generations will sit under.

We build cathedrals that are like a testament to our time and can be recognized by future generations. When we engage in activities like moral progress and political progress and scientific and technological progress, part of how we have meaning in doing those things is because it's like we're part of this grand relay race where we have taken the baton from previous generations, the things they did badly and the things they did well.

And we can hand that baton on to future generations, making the world a little bit better, giving them like, you know, a little bit of a higher platform to stand on. And yeah, I think this is just a very widely shared view. And I think it kind of resonates with me intuitively too, where if I found out that everything was going to end in 50 years, I wouldn't be like, oh, well, I guess I'll just, you know, focus on issues that just concern the near term then. I would, you know, I would be devastated, even if that didn't personally affect anyone I knew or, you know, perhaps it's not 50 years, perhaps it's 200 years.

Nonetheless, I would still, I would be absolutely gutted. Yeah, it's really interesting because it's not something that I've ever been aware of, but it is an integral piece of how I think about what we're doing here, which is when you think about human progress, you think about all the good things we do and all the bad things we can avoid. The idea that that is all going to be brought up short in a few short years, right? I mean, we just think of like, well, what's the point of building increasingly wonderful and powerful and benign technology or creating beautiful art or literature or just making

progress on any front philosophically? on any front, philosophically. I mean, beyond just securing your day-to-day comfort and basic happiness, there's something forward-leaning about all of that. It's a check that, in some sense, never gets cashed or fully cashed, or it just gets continually cashed in the future, if not by our future selves, by others to which we're connected. And the idea that we could ever know that there's just a few more years of this and then it's all over, I mean, we all know that personally with respect to our own mortality. But there is something about the open-endedness of the whole project for others. You know, even if you don't have kids, that changes. Again, I really don't even have my intuitions around this are so unformed, but it really does

feel like it changes something about day to day life. If the rug were truly pulled out from under us in a way that we had to acknowledge, you know, leaving everything else intact, right? I mean, all the buildings are still standing. We've got, you know, there's still a concert Saturday night in the same auditorium and you have tickets, but you knew the world was going to end and you were just going to get a collective dial tone at some point in the near future. You know, I don't know to what degree, but it would change everything to some degree. Yeah. So in What We Are With The Future,

I actually start and end the book on the same note, which is a note of gratitude, where the book is dedicated to my parents, Maya and Robin, and their parents, Tom and Nina and Frank and Daphne, and so on, all the way back, all the history that led us here. And at the very end of the book, in the very last page, there's, in the print version, there's this QR code. And if you scan it, it takes you to a little short story I wrote. I'm not claiming it's any good, but I did it because I think it's

so hard to visualize the future, especially to visualize how good the future could be. And in the end of that short story, one of the characters reflects on everything that led, and it's this depiction of this, I'm not saying it's the best possible future, but at least a good one. And one of the characters just reflects on just everything that led to her life today all of the hardship the ways in the past in which people endured sickness and suffering and rides on the subway packed in with other people who are you know miserable on their commute and just being deeply thankful for all of that and that's certainly how how I feel, where, you know, the past has given us,

past generations have given us this mixed bag. They've given us many ways in which the world is broken, many ways in which we still see enormous suffering. But they've also given us wonderful, good things too. I live in this like liberal democratic country that is exceptionally unusual by historical standards. I don't need to suffer pain when I engage in a surgery. I don't need to risk for my life from diseases like syphilis or tuberculosis. I can see the kind of, just on my phone, I can just see the most wonderful art produced

in the entire world at any time. I can see the world I can go traveling in, like India, Sri Lanka, Ethiopia. Wow, I'm just so grateful for all that's been done. The very least that I can do is try and take that mixed package of goods and goods and bads and try and pass it to the next generation. Try and ensure that there even are future generations that I can pass it to, but also to make it better. And so that's kind of, yeah. Normally I trade pass it to, but also to make it better. Normally, I trade in these logical, rational arguments, but at least in terms of what motivates me on a gut level,

that's a significant part of it. So we have something like 500 million to 1.3 billion good years left on this planet if the behavior of our sun is in any indication. So the challenge here is to not screw it up. And, you know, we can leave aside the possibility of reaching for the stars, which we will almost certainly do if we don't screw things up and, you know, spreading out across the galaxy. But even on this lowly planet, we have just an extraordinary amount of time to make things better and better. And we're just at the point where we can get our hands around the kinds of things that could ruin everything, not of our own making, like, you know, asteroid impacts and naturally spawned pandemics. And there's clear ways in which our time is unusual. I think at one point in the book,

you discuss how for virtually all of the past, we were culturally isolated from one another, and we may one day be again culturally isolated if we expand out across the galaxy, which is to say that there will be a time in the future where human communities grow far enough apart that what they do has really no implication for what anyone else does outside of their community. And that was true of most of human history until recent centuries. I think there's a more fundamental claim in your book, which is that there's good reason to believe that we are at something like a hinge moment in history where the things we do and don't do have an outsized effect over the trajectory of human life in the future. And, you know And there are things that have come up briefly so far

like value lock-in, which are related to that, which we'll talk about. But what do you think about this question of whether or not we're living at a hinge moment and how we can know and how we may or may not be mistaken about that? I think it's clear that we're living at an extremely unusual time that gives a good argument for thinking that we're living at an unusually impactful time too. And I'm not going

to claim we're living at the most unusual time or impactful time in history. Perhaps the next century will be even more influential again. But it's certainly very influential. And the core reason for this is just kind of think about the tree of technology that civilization can kind of go through from the very first stage, there's flint axes and spear throwers and fire and the wheel. And then nowadays, there's maybe it's fusion technology and advanced machine learning and biotechnology that are current developments and at some point there will be future technology that perhaps we've not even imagined yet here's the question just relative to both the past and the future how quickly are we moving through that technology to the

and i think there's very good arguments for thinking that it's much faster. We are moving much faster through that technology tree than for almost all of human history and almost all of the future if we don't go extinct in the near term. And I think that's clear. So economists measure this generally with economic growth, where at least a significant fraction of economic growth comes from technological progress. And the economy is currently growing far, far faster than it has done for almost all of human history, where today the economy doubles every 20 years or so. So it's growing around 2% or so, and compounding that, it gives you a doubling time of something like 25 years or thereabouts. Exactly. Yeah, 2 two three percent when we were hunter-gatherers growth was

very close to zero as farmers growth was more like 0.1 percent so a doubling every a few hundred years or so and so relative to the past we are speeding through technological development much much faster than before, you know, for most of human history before. I mean, honestly, I say before the scientific industrial revolution, really, it's before about the year 1900, because there was a slower kind of ramp up. But then secondly, I think we're speeding through that progress much faster than for most of the future too. And the argument for this is that we'll just assume we keep growing at about two percent per year for another 10 000 years how much growth would there have to be in that case well it turns

out because of the power of compound growth that it would mean that for every atom within reach that is within 10,000 light years, there would be, I think, it's 100 billion trillion current civilizations worth of output. And now, you know, maybe that's possible. I think it's extremely unlikely. this level of progress, of economic and technological progress, we would get very close to basically discovering almost everything that is to discover on the order of a few thousand years. However, as you said, there are hundreds of millions of years to go potentially. And so that means that, as I said, we're moving particularly quickly through the tree of technologies. moving particularly quickly through the tree of technologies. And every technology comes with its distribution of powers and capabilities and benefits and disadvantages. So if we take the ability to harness energy from the atom, well, that gave us clean energy. It gave fire

nuclear power. That's an enormous benefit to society. It also gave us nuclear weapons that could threaten wars more destructive than any we've ever seen in history. And I think future technologies, such as biotechnology that can allow us to create new viruses, or advanced artificial intelligence, that each one of these give us this new distribution of powers. And some of those powers can have extraordinarily long-lasting impacts, such as the power to destroy civilization, or such as the power to lock in a particular narrow ideology for all time. And that's why the world today is at such a crucial moment. Yeah, the principle of compounding has other relevance here when it comes to this question

of just how influential a moment this might be in history. When you just look at how effects of any kind compound over time, it does cast back upon the present kind of an inordinate responsibility to safeguard the future, because we have this enormous potential influence. I mean, just the slightest deflection of trajectory now potentially has massive implications, you know, 20, 30, 40 years from now. And it's obviously hard to do the math on these things. It's hard to predict the full set of consequences from any change we make. But if you just look at what it means to steadily diminish the probability of a full-scale nuclear war year after year, even just by 2% a year, if we could just get that virtuous compounding to happen, roll that same logic to every other problem you care about, it's very easy to see that

getting our act together, however incrementally now on all these fronts, matters enormously in the not-too-distant future. Absolutely. I mean, my colleague Toby Ordon, his wonderful book, The Precipice, just suggests that, well, what's happening now is that our technology is advancing more rapidly than our wisdom. And it's a beautiful beautiful idea and what we're trying to do is accelerate wisdom you know we're trying to just make us a little bit wiser a little bit more cognizant of you know there's most people fall into one of two camps there are the people who are really into technology the kind of stereotype of the silicon valley people and they think

technology is great and we just need to keep advancing it. And then there are people who are not really into technology and the people who are very skeptical of Silicon Valley and worry about what technology could be doing to society. And I think I just really want people to inhabit both frames of mind at once. Technology has benefits and costs. Some technologies are just like anesthetic. I'm like, oh yeah, that's just really good. Let's have more of these like, you know, no downside, unashamedly good technologies. Let's speed them up faster. For technologies that are very dual use, or just extremely powerful, and we don't know how to handle them. Well, let's then be careful and considerate about how we're developing these technologies, have good norms and regulations

in place so that we don't basically just walk backwards into a chasm where we wreak global disaster because we're not even paying attention to the impacts that technologies that are just around the corner will have because of this reason of out of sight and out of mind because we're barely thinking about you know we barely think about like the end of the next year certainly not the end or the end of the next political cycle let alone the decades ahead that could bring about radical changes and how they might impact the centuries ahead. Well, simply by uttering the phrase political cycle, you have referenced what I consider a major problem here, which is we have a politics that is almost by

definition a machine of short-termism, whereas it whereas taking a long-term view is antithetical to the whole project of becoming president and staying president and then raising money for your next campaign. I mean, you're just not, to speak specifically of America at this point, I don't know that it's much better in any other country, you're not incentivized to think about the far future and to figure out how to help all those people whose names you will never know and who will never vote for you, much less give you money.

It's just not in the cards, right? And so one thing that has to be a fairly high priority is to figure out how to engineer a system of political and economic incentives such that we more effortlessly take the future into account. Absolutely. And honestly, it's extremely tough. So I've worked on this topic. I have an article on long-termist political institutions with another philosopher, Tyler John. And there are some things you can do, such as, you know, some countries have an ombudsperson to represent future generations.

I'm particularly attracted to the idea of at least an advisory, at least as an advisory body, a kind of citizens' assembly, so a randomly selected group of people from the populace who are explicitly given the task of representing the interests of future generations and then making recommendations about what the government could do differently in order to make the future go better. I think we can also build in just more information, so more think tanks or quasi-governmental organizations that have the task of really trying to project not just the next few years, but the next few decades, even the next few centuries.

These are some changes we can make. However, the fundamental problem is that future generations are being impacted without representation. they do not get a vote. And the fundamental problem is we cannot give them a vote. There's no way they can represent themselves. And what do we do about that? Honestly, I think it's going to be very hard to make major progress unless there's some significant cultural change. So if we imagine a world where governments are responding to the views and preferences and ideals of their populace, of the electorate, and that electorate cares about the long-term, well, that's a situation where we can align political incentives with the long-term good.

And so that's what I think we really need to get to. That's what I think we really need to get to. It's just amazing how hard it is, even for normal people who have children, to prioritize the next decade. A decade where they have every expectation of being alive and suffering the consequences of our present actions, and they have certainly every expectation that their children will be alive in those decades. When you look at a problem like climate change or anything else that is slowish moving and has various features like, you know, being in part a tragedy of the commons, running counter to any individual's personal economic incentives, etc. It's just, it all falls by the wayside, even when you're not talking about hypothetical people who you will never meet, right? I mean, it's just, there's something so hard about focusing on the big picture. And so, yeah, it does, I mean, when you talk about cultural talk about cultural change, it's hard to see the thing

that would be so sweeping that it would capture more or less everybody all at once. But perhaps that's not even the right goal. I mean, maybe there are only something like 5,000 people whose minds would have to change to fundamentally redirect human energy. I mean, this is an elitist claim and argument, but I suspect that something like that might also be true. Yeah, I think I'm more optimistic on this in general, perhaps, than you are, where, yeah, in the course of doing research for What We Owe the Future, I, you know, dug a lot into the history of various social and moral movements, such as environmentalism, feminism, animal welfare movement, and in particular, the abolition of slavery. And one thing I think it's

easy to underestimate is just how weird these ideas were in the past, at least in particular for the kind of political elite, or at least the people who had power, where you don't have to go that long ago. Let's just focus on the environmentalist movement. I think there's many analogies here where the environment can't represent itself. It has an advantage that you can see the environment, you can relate to it. But 150 years ago, concern for the environment was really not on the table. Maybe you had poets, Williams Wordsworth, starting to cultivate love for nature, but there just really wasn't anything like a movement. Whereas now, it is just a part of common sense morality, at least in many countries around the world, that is, by historical

standards, that is amazingly rapid model change. And so at least if we're able to say, look, we're not going to be able to make this change to convince everyone of this long-term outlook, to convince everyone of having concern for future generations. We're not going to be able to convince them within the next few years or even two decades. But when we look like 100 years out, maybe we can. Maybe actually we really can affect that sort of sweeping moral change where concern for future generations is just on the same level as concern for other people in your community or people of a different ethnicity than you.

I'm holding out hope. I'm holding out hope that those sirens aren't people coming to arrest you for your thought crimes. I'm sorry. This is in New York. Are you recording this podcast while robbing a bank? Look, it's for the greater good, right? The money can go further.

So that's a, you know, don't do that. Just in case my followers take me too literally. Let's go on record. We're against bank robbery. Yeah. At least currently. I'm with that. I won't say anything about value lock-in on that point.

Let's talk about this issue of value lock-in and what you call moments of plasticity and trajectory changes. How do you think about, and you might want to define some of those terms, but what is the issue and opportunity here? Sure. So at the moment, there's a wide diversity of moral views in the world and rapid moral change. So the gay rights movement can build in power and momentum in the 60s and 70s, and then you get legalization of gay marriage a few decades later.

Clear example of positive moral change. And so we're very used to that. We think it's kind of just part of reality. We wouldn't expect that that might change in the future, such that moral change might come to an end. The idea of value lock-in is taken seriously that maybe it will, actually. Maybe moral progress will stall. And so value lock-in is the idea that some particular set of, some particular ideology or moral view, or kind of narrow set of ideologies or moral views, could become globally dominant and persist for an extremely long time. And I'm worried about that. I think that's a way in which the long-term future could go that could just lead to enormous loss of value, could make the future dystopia rather than a positive one.

And, you know, it's easiest to see this by starting off with maybe the most extreme case, and then kind of thinking about ways you might not be quite that extreme, but close. And the most extreme case is just, imagine a different history where it wasn't, you know, the US and UK and liberal countries that won World War II. And instead it was Stalinist USSR,

or let's say the Nazis who won. Now they really aimed ultimately at global domination. They wanted to create a thousand year empire. If that had happened and they had a very narrow and morally abominable ideology, if they had succeeded, would that have been able to persist forever, or at least for an extremely long time? And I think actually the answer is yes, where ideologies in general and social systems can certainly persist for centuries or even thousands of years. So the major world religions are thousands of years old. But technology, I think over time, will give us greater and greater power to control the values of the population that lives in the world

and the values in the future as well. Where at the point in time where we develop the ability to like stop aging such that people's lifespans could last for much, much longer. Well, then a single dictator could, you know, it's not confined to the lifespan of it. The rule is not confined to the lifespan of a single individual, 70 years, but instead would be thousands of years or however long that ruler would last. Even more extreme, I think, is the point of time at which the ruling beings are digital rather than biological. And again, this feels like sci-fi but you know we're talking about i actually think many people i respect greatly think that moment is coming in

in a few decades time but let's just say it's like you know plausible within this century or centuries well consider what the environment is like for a ruler who is a digital being. Well, they don't die. They are in principle immortal. Software can replicate itself, and any piece of machinery would wear out, but the AI systems themselves could persist forever. So one of the main reasons why we get moral change over time, namely that people die and

are replaced by people who have slightly different views, that would be undermined. I think in this kind of scenario, other causes of change would be undermined too, or taken away. So I think the fact that we have competition between different model ideas, because we have this diversity of model ideas. Well, if the Nazis had won World War II, if they had established a world government, you know, again, I'm not claiming this is likely or possible, but imagine the counterfactual history. Well, then we would lose out on that competition of ideas.

There wouldn't be that pressure for model ideas to change. And then finally, it's just potentially a changing environment. Like upsets happen, you know, civilizations fall apart and so on. But I think technology in the future could allow for greater and greater stability. So again, I think AI is particularly worrying here where, you know, again, we're imagining the leaders of the Nazis controlling the world. Okay, well, one way in which dictatorships end is insurrection, whether that's by the army or by the people. If your army is automated, if you have kind of robots policing the streets, then

that dynamic would end, similarly with your police force. And again, all of this seems like sci-fi, but I think we should be thinking about it. We should be taking it seriously, because it really could come sooner than we think. And the technologies we have today would have looked like wild sci-fi just a century ago. Yeah, well, maybe we'll close out with a conversation about AI, because I share your concern here, and I share your interest in elucidating the bad intuitions that make this not a concern for so many other people. So let's get there in a second. But remind me, what is the lock-in paradox that you talk about in the book? Sure. So the lock-in paradox, it's like the liberal paradox of tolerance. But I push in

the book that we don't want to lock in any particular values right now, because probably all of the values we're used to are abominable. You know, I think we're like Plato and Aristotle arguing about the good life, and we've not realized the fact that we all own slaves is, you know, maybe morally problematic. So I think that there's enormous amount of model progress yet to come, and we want to ensure that we guarantee that progress and so even if you know i think motivation of thinking oh well it's you know 21st early 21st century western liberal values that's the best thing and we should just ensure they happen forever i'm like no we need progress the lock-in paradox however is that if we wanted to create a world where we do get a sustained period of reflection and moral progress, that does mean we'll need some restraints. free debate commitment to tolerance of opposing moral views restrictions on ways of gaining power

that aren't via making good arguments and moving people over to your side kind of in the same way that like you know the u.s constitution at least aspirationally is trying to bring about this like liberal society where many different world views can coexist and where you can make moral progress but in order to have that and not fall into a dictatorial the dictatorship you needed to have these restraints on any individual's power and so i think we we may well need something like that for the whole world where we stop any particular ideology or just set of people from gaining too much power such that they can just control the world so that we can just have continued reflection, insight, empathy, and moral progress. statement of this, which is something that allows for the kind of incrementalism you've just sketched out and the error correction that I think is necessary for any progress, including ethical progress, but which locks in the methodology by which one would safeguard the principle of error correction and sensitivity to

the open-endedness of the exploration in ethical space, right? I mean, it's like, you know, I think there's probably, whether you define it as consequentialist or not, I think it's probably less important. I mean, the consequentialism, as we know, has its wrinkles. But I think there's an almost axiomatic claim about the primacy of safeguarding the well-being of conscious creatures at the bottom of, I think, any sane moral enterprise right now. It's not how you think about well-being, how you aspire to measure it or quantify it, how you imagine aggregating these quantities. There are many devils in all those details, but the basic claim that, you know, you have to, at bottom, care about the happiness and suffering of conscious creatures and marry that to a truly open-ended conception of, and, you know, perpetually refined and

refinable conception of what well-being is, all things considered, you know, and that's where all the discussion remains to be done. I think those are the bootstraps in some form that we have to pull ourselves up by. Yeah, so it is a very notable fact that I think basically all plausible moral views that are on the table see well-being as one part of the good. Perhaps they think other things are good too, art, flourishing natural environment, and so on. But at least, you know, happiness, broadly construed, flourishing life, avoidance of suffering, those are good things. And... I would just, well, just not to be too pedantic here, but in my brain, all of that collapses to just a fuller understanding of what well-being is. I don't think any, you know, reasonable

definition of well-being would exclude the things you just mentioned. And the cash value of the things you just mentioned has to be at some level how they impact the actual or potential experiences of conscious creatures. This is an argument I've made somewhere. If you're going to tell me you've got something locked in a box that is really important, it's really valuable, it's important that we consider its fate in everything we do, but this thing in the box cannot, will not affect the experience of any conscious creature now or in the future, right? Well, then I just think that's a a contradiction i mean we what we what we mean by value is something that could be actually or potentially valued by someone

somewhere great so i think we differ a little bit on this where i mean i also think like my best guess is that value is just a property of conscious experiences. And, you know, art and knowledge and the natural environment are all good things insofar as they impact the well-being of conscious experiences. But that's not something I would want to bake in. I'm not so certain in it. You know, there are other views where the satisfaction of carefully considered preferences is of positive value, even if that doesn't go via the change in any conscious experience. And other people do have the view that... But so again, that's the phrase I'm actually not familiar with. When I hear you say

satisfaction of anything, including preferences, aren't we talking about conscious experience, actual or potential? So on this view, it's no. So suppose that Aristotle wanted to be famous, and he wasn't very famous during his time. I'm actually not sure. Let's say that he was only much less famous than he is now. Does the fact that we are talking about Aristotle now increasing his fame, does that increase his well-being?

And it's obviously not impacting his consciousness. I'm prepared to answer that question. I'm sure you're prepared to say no. And honestly, and my best guess view is completely agreeing with you that anything we do to talk about Aristotle now, we could say he smells and we hate him. It doesn't change his well-being at all. However, that's not true. Other people disagree, and I think other smart people disagree. And so I certainly wouldn't want to have, wouldn't want to say, look, we figured that out. I would want to say, look, no, we've got plenty of time to debate this. I think I'm going to win the debate. Maybe the other person thinks they're going to win the

debate, but we've got all the time in the world to really try and figure this out. And I would want to be open to the possibility that I'm badly wrong. I guess we are kind of badly wrong on this. Well, that's interesting. I would love to debate that. So maybe as a sidebar, you and I can figure out who and when and where. Absolutely. Always happy to talk about, yeah. Okay, well, so let's talk about AI because it is, in some sense, the apotheosis of many of these concerns.

And I agree that should anything like a malevolent AGI be built or be put in the service of malevolent people, the prospect of value lock-in and Orwellian totalitarianism, it just goes way, way up. And that's worth worrying about, whatever you think about the importance of future generations. But there are several stumbling blocks on the path to taking these concerns about AI seriously. The first is, and this really is the first knee-jerk reaction of people in the field who don't take it seriously, it's the claim that we're nowhere near doing this, that our language models are still pretty dumb, even though they can produce some interesting texts at this point. There's certainly no concern that there could be anything like an

intelligence explosion in any of the machine learning algorithms we're currently running. AlphaZero, for all of its amazing work, is still not going to get away from us, and it's hard to envision what could allow it to. I know that there are people on our side of this debate who have given a kind of probability distribution over the future, giving it a 10% chance it might happen in 20 years and a 50% chance it'll happen within 50 years, etc. But in my mind, there's no need to even pretend to know when this is going to happen. The only thing you need to acknowledge is that not much needs to be true to make it guaranteed to happen. There are really two assumptions one needs to make. One assumption is we will continue to make progress in building intelligent machines at whatever rate, unless something terrible happens.

This brings us back to the collapse of civilization. The only thing that would cause us to stop making progress in building intelligent machines, given how valuable intelligence is, is something truly awful that renders some generation of future humans just unable to improve hardware and software. So there's that. I mean, so barring catastrophe, we're going to continue to make progress. And then the only other assumption you need is that intelligence is substrate independent, right? That it doesn't require the wetware of a biological brain. It can be instantiated in silico.

And we already know that's true, given the piecemeal AI we've already built. And we already know that you can build a calculator that's better than any human brain calculator. And you can do that in silicon. And there's just no reason to think that the input-output properties of a complex information processing system are magically transformed the moment you make that system out of meat. And so, again, given any rate of progress, and given the assumption of substrate independence, eventually, and again, the time horizon can be left totally unspecified, you know, five years

or 500 years, eventually we will be in the presence of intelligent machines that are far more intelligent and competent and powerful than we are. And the only other point of confusion that I've detected here again and again and again, which can be easily left to one side, is the question of consciousness, right? Will these machines be sentient? And that really is a red herring from my point of view. I mean, it's not a red herring ethically. It's incredibly important ethically because, youically because if they are conscious, well, then we can have a conversation about whether they're ethically suddenly more important than we are or at least are equals and whether we're creating machines that can suffer, et cetera.

If we're creating simulated worlds that are essentially hell realms for sentient algorithms, I mean, that would be a terrible thing to do. So yes, it's all very interesting to consider, but assuming that we can't currently know, and we may one day never know, even in the presence of machines that say they're conscious and that pass the Turing test with flying colors, we may still be uncertain as to whether or not they're conscious. Whether or not they're conscious is a question that is totally orthogonal to the question of what it will be like to be in the presence of machines that are that intelligent, right? I mean, I think you can dissociate, or very likely you can dissociate consciousness and intelligence. I mean, yes, it may be the case that the lights of consciousness magically come on

once you get a sufficient degree of complexity and intelligence on board. That may just be a fact of our world. And we certainly have reason to expect that consciousness arrives far sooner than that when you look around at animals who we deem conscious, who are far less intelligent than we are. So I do think they're just separable questions. But the question of intelligence is really straightforward. We're just talking about, you know, conscious or not, we're talking about machines that can engage in

goal-oriented behavior and learning in ways that are increasingly powerful and ultimately achieve a power that far exceeds our own, given that it seems virtually inevitable given these two measly assumptions, again, any progress and substrate independence for intelligence, we have to acknowledge that we or our children or our grandchildren or some generation of human beings stand a very good chance of finding themselves in relationship to intelligences that are far more powerful than they are. And that's an amazing situation to contemplate. And I'll just remind people of Stuart Russell's analogy here, which somehow, at least in my experience, changes people's intuitions. On his account, it's like we're in the following

situation. We just got a message from some distant point in the galaxy, which reads, people of Earth, we will arrive on your planet in 50 or 75 or 100 years, get ready, right? You know, you don't know when this is going to happen, but you know it's going to happen. And that statement of looming relationship delivered in that form would be quite a bit more arresting than the prospect of continued progress in AI. And, you know, I would argue that it shouldn't be. I mean, they're essentially the same case. Yeah, I just think that framing is excellent. I mean, another framing is if you're chimpanzees

and you see there's other species, homo sapiens, that are perhaps less powerful than you to begin with, but they seem to be getting rapidly increasing in their power, you'd at least be paying attention to that. And yeah, you remind me a little of the second book of The Free Body Problem, where the premise, I hope it's okay to say, is aliens will come.

And I think it's in a thousand years' time. And they have far greater power than civilization on Earth does. But the entire world just unites to take seriously that threat and work against it. And so essentially, I completely agree with you that the precise timelines are just not very important compared to just noting if, well, not if, but basically when it happens that we have AI systems that can do everything that a human being can do except better. That will be one of the most important moments in all of history, plausibly. And, well, we don't know exactly when it'll come,

but uncertainty counts both ways. It could be hundreds of years, and it could be soon, for all we know. But certainly, relative to the tiny amount of time we're thinking about it, well, we should be spending more. But I'll actually just comment on the timelines as well, where there's a recent survey, just came out last week, in fact, of just as many leading machine learning researchers as possible. I'm not sure of the sample size, but if it's like the last survey, it'll be in the hundreds. And they asked, well, at what point in time will you expect,

you know, do you expect a 50-50 chance of developing human-level machine intelligence? That is AI systems that are as good or better than human beings at all tasks, essentially, that humans can do. And they say 37 years. So that's in my lifetime. I will be entering my retirement

for the 50-50 chance of advanced AI. When I talk to people who are really trying to dig into this, they often have shorter timelines than that. They often think that it's 50-50 within more like 20 years and substantial probability on shorter timelines. I tend to be on the more skeptical end of that. But basically, within any of this range, concern about AI, it's not merely like, oh, we should be thinking about something that happens in 200 years because maybe that'll impact 2000 years. It's like, no, we should be thinking about something

that has a chance of catastrophe. We're on the same survey. The probability that those machine learning researchers, that's people who are building these things, who you think would be incentivized to say what they're doing is completely safe the probability of an extremely bad outcome as bad as human extinction or worse uh that have the typical probability was five percent of that so put those numbers together and you have 2.5 chance of dying or being disempowered by ai according to machine learning researchers, not cherry-pecked for being particularly concerned about this, you have a 2.5% chance of dying or being disempowered

by artificial intelligence in your lifetime. That's more than your chance of dying in a car crash. And we think a lot about dying in car crashes and take actions to avoid them. There's like a huge industry, a huge regulatory system so that people do not die in car crashes and take actions to avoid them. There's like a huge industry, a huge regulatory system so that people do not die in car crashes. What's the amount of attention on risks from AI? Well, it's growing.

Thankfully, it's growing and people are taking it really seriously. And I expect that to continue into the future. But it's still very small. It still means that smart, morally motivated people can make a really transformative difference in the field by getting involved, working on the technical side of things to ensure that AI systems are safe and honest, and to also work on the governance side of things to ensure that we as a society are developing this in a responsible way, perhaps digging into, you know, AI is not a monolith. There are different sorts of AI technology, some of which are more risky than others, and ensuring that the safety enhancing parts of AI come earlier, the more dangerous parts come later. This really is something that people can be contributing now. I think they are.

I think it's hard. I'm not claiming that this is like, it's clear cut what to do. There are questions over like how much progress we can make, but we can at least try. Yeah. And I think the crucial piece there, again, this is courtesy of Stuart Russell, who is my vote for, you know, one of the wisest people to listen to on this topic, just to make the safety conversation integral to the conversation about developing the technology in the first place. I mean, he asks us to imagine a world where, how perverse it would be if engineers who were designing bridges thought about bridge safety only as an afterthought and under great duress, right? So there's the question about building bridges, all of the resources put toward that end, but then only as an afterthought do engineers have conversations about what they call the

not falling down problem with respect to bridges, right? I mean, no, it's completely insane to conceive of the project of building bridges as being separable from the project of building bridges that don't fall down. Likewise, it should be completely insane to think about the prospect of building AGI without grappling with the problem of alignment and related matters. So in defense of more near-term concern here, many people have mined the history here and found these great anecdotes that reveal how bad we are at predicting technological progress. I forget the details here, but I think it was the New York Times or some esteemed U.S. paper that confidently published an editorial. I think it was just something like two months before the Wright brothers conquered flight, confidently predicting that human beings will never fly or it will take millions of years to engineer, because it took evolution millions of years to engineer the bird's wing. It would likewise take us millions of years to engineer the bird's wing. It would

likewise take us millions of years to fly. And then two months later, we've got people flying. And I think there's one anecdote that you have in your book that I hadn't heard, but the famous physiologist J.B.S. Haldane in 1927 predicted that... I was going to mention this one, yeah. I'll let you do it. Yeah, yeah. Go ahead. Yeah, go ahead. What was that reference? Oh, yeah, well, so, yeah. One of the great scientists of his time, evolution biologist,

and had some remarkable essays on, you know, the future and predicting future events and how large civilization could get. And he considered the question, will there ever be a return trip to the moon? And he said, yeah probably in eight million years and that was yeah did you say 1927 yeah yeah so that shows you like people like to make fun of people in the past making these bold uh scientific you know predictions about technology such as the early computer pioneers at the, I think it was the

Dartmouth Conference, saying, oh yeah, we'll build human-level artificial intelligence in about six months or so. And people mock that and say, oh, that's so absurd. But what they forget is that there are just as many people on the other side making absurd claims about how long technology will take. In fact, technology, look, things are hard to predict. Technology can come much later than you might expect. It can also come much sooner. And in the face of an uncertain future, we have to be preparing for the near-term scenarios as well as the long-term scenarios. Yeah, and the thing we have to get straight in the meantime is that the incentives are currently not appropriate for wisdom and circumspection here. I mean, it's just, we have something like an arms race condition,

both with respect to private efforts and, you know, rival governments, one must imagine, right? And we're not, you know, we haven't fused our moral horizons with the Chinese and the Israelis and anyone else who might be doing this work and gotten everyone to agree just how careful we need to be in the final yards of this adventure as we approach the end zone. And it does seem patently obvious that an arms race is not optimal for taking safety concerns seriously. It's not what we want. And I mean, the key issue is just how fast do things go? So supposing the development of artificial general intelligence, suppose that's 100 years away or 200 years away. Then it seems more likely to me that safety around AI will look like safety around other technologies. So you mentioned flight. In the early days of flight, a lot of planes crashed, a lot of people died. learned primarily by trial and error what systems make flight safer. And now very few people die in plane crashes compared to other modes of death,

especially in more developed countries. But we could learn by trial and error because a single plane crash would be a tragedy for the people involved and for their families, but it wouldn't destroy the whole world. And if AI development goes slowly, then we could, I, then I would expect we will learn by that same kind of process of trial and error and gradual improvement. But there are arguments for thinking that maybe it will go fast. In fact, leading economic models, if you assume that there is a point at which you get full substitutability between human labor and artificial labor, like as AI, in particular, the process of developing more advanced AI systems, then you get very fast growth indeed.

And if things are moving that quickly, well, this slow incremental process of trial and error that humans, that we as a society normally use to mitigate the risks of new technology, that process is substantially disempowered. And I think that increases the risks greatly. And again, it's something that I really think, you know, that prospect of very rapid growth is at least something that should be very rapid progress should at least be on the table, where again this survey that just came out if you ask leading machine learning researchers though i'll caveat i don't think they're really the the people that one should think of as experts here but they're

still kind of relevant people to consider they think it's about even that that will happen like about 50 50 chance that there will be something akin to what is known as the intelligence explosion that doesn't necessarily mean the kind of classic it's two days but where we move from that there will be something akin to what is known as the intelligence explosion. That doesn't necessarily mean the kind of classic, it's two days where we move from no advanced AI at all to AI that's more powerful than all the rest of the world combined. Perhaps it occurs over the course of years, or even like a small number of decades. But nonetheless, if things are moving a lot faster than we're used to, I think that increases the risks greatly. Well, it seems like things are already moving much faster than we're used to, I think that increases the risks greatly. Well, it seems like things are already moving much faster than we're used to. It's hard to see where we get off this ride because the pace of change is noticeably

different. I'm not a full Kurtzweilian here, but it does seem like his claim about accelerating technological change is fairly palpable now. I mean, I guess there are sectors where virtually nothing is happening or improving. I mean, that is also true. But when you're talking about information technology and what it does to politics and culture. The recent years have been kind of a blizzard of change, and it's hard to know where the brake is to pull. Yeah, and people should just take a look at the recent developments in machine learning, where they're not very famous yet, but they will be soon,

and it's truly remarkable what's happening. I mean, obviously it's still far away from human-level artificial intelligence. I'm not claiming it's not. But when forecasts have been done, again, of people working in the area, of what sort of progress we expect to make, things are moving faster,

even than the kind of forecast of people who thought it was expect to make, things are moving faster, even than the kind of the forecast of people who thought it was going to be moving fast. So in particular, a recent, they're called language models. It's where they just get trained on enormous amounts of, enormous corpuses of human text. But one got trained to solve, do math proofs, essentially, at about college level, kind of easy college level. And the innovation behind this model

was very, very small indeed. They actually just cleaned up the data so that it really could read formula online, whereas that was a problem beforehand. And it just blew everyone's predictions out of the water. And so now you have this model where you say, you give it a mathematical problem, kind of early college level, prove that so and so, or like, what's the answer to this question of so sure proof, and it will do it with 50% accuracy.

And that was something that people weren't expecting for years. If you look at like other language models as well, I mean, okay, I'll give an anecdote. So one of my responsibilities for the university is marking undergraduate exams. And so I asked GPT-3, not even the most advanced language model nowadays, but one you can get public access to. I was curious, how good is GPT-3, this language model, compared to Oxford undergraduates at philosophy who studied philosophy for three years? And so I got it to answer these questions, like some of the questions that the students most commonly answered. That's great. And it passed.

It was not terribly good. It was in the kind of bottom 10% of students, but not the bottom 2% of students. of students, but not the bottom 2% of students. I think if I'd given the essays to a different examiner, I don't think that examiner would have thought, oh, there's something really odd here. This is not a human writing it. They would have just marked it and thought like, not a very good student. You know, a student that's good in some ways has good structure. It like really knows the art, like the essay form, but sometimes gets like a little bit derailed by arguments that aren't actually relevant or something that's what it would have thought but would not have thought oh this is clearly nonsense this is an ai or something and that's

just you know that's pretty striking we did not have anything close to that just a few years ago and so i don't know anyone who's like this know, I think it's fine to be like an enormous AI skeptic and think that AGI is very far away. You know, maybe that view is correct. Like I say, there's an enormous amount of uncertainty, but at least take seriously, like, this is like a fast moving area of technology. Yeah. Yeah. And obviously there are many interesting and consequential questions that arise well short of AGI. I mean, just on this front, I mean, what will be the experience of all of us collectively to be in the presence of AI that passes the Turing test, right? It does not have to be comprehensive AGI. It doesn't have to be empowered to do things out in the world. But just when Siri or Alexa become indistinguishable from people at the level of their conversation and have access to the totality of human knowledge,

when does Siri on your phone become less like a glitchy assistant and more like an oracle. That could be fairly near term, even with blind spots and something that disqualifies her as proper AI. I think the Turing test will fall for most intents and purposes. For some people, the Turing test has already fallen. I mean, we just have this kind of laughable case of the Google engineer who thought that their current language model was sentient. It was absurd. Yeah. I mean, there was no reason to believe that, but I would be very surprised if it took more than a decade for the Turing test to fall for something like a chatbot. Yeah. I mean, I'm actually curious about why there haven't been attempts

at the Turing test using language models. My thought is that they actually would have a pretty good chance of passing it. So I'm unsure on that. But one thing you mentioned is, yeah, this idea of like, oh, maybe rather than Siri, you have this incredibly advanced personal assistant. It's kind of like an article.

I just want to emphasize, this is one of the examples where I talk about accelerating the safer parts of AI and maybe slowing down or not investing much in the more dangerous parts, where this idea of kind of Oracle AI, just something that is, it's not an agent trying to pursue goals in the world. Instead, it's just this input-output function. You put in text, it outputs text. And it is like the most helpful, extraordinarily smart and knowledgeable person that you've ever met. That's the technology we want to have earlier rather than later,

because that can help us with the scariest issues around alignment. That can help us align AI that is more like a kind of agent acting in the world. And some of the leading AI labs are taking exactly that approach, and I think that's the right way to go. Well, Will, it's always fascinating. It was a joy, thank you Sam so to be continued, best of luck with the book again, that is what we owe the future

and we have by no means covered all of its contents and until next time Will, thanks for being here thanks so much Sam Thank you.