Into the Impossible With Brian Keating - Nick Bostrom: Will Artificial Intelligence Lead Us to a Utopian Future? [Ep. 414]

Episode Date: May 12, 2024

Join my mailing list https://briankeating.com/list to win a real 4 billion year old meteorite! All .edu emails in the USA 🇺🇸 will WIN! There are many stories, notions, and theories about what might happen if the development of artificial intelligence goes wrong. But what if it goes right? What if we develop superintelligence safely, govern it well, and make good use of it? These are the profound questions that today’s esteemed guest, Nick Bostrom, has explored in his highly anticipated new book, Deep Utopia. Nick is a Professor at Oxford University, where he heads the Future of Humanity Institute as its founding director. He is the most-cited professional philosopher in the world aged 50 or under and is the author of some 200 publications, including Anthropic Bias (2002), Global Catastrophic Risks (2008), Human Enhancement (2009), and Superintelligence: Paths, Dangers, Strategies (2014), a New York Times bestseller. With a background in theoretical physics, computational neuroscience, logic, and artificial intelligence, he is a leading expert in this field so without further ado, let’s dive right into his beautiful utopia! Key Takeaways: 00:00:00 Intro 00:02:37 Judging a book by its cover  00:06:01 Failed utopian experiments 00:10:47 The future of work 00:16:16 The limits to growth and superintelligence  00:28:30 Simulation hypothesis  00:39:14 The potential for AI alignment  00:47:53 Why Nick hopes there are no aliens 00:50:19 Audience questions  01:03:49 Outro — Additional resources: 📝 Get one month of Snipd Premium for free with this link: https://get.snipd.com/Cx7S/brianSnipd Snipd lets you take Smart Notes 🧠 with AI 💡 — it’s my favorite podcast player 😀 ! 📚 Deep Utopia by Nick Bostrom: https://a.co/d/bCDkoX0  ➡️ Follow me on your fav platforms: ✖️ Twitter: https://twitter.com/DrBrianKeating  🔔 YouTube: https://www.youtube.com/DrBrianKeating?sub_confirmation=1  📝 Join my mailing list: https://briankeating.com/list  ✍️ Check out my blog: https://briankeating.com/cosmic-musings/  🎙️ Follow my podcast: https://briankeating.com/podcast Into the Impossible with Brian Keating is a podcast dedicated to all those who want to explore the universe within and beyond the known. Make sure to subscribe so you never miss an episode! Learn more about your ad choices. Visit megaphone.fm/adchoices

There are many stories, notions, and theories about what could happen if the development of artificial intelligence goes wrong. But what if it goes right? What if we develop superintelligence safely, govern it well, and make good use of it? Could we have unparalleled prosperity? These are some of the profound questions that today's esteemed guest Nick Bostrom has explored in his highly anticipated new book, Deep Utopia. Nick Boschrom is a professor at Oxford University, where he was the founding director of the Future of Humanity Institute.

Also, the creator of many popular concepts and even memes in the memosphere, such as the simulation hypothesis, and the so-called paperclip problem of AI alignment. He's an expert in computational neuroscience, artificial intelligence, and the leading expert in this field. So without further ado, let's dive deep into his beautiful utopia. Any sufficiently advanced technology is indistinguishable from magic. Open the pod bay doors. And Nick, first of all, how are you doing today? Great, thanks. How are you? It's great to have you back on.

As you know, on this tradition that we have on this podcast is to judge books by their covers. So that means we're going to go through the book and this wonderful new book, I read and listened to using tools that we discussed. But I actually want to do something I'm never done before, Nick. If you'll humor me, I want to judge a press release by the press release. So I got this press release from your wonderful agency, 4THA. Shout out to Jessica over there, Jessica Pellion and to your wonderful assistant, Emily. So it describes the book, and we're going to get into that deep utopia, life and meaning in a solved world.

And then it says about Nick Bostrom. Nick Bostrom is a professor at Oxford University where he heads the future of Humanity Institute is founding director, most cited professional philosopher in the world age 50 or under, and is the author of 200 publications. His writings have been translated into more than 30 languages, TED Talks, etc. He has been a guest on the Joe Rogan podcast, the Lex Friedman podcast, and Brian Keating's Into the Impossible podcast. Ah, yes, yes. The best press release I've ever read. And let's say Taylor, it's every podcast host, you know, if this is a Joe Schumerer.

We really hit the bullseye on that one. So you get all the points for that. So, Nick, we are going to judge the book by its cover now. This is this book, in addition to being really provocative, intriguing, wonderful to read, is also beautifully published and bound. I wasn't familiar with the publisher, Idea Press, publishing, but it's a sturdy book. It's a dense book. And I want you to take us through three things, the title, the subtitle in art. and also this art, this provocative pig and wolf,

and what role do they play in this wonderful new book? Well, yeah, where do you want to start? The title is, I mean, I guess sort of self-explanatory, but the deep there means it's kind of a different question I'm trying to ask and explore than the question traditionally associated with the term utopia. Often the term has been used for people who come up with some scheme for how society should be organized.

some system of governance or gender relations or culture or some economic vision. But those are not really at all the questions for this book. This book instead explores what happens if we truly succeed with AI and everything else and end up in this condition of what I call a solved world, where all sort of practical needs can be catered to by robots or super-intelligent AI's, etc. So it's much more a book about our... fundamental values than it is about the practicalities of how to organize human society. The subtitle, I guess, just clarifies that a little bit, maybe.

The illustration there, yeah, I actually made the cover myself, and at one point, that was a survey of different earlier designs as well, and people were voting, and I actually didn't go with the one that received the largest number of votes, because in the end, I felt this cover more expressed the spirit of the book, which, to be fair, to the people who voted, at that point, the book wasn't available. So it would be hard to see, like, if you just see a snowman, like a weird cover for a book, like about philosophy and stuff. So I think I had the advantage of knowing what the book was about. But yeah, I felt it expressed. It has a sort of playful spirit. And the snowman somehow seemed expressive of that. And there's a mitten. And then

there's the wolf and the hog and the fox actually it's a fjordor the fox so he makes an appearance inside it has these three different distinct elements that are interplay with one another so there's this fictionalized version of a professor bostrom who's giving a series of lectures so that's element one the actual content of those lectures there are assigned readings that this professor character gives to the people attending the lectures. And so one of those assigned readings is this story, the epistles of Fyodor of the Fox and his friend Pignolius, which is the pig character there. And then the third element are dialogues, conversations between some of the people who

are attending these lectures and they go off afterwards and continue to discuss various questions and kind of poc fun at what they've heard, etc. So, yeah, so, but the pig and the fox are from that, the epistles of Theodor of the Fox reading. Well, I want to start off with the first real question of the day by naming a couple of utopias. And the first one is in my home state of New York, my original home state, I'm in California now, but the Oneina community, a socialist community in upstate New York, practice complex marriage where all members were considered married to each other. While initially successful, it eventually collapsed due to internal power struggles and scandals. The Soviet Union's communist utopia, a Bolshevik revolution aimed to create a classless communist utopia. Instead, it created decades of authoritarian rule.

Jonestown cult 1955 to 1978, it led to the largest mass suicide murder of 900 people up until 9-11. And last but not least, the Biosphere 2 experiment create a solid. self-sustaining miniature replica of the Earth's ecosystem in Arizona, but it was plagued by infighting oxygen depletion and the ability to, inability to fully replicate natural systems, ultimately failing to achieve its utopian goals. Nick, what is your, based upon prior, so if we just had Bayesian reasoning, what would you expect the probability of getting to an artificially induced deep utopia, given that the natural stupidity of humans seems to have precluded it heretofore?

Yeah, I mean, if we can avoid failure by oxygen depletion, at least, I think that would be. Yeah, no, I think certainly the track record of utopian visions has been poor. It depends a little on what you count. I mean, in some sense, you could see regard like the United States of America as a kind of utopian experiment in like, oh, we're going to have this like democratic constitutional thing. Everybody's got to be free and have these liberties. But I think, yeah, so depending on how it is exactly like your definition of utopia, your reference class might include some more successful and a lot of less successful instances.

What's been bad is not so much trying to imagine a better way for things to be. I think that has been an inspiration for many people to work towards reform and improvement. But the idea that you could have a blueprint that somebody has figured out, and especially if the idea is done that, that needs to be like coercively imposed on unwilling people, that I think invariably led to bad outcomes, ranging from kind of unattractive all the way to sort of genocidal. It's interesting if we see, say, fictional attempts.

There are a lot more dystopias. than utopias. I mean, if most people could probably rattle off a number of dystopia, like, Brave New World, 1984, the Handmaid's Tale, the average person might not be able to name a single utopia. And even of the ones that have been made, like if you actually read them, it's probably, you probably wouldn't actually want to live there. They often tend to not to saccharine or sort of cloying or like some monoculture that, you know, maybe some of them would be fun to visit for a weekend, but like you wouldn't sort of want to permanently move in. That is kind of interesting in that it's actually even when you're allowing yourself

the space of like a work of literature that can just make stuff up, right? It doesn't even have to work in practice, but even just like even that exercise of imagination is quite hard to conceive of a situation for humans that is radically different from the current one and still appealing. Like it's easy to imagine small tweaks to the current condition. Like suppose everything is exactly the same except we eliminate childhood leukemia. That seems great. Like that would be better than what we have now, right?

But as you sort of try to imagine more and more improvements, you very quickly get to this point where it becomes kind of bland or uninteresting or some other things seems to be lost, which I think is an interesting fact about either the human imagination, or our human values. Hey there, it's me, your favorite professor in the known multiverse. Do you want to know what my utopia looks like? It's a world where everybody who made it this far in the video also subscribed to my YouTube channel or followed me on their favorite podcasting platform.

Jokes aside, it really does help me out. And right now, only about 50% of you who are enjoying great guests like Nick Bostrum, Sam Harris, the late great Dan Dena, and others are actually subscribed and engaging with content. So please help me out. It does tremendous good deed for me to get me deeper dives with some of the most brilliant interviews on the planet. Please subscribe, follow, and share the podcast. Now back to the conversation. Absolutely.

And, you know, I look at these things and, you know, I'm a practicing Jew. And I kept thinking of, you know, this notion of Jewtopia, which would be, you know, kind of re-instituting the laws, you know, thousands of year old laws and just how that would not necessarily be utopia. a utopian vision at all in a modern society. I mean, after all, we live better, the average person, you know, the median human being in the West lives better than any of the richest monarchs of Europe, you know, 80 years ago, 60 years ago. So the incentive is almost nothing to try to perturb from where we are right now. And yet, and yet there's always seems to be this desire that, you know, there will be some future. This time is different. You know, we can make, we can make the utopia in various images. But I want to stick kind of close to you and what you and I do, which is,

profess as professors in various disciplines. You have a physics background, but are much more well known for philosophy. I love this chapter where you talk about some practical implications of it. You speak about templates of OTIM. So instead of looking at, you know, possibilities that could come in the future, you look at actualities that are occurring right now. So you talk about the status of children in OTIM in a world where labor markets and so forth, participation can be considered. You talk about the status of children where they're loved beyond compare, but they have no status, no power. You talk about aristocrats, you know, like in the Gulf States, Bohemians, monastics, retirees, which come off, all of these come off very ambiguous as to

is their status utopia, utopic or not. But the one I want to focus on is the plight of students, because that's very close to the theme of this book, but also what you and I do. And you say, Although students don't do much paid work, and certainly not my students, you know, they're lucky if they make minimum wage, Nick. But anyway, students don't do much paid work. One could think of studying as a work analog, something they have to do, whether they feel like it or not, which has deferred payoffs. Students often have low incomes, few financial responsibilities, and limited need. But they have somewhat high social status.

In other words, people think about what they'd like to do, and it's usually go back to school or, you know, maybe even teach. talk about what would you see in such a utopian vision? What would you see for what I call the second oldest profession? What you and I do has largely unchanged since the year 1080 in the first Western University in Bologna, Italy, when there was a sage on a stage with a piece of rock and he was scraping on another piece of rock to raptured students. Tell me, what do you view, what would the day in the life be like for you and me, you know, in such a deep utopian environment? Well, so in a solved world, right? So this is a setting at technological maturity.

So we imagine at this point that technology has run its full course. And so we have in particular advanced AIs that can do all the things that we can do with our brains and robots that can do all the things we can do with our bodies. And hence, basically full automation of all economic tasks. with some exceptions that we can get to. But in this condition, we would then have like full unemployment. There would be no economic work that humans would need to do. Not only that, but a lot of other things that we do that is not in order to make money,

but we have to do a whole bunch of other things to, like if you want to be fit, you have to go to the gym and work hard at the gym, right? If you want to make friends, you have to invest time and be there for them. when they are in a dark place, et cetera, et cetera, right? If you want to have a healthy mouth, you have to brush your teeth every day, et cetera. So you can go through a lot of what fills our days, even for people who don't need to work for a living.

Like there's a lot of instrumental activity. We do one thing in order to achieve something else. But at technological maturity, that would all drop away. So you might think, well, how could, like, you know, you can't, like, rent. the robots to run on the treadmill on your behalf, right? That's something you have to do yourself. Well, now if you think it through at technological maturity, you'd be able to take a pill that would induce the same physiological effects as exercise. And you can kind of go through the list

of leisure activities one by one, and you'd find that many of them would become otios or unnecessary in this solved world. And so that then raises this question of what would give meaning and purpose to human existence in this condition where we didn't need to do anything. That then kind of sets up, which is really the heart of the book, the first bits are more kind of throat clearing where I like talk a little bit about this economic stuff, but then it's really just to get to the point where we can ask the questions about our fundamental values. What would you do if you didn't have to do anything? Because robots could do everything better than you. And that includes, of course, my job in like being a philosopher, thinker and stuff

like that, that's also something that AIS will surpass us at if this AI transition succeeds. So it's not the case that, you know, all the boring, simple tasks would be automated and then we would be the only ones who could do the creative stuff. The creative stuff also will be better done by machine at technological maturity. It's peak pollination season, and my business is scaling fast. To keep the nectar flowing, I need a phone plan with top priority data speeds. That's why I chose GoogleFi Wireless. My connections stay strong even when the hive is buzzing.

Plus, unlimited plans started $35 a month. Now, that's a deal that doesn't stay. Explore Google Fi Wireless plans today. Plus taxes and government fees. Google Fi Wireless is not subject to data traffic deprioritization during times of high network usage. Is superintelligence a prerequisite for technological maturity? Yeah, it would be included in the affordances at technological maturity. It's kind of the definition of the concept would be a condition in which all those

technologies that we know to be physically possible and for which there is some possible pathway from where we are now to there have been developed, or at least some close enough approximation of that. And certainly, I think, greater than human artificial intelligence is one such. possible technology. So it would be included in that mix. I guess you could theoretically, although less like realistically, but theoretically, you could imagine getting to a place without superintelligence that would unlock almost all the same affordances by just say if human programmers had millions of years of writing software to automate a task one at the time.

Like eventually when you've kind of made enough of that software, maybe you could automate basically everything. And so you could maybe achieve the same, like, end results without having this general purpose machine intelligence to do it. Now, I think it's much more realistic that we will get there much faster through first creating powerful AIs that can then figure out how to do all the rest. But like, at least conceptually, that there would be more than one way of kind of reaching an equivalent endpoint. 52 years ago, the club of Rome, which I don't think exists, or at least they're not asking me to join it, published a report called The Limits to Growth. And it was the first to model the planetary interconnected systems that are depending, are the dependent variables upon which the future

of society, industrialization, food production, population growth, and resource depletion would eventually, according to them, lead to carrying capacity and then it would overshoot and collapse. We haven't hit that. There's Ehrlichman and many others predicted, you know, kind of a dumerism and the rise of all sorts of population bombs and things that never came to happen. And yet, I'm an astrophysicist. So we live on a planet. We live on a rocky planet surrounding a medium, you know, mediocre star that provides a lot of our needs. But we're nowhere close to getting towards what my first guest on the podcast was the late great Freeman Dyson. And, you know, his Dyson sphere, they're very long way off perhaps impossible. Are there planetary limits to growth that could preclude superintelligence from

ever coming about? There are planetary limits. I think they're fairly high up if we imagine more advanced technology that could do a whole bunch of stuff. Even if we stipulate, we were restricted to the resources of Earth and the solar insulation coming down and hitting the surface of our planet. There's still plenty of room just within those parameters. I do think clearly for a technologically mature civilization space expansion would be possible, including Dyson spheres or the equivalent sort of harvesting the astronomical amounts of energy and other resources that exist out there. There have been various studies on kind of conceptual designs for different kinds of space

probes and for Neumann probes and constructions like the Dyson's that seemed perfectly consistent with the loss of physics. And so I'd imagine a technological measure civilization could develop and deploy those. That's being said for the purposes of asking these questions about Deep Utopia, it doesn't so much focus on the extensive question, like just how many people could you sustain, you know, over on a planet or a galaxy or beyond, like it more thinks in terms of an individual human life, like yours, for instance,

what would be the best possible continuation of that? If we imagine, for instance, that you could fix a lot of the diseases and poverty and injustice, et cetera, et cetera. Like if all those practical difficulties that beset human existence were somehow removed, what would be the best possible continuation? And in particular, what would that continuation look like as we move beyond the normal span of human life?

like if you could live for not just 70 years or 80 years, but if you could continue to flourish and maybe develop more beyond that, like what kinds of life trajectories would be possible and which ones of those would be best and how could they intersect with other human life trajectories and maybe life trajectories of digital minds and other creatures, uplifted animals perhaps, as well.

And so to some extent, kind of the internal question of what are the best possible lives, and small communities can be separated from the question of just how many billions or trillions of, you know, humans could there be, or how many human planetary civilizations

could we establish through the universe? There's like, there's one little handout that addresses that in the book, like there are two pages on that or something, but it's not kind of the core. I want to turn next to the notion of superintelligence and a lot has changed. It's just incredible to imagine

our first conversation took place in August of 20, 2022. And since then, you know, just an incredible explosion and AI, consumer facing and even full lift commercial grade, but even in the realm of, you know, what I practice and beyond just machine learning, just in quotes, which has been around forever, but all the way up into, you know, utilizing AI as a research tool, if not for constructing new laws of physics. So as I spoke about in that first interview, this guy here, you know, I'll Albert Einstein, part-time philosopher, but he said the happiest thought he ever had is if he was in free fall, he'd experienced no gravitational field. And I pointed out to you that I couldn't

conceive of how silicon or quantum version of Einstein could, A, experience a happy thought. What does that even mean to a computer or a digital in-silico version of a conscious mind? And two, what would it be like to visualize something if it's not embodied in some sort of sensory package, not just with accelerometers? Obviously, it can do that in any modern car. So is it what would you think? I posit that the real turning test is when AIs can come up with new laws of physics, which to my knowledge is not even remotely close to being done. So what would you say is that, you know, sine qua non of, you know, when you will think we've broached that. that Rubicon and truly obtained AGI.

Yeah, I mean, I think the criteria of discovering new laws of physics would rule out a lot of humans from counting as well as generally intelligent, right? It's like a small sliver of humans who have ever either actually found a new law physics or who were like realistically on the path there. So aside from some little group of theoretical physicists, so I think that would be too high a bar. I mean, it would obviously be a higher level of AGI if the AI could also do that. But I think even somewhere short of that, you could still have something that I think ought to count. That's ultimately a verbal question, right?

Like you could define the word to mean whatever. Yeah, I am obviously very impressed with what AI has been achieving over the last, even just the last few years. Like it's, you know, it might be now that we are kind of in the early stages of an intelligence explosion. and the coming years will tell. So you might be, I mean, in fact, like when I was writing this book, I was at some point, it took a long time to write it, and I was worrying that that would kind of, it would arrive after the singularity. So I think at least mission accomplished in getting it out before the singularity.

And ideally, we did not time so that people could like read it as well before our efforts become redundant. Yeah, no, I think this is one reason for, I mean, I think you could read the book just, if you wanted to, like, as a purely theoretical exploration, like philosophers like to construct thought experiments, and it's a way of thinking about human values and their basic constituencies. If you kind of almost like a particle accelerator, smash little bits of matter together at extreme energies, right, and create sort of exceptional conditions where they split apart and you can study the quarks and components.

And then you can figure out, ah, right. So what we can see in particle accelerator, probably the same principles are at work all the time. It's just we can't see it in ordinary matter because it's kind of hidden inside deep inside the nucleus, right? And so similarly with some of these philosophical thought experiments, you create extreme conditions that allow you to smash values together and see their study their basic constituents. And then you can kind of extrapolate from there and say that in our ordinary lives, maybe the same constituents are present. It's just we don't see them because they're kind of concealed behind all kinds of practicalities and limitations that, you know, fill up our daily lives. So that's one way you could read this book. Now, for me, there is another way of reading it as well, which is that if we're actually in the early innings of an intelligence explosion or if that's something that might happen, perhaps within the lifetimes of some of the people watching this, you know, then there might be.

a time when some group of humans, whether it's like a few people in some AI lab or some government or more of a humanity-wide process of deliberation, we'll have to actually form a view about what we want with this fantastic technology that we will develop. Like, what do we want the long-term trajectory to be? And that will be very hard at deliberation. And the conclusions might shape things for millions of years to come potentially. So I felt it would be nice if whoever goes into that deliberation had something like to read in preparation to kind of partly equip them with like some useful concepts and questions and considerations and also ideally to put them in the right spirit so that they could approach such a deliberation with the right

kind of attitude i think a kind of sense of expansive generosity the playfulness thoughtfulness open-mindedness, which I'm hoping the reading experience of this book will also help convey to the reader. How has AI changed your day on a day-to-day basis since we last spoke two and a half years ago or two years ago? I haven't really, I mean, I do sometimes use these LLMs, but for small little tasks here and there, not enough to really change my day so far. A lot of one I'm doing, you can't really just prompt an LLM to do it with the current capability levels. But I think a mistake many people make still today, even as awareness of the potential of ADI has like spread, is that they overindex to what AI is now.

So people think, oh, well, what is AI? Well, it's basically chat DPT4 and we can start to think about all the different applications from that, how you could apply it, like, you know, maybe develop a healthcare app and like a translation app and a tutor app. But it seems very plausible that just as, you know, Gemini or TTIPt4 is miles ahead of where it was only four years ago. Similarly, four years from now, we will have way more generally intelligent AI systems. And we should be looking more at where the puck is going.

rather than thinking we have now basically achieved the kind of AI will have and we just need to think about how it's going to impact society. It's very much a moving target. Can you talk about simulation theory and how it might be resonant, especially with their ethics, the ethical responsibilities that either humans or Silicon may have for the denizens of the deep utopia? First recapitulate for the audience that may not have heard the first conversation. simulation theory, what is it? And then how does it resonate with this problem of utopia as you articulate? Yeah. So the simulation hypothesis is we are living in what is literally a computer simulation constructed by some advanced civilization. In fact, this would be one of the affordances at technological maturity, the ability to create perfectly realistic virtual environments,

at least realistic enough that the people inside them couldn't tell them apart from physical reality. And also to simulate brains in computers at the sufficient level of granularity that these simulated brains would generate conscious experiences. The simulation argument tries to show that one of three alternatives, at least, is true. One of which is this simulation hypothesis that we are living in a computer simulation. And there are two other alternatives, either almost all civilizations that are current stage of technological development go extinct before reaching technological maturity. It's like a second alternative. And the third is that there is some extremely strong convergence amongst all technologically mature civilizations, that for one reason or another, they all just refrain from using their immense computational resources for the purpose of creating these computer simulations, ancestor simulations. So it imposes the simulation argument as a constraint on what we can coherently believe about the future and our place in the world.

Now, if we zoom in on the first alternative there, the simulation hypothesis, then yeah, it is an interesting question how that would interact with these questions of utopia. I mean, logically speaking, you could still ask whether we are in a simulation or not in a simulation, what would be the best possible continuation for us? the practicalities might be significantly influenced by the simulation hypothesis. It either might make it easier or harder for us to actually attain that. It would be more in the hands of the simulators than in our own hands if we are living in a simulation. Certainly they would have a say in the matter, right? They could if they wanted to either teleport us into utopia right away or switch off the simulation and we pop out of existence or not just there it's in different directions.

So I think the simulation hypothesis would indicate that there are additional levels of reality over and above the one we sort of see around us and that there are very powerful beings in the form of the simulators whose values and goals would ultimately perhaps determine our fate. But yeah, the value questions are I guess at least. at first sight orthogonal to the question of whether we're in a simulation. Although it is interesting to explore there might be certain linkages at certain places. If you thought one of the things that make our lives currently valuable is that they could have

immense causal significance. The fact that we can, depending on our choices and our actions, change things for the better or for the worse, potentially on a very large scale, especially if you think we're kind of close to an intelligence explosion, and maybe the actions we take over the next few years could have potentially astronomical implications. If you think that having such significance in your life, it makes that life better in its own sake,

then you might think if we're in a simulation, well, it's not the case that our current lives perhaps have such big significance because the simulation, like ultimately we have to share credit or blame, I guess, with the people constructing the simulation and maybe our ability to have causal impact would be less. It depends on which particular kind of simulation, but like at least for some classes of simulation, that would be the case. Then you might think our lives already have less of that kind of causal significance

than you might have thought. And so then if you then consider this condition of a solved world, you know, where like one of the problems there is maybe we wouldn't be able to have much significance if we lived in a solved world because we wouldn't like we couldn't have significance by solving problems if all the problems are already solved and whichever problem weren't solved could in a way be better sold by AIs. Well, if our current existences are already more impoverished of significance than we had thought, then maybe that problem is less of a problem because we wouldn't be losing

much significance. Or you might think right now part of what makes it so exciting to be here. human is that we discover new things for the first time. Our scientists do, and individuals maybe live lives that have never been lived before. So there's like novelty and originality. Maybe that would be less likely to be true if we were simulated, if that were just a lot of other simulations, or maybe the same or a similar simulation had been run before. I call that the Kardashian problem. Why are there so many Kardashians? Yeah. Why do we need about that? Yeah. Let's start with bug. Now, it might be that even if we're not in a simulation, we are anyway not so novel and

unique. I mean, particularly if we are in an infinite universe, there would already be infinite number of people arbitrarily similar to us who have lived similar lives. But maybe you think, we only need to be unique within the part of the universe that we can causal interact with. And so then if we just have an infinite universe, like most of that would be outside of our causal reach. So if you somehow say that doesn't matter, the fact that we are kind of redundant with what has already happened elsewhere. But in the simulation hypothesis, there is a possibility we could causal interact with other levels of the simulation stack, et cetera. So maybe then the fact that there have been other people like us before us and will be after maybe that would reduce. So these are more subtle ways in which you might think certain kinds of human values could be affected by the fact that we are in a simulation, if that is indeed a fact.

Hey, it's me again, Professor Brian Keating, I mean Brian Keating, with a quick request that you join my Monday Magic mailing list, where I share conversations and deeper thoughts in the deep dives into deep utopias like this. Every week I send you something about the universe that you haven't thought about before. And you'll even get a special chance to win a meteorite, a piece of space schmutz, a real meteorite, 4 billion years old plus. You get all the information about it. And for your chance to win, go to Brian Keating.com slash list. And if you're a college student, professor, postdoc, engineer, whatever, you're guaranteed to win one. Go to Brian Keating.com slash edu. For those of you who have a dot edu email address in the United States of it.

America. Don't miss out. Now, let's get back to the episode. You know, naturally, when I spoke about this with Ben Shapiro many years ago, he basically tried to convince me that, you know, the simulation hypothesis is just an all-powerful, you know, deity like, you know, modernized with newfound layers of technology, but essentially it boils down to religion. Because it brings up things like ethics of the simulators to the simulated and the granularity and what details do you feel. And what details do you focus on, you know, for moral good, for ethical reasons? What do you say to critics that basically say it's just a rebranding of a good old-fashioned or bad old-fashioned religion? Yeah, so the simulation argument doesn't start with a religious set of assumptions. It starts by

just taking for granted that the kind of standard scientific view of the world is correct, that there is this physical universe around us, that we can discover its properties and invent more and better technologies, including in particular compute technologies and VR, etc. And then from that assumption, thinking through what kind of technologies that means we would eventually develop if we succeed at attaining technological maturity. And then, you know, with some probability theory,

you then get this constraint that one of those three propositions is true. Now, it is true that although the starting, point is just standard kind of materialistic view of the world, right, that scientists tend to assume. The end point, if we in particular pick that alternative, the simulation hypothesis, does have a lot of theological resonances and parallels and analogies. It kind of moves us in that direction of we would have to recognize that our world might be created, at least the part of the world that we inhabit and see an experience.

experience by some entity that might be far greater than us, maybe not literally omniscient, but super intelligent and maybe not literally omnipotent, but able to intervene inside our part of reality, including in ways that would contravene the loss of physics that we have observed. Like, I mean, if you have access to the sort of the root access to the simulation, it could, in theory, go in and tweak and change it. So that, I guess, kind of, takes you some ways towards some traditional theological conceptions where you could kind of compare our relation with the hypothetical simulators to how our

relations with gods have been conceived, but with big differences as well, like the simulation hypothesis doesn't assume the gods would be literally infinite. And so they are kind of, I think, logically speaking, orthogonal. you could have like a simulation argument without a theological god or with it. I mean, there's no particular reason why you couldn't have, say, a Christian god creating a universe within which there are creatures like humans who eventually develop some clever technology where they could create little virtual worlds inside those computers, etc. Like that could all be consistent.

It doesn't really answer or try to answer any of those questions. But there are kind of interesting structural parallels. I guess the thing that comes up when I think about this is, you know, are we kind of past the Rubicon, you know, in religion, we don't have a chance to get root access, as you say, because, you know, you have this omnipotent, omniscient being that controls the bash script. But do you think we're past the point of being able to control AI alignment? You've spoken about that a lot. Or have we reached, you know, the great filter, you know, in a sense of, or are we coming to it or have we passed it? When it comes to AI alignment, values alignment problem. With respect to AI alignment, the jury is out still, right? We haven't yet developed a kind of AI for which having scalable methods of AI alignment is necessary to avoid existential catastrophe.

So far, the AIs we have, we're starting to have various kinds of impact, but they don't yet pose any, like, threats to the survival of the human species or any risk, it seems, of kind of permanent. demonstantly destroying our future, but the AI systems will become more powerful, and I think we might soon need to have scalable methods of AI control. By contrast in 2014, this was when my previous book came out, Superintelligence, which tried to draw attention to this alignment problem. Back then, it was completely ignored by academia and generally kind of people were dismissing

it as like science fiction or futurism, but there's been a big shift since then. So now it's become very mainstream. All the frontier AI labs, Anthropic, Open AI, Google DeepMind have research teams now working on trying to develop solutions to this. And some of the smartest young people I know are going into this field of AI alignment. But we still, I think, don't have a solution that we could be confident would work, even if you scaled up the kind of capability of the underlying AI systems to arbitrary levels. But a lot of ongoing work on exactly that. So hopefully we will figure it out, but we don't know exactly just how hard that problem will turn out to be. So one of the things the human mind is very difficult, as difficulty with is, you know, judging exponential trends.

We tend to think linearly and that extrapolation is essentially linear regression that were evolved to do exponential behaviors and technologies. or they sneak up on you, you know, as it said, gradually and then suddenly. But you do see, you know, punctuated points of equilibrium where there'll be some new technology and it'll be adopted by a very, very, you know, minute few, maybe the aristocrats, as you speak about in the book. Could you have a deep utopia without first having a shallow utopia, a two-tiered or multi-tiered, you know, situation where there's utopia for, you know, for me, but, you but not for the, how do you, how would you envision that this would occur?

Or would it just occur all at once? Yeah, so those are, I guess you could make different empirical scenarios about how things might unfold. My guess is when we really get into the kind of steep part of the intelligence explosion curve, I think things will happen relatively rapidly so that I don't think that will be many decades where we sort of just incrementally better AI until at the end of those several decades, we have superintelligence. But I think once you get the kind of positive feedback loops, once AI is good enough to like really start to make as much or more contribution to AI research

as humans are doing, then I think you might get something quite fast so that it will all shake out within a few years for better or worse. And then, I mean, we don't know this at all. but it's conceivable that you might get a kind of lock-in so that whatever conditions are then established become permanent in the sense that one of the technologies produced by the superintelligence is the ability to prevent itself from getting deflected

from whatever course our civilization is on at that point. And incidentally, this is something that can't be completely allowed even without superintelligence, just with, say, current levels of AI, if applied more broadly, you could imagine it rolled out to unprecedented levels of surveillance or automated propaganda or censorship regimes or other ways of kind of stabilizing the global discourse so that some particular orthodoxy is frozen in. And it just becomes a lot easier to stigmatize anybody who deviates from that. And if you get a sufficient level of that in,

then it might sort of just lock itself in so that with each kind of subsequent generation, it becomes even more efficient at weeding out dissent. That I think is a possible outcome because you could think of sort of global discourse and our political dynamics and discourse dynamics as kind of taking place within a certain frame defined by various parameters,

like how easy it is to do mass communication or to understand what keep track of what each person has said in the past and what their views are, even if they don't say it explicitly, can you read between, we know from history that when you change the knobs on some of these parameters, it can have profound effects

on what kind of political equilibrium emerge. So with the invention of writing, you could have states that could keep track tax records and stuff and you could then that just changed what kind of political systems you could have like when with a printing press we again we had like decades of religious war and then we had the rise of democracy that all became feasible with writing very hard to do on a large scale without writing again radio and television like broadcast technology seems to have had a big impact and maybe helped some of these like totalitarian ideologies proliferate like when one leader

could speak to the whole nation and get the masses on board to march according to their vision and you have like Nazism and communism and other and more recently with social media. I mean, we're still in the midst of that. We don't know how it will shake out, but it seems like it's having some kind of impact on our political discourse and on elections, et cetera. And so now we're introducing a new element. Like up until now, it has been possible for a few decades to let's say record everybody phone calls and text messages.

But then what do you do with all of that information? Like if you have to have a human analyst kind of reading through all the little boring text messages everybody has ever sent, like you can't really do that much except target specific people of interest. But with current LLMs, you could imagine just having them read through every single scrap that anybody has written and create a very detailed profile with sentiment analysis, figuring out exactly what are they really thinking of their current regime. Do they have any forbidden thoughts reading between the lines?

And then you could couple that with some kind of social scoring system that could then just automatically create fine-grained incentives for everybody to queue the line. I will take some years to roll that out, but it seems now within the kind of technological envelope. Our ignorance also means we can't draw out good outcomes. You might think the ability to more easily find information and form richer, more contextualized opinions about various topics and people,

as opposed to just what somebody is putting out in their PR statement. Like now you can find much more like they could imagine that that allows kind of human civilization to gradually self-correct into a more sensible and sane level of functioning, where we start to be able to come more aware of the way it's currently dysfunctional and some system where ideas for improving that could kind of more easily get support. and we might then kind of spiral into like a much healthier dynamic. Both of those, I think, are possible insofar as we can tell. Because we don't have the kind of predictive social signs that can allow us to anticipate

what happens if you change these parameters. So we'll, you know, as long as there is ignorance, there is hope. And there's a lot of ignorance. I think we found the title to this episode, Nick. So back in 2008, you wrote a story called Why I Hope the Search for Extraterrestrial Life. Finds Nothing. Where are they in the MIT Tech Review? And back then, you also made reference to utopian societies, including alien cultists and the cult, a collection of those that believe that aliens are here. And not only that, they will lead to, many believe, to utopian future. Have you

updated your conclusion that you hope there are no aliens from 2008? Or could they they enter, perhaps usher in the era of deep utopia? Well, I mean, that would be possible if we encountered friendly aliens with super advanced capabilities. They would presumably be able to use those capabilities to help fix a lot of what ails us currently. That seems possible. Now, I don't think my probabilities have updated that much regarding either the likelihood

of encountering extraterrestrials. or the desirability of finding more primitive forms of extinct life, like that paper discussed. There is a guy Professor Robin Hansen, who's more recently written some stuff, developed some interesting new models for how to try to estimate the likelihood of us sharing the observable universe

with alien civilizations. And you could, you might want to interview him separately on that, it gets a little bit intricate. I saw that's like, I think the most interesting new piece of analysis on that topic that has emerged in the intervening years. But, yeah, it hasn't led me to make any big update on that particular question. And you, so I take it you're not particularly swayed by the claims of physical evidence of non-human biological material as claimed by certain members of the U.S. military. haven't swayed you in the past two years since we've spoken that actually the aliens do exist? No, I'm skeptical of that.

Like, that said, I haven't actually looked at any, like, detail about the specifics on that. So I don't feel particularly competent to opine on it, but I am skeptical. You said this place was steps from the water. We just haven't found the steps yet. How much did we save? Enough. Enough to get lost. Or you could book a stay with Hilton.

Welcome to your oceanfront room. Just steps from the water. The Hilton sale is on now. Book on Hilton.com or the Hilton app and save up to 20% to get the stay you expected. When you want savings, not surprises. It matters where you stay. Hilton, for the stay.

Well, in the last few minutes that we have left in the hour, I want to ask you some questions posed by my audience, and I want to remind the audience, you can always ask me questions for my kids. guest on Twitter or on YouTube or Instagram on Dr. Brian Keating, D.R. Brian Keating and all of those. And if you have a dot edu email address as Nick does or he has a version of it, you're eligible to win for a limited time when a sample of alien material, extraterrestrial material. This is a chunk of the four billion-year-old solar system, a meteorite, a real genuine meteorite

along with some data sheets about it, in some ways you can watch meteor showers and learn more about meteorites. If you go to Brian Keating.com slash edu, if you have an EDU email address. Otherwise, you can take your chances at Briankeating.com and I give away one of those to a non-student or non-faculty member every month as well. Okay. So, Nick,

a few more minutes to answer questions. First one comes from Michael Wessel, who asks, were you, Nick, aware of the 1964 book Simulacron 3 by Daniel F. Galui? And if so, what are the main differentiating ideas between your

simulation hypothesis and his. I'm not aware of it. Okay. That's a rapid fire answer. Good. Raisa's hand asks a question that we touched on already. Dr. Kating would be great. If you asked him about his idea on the great filter.

Do you have anything new to say about the great filter? I guess we talked about that a little bit. I guess the part that I didn't fully answer was where we might now be past it or not, which I think came up in passing in one of your questions. And so I think we are past the most likely places where the great filter would be. I think there are like some very plausible candidates early on in the evolutionary trajectory of life. I mean, even from getting to any kind of simple replicator on an Earth-like planet, for aught we know, that might be really hard.

And then subsequent steps, such as from Prokaryote to UK, It looks like it took, you know, something like one and a half billion years here on this planet. You know, maybe that was just really unlikely to ever happen. And on, you know, tend to the power of 50 planets that reach procreate. Maybe like, you know, only one gets to the eukaryotic state. For all we know, that could be the likelihood. And we are just lucky to find ourselves on one of the planets where by some astronomical coincidence, it happened. And there are a few other places like where the great filter could be in our past.

I think there are significant existential risks for humanity in the future. But A, some of them are such that they wouldn't account for the absence of sightings of extraterrestrial civilizations, even if they happen. So superintelligence going wrong, say killing off all humans, it's kind of an existential risk, right? but there's no reason this super intelligence might not then build space colonizing devices for non-mon probes and expand through the universe. And so that kind of existential catastrophe wouldn't account for the absence of sightings. There are a few others that we could point to, like experiments in synthetic biology and increasing capabilities there, could for example make it too easy

to create various forms of weapons of mass destruction, if we're unlucky, if that's like how the signs unfolds. Those could pose existential risks. The problem there is more, it's a little hard to see how they could be so likely to destroy a civilization at our stage that they would kill off, say, more than 90% or 99% of civilizations at our stage. You would imagine every once in a while, there would be some civilizations at our stage

that would have its act together sufficiently to police those kinds of misuses from, say, synthetic biology. Like maybe, you know, in one civilization out of a thousand, you have some very rigid world state that controls what everybody does. And if some dangerous synthetic biology is discovered, they just ban anybody from accessing biolabs

without being continuously monitored by trusted. Like, maybe that would account, maybe 90% of our, if we are unlucky, like civilizations are 99.9.9%, but it's hard to see that at least some branches would get around that with some sufficiently like unusual forms of social organization. So for something to be a great filter, it wouldn't be enough that it kind of called 90% or 99% of all life trajectories that reached it, right? You really need something that maybe doesn't even let through more than like one in a billion or something like that. So some of these more contingent looking disasters wouldn't really be plausible candidates.

It still might be very worth worrying about from our point of view, like something that kills even just 20% of civilizations that reach our stage, like very well worth paying attention to so we can like route around it, but less likely as candidates for a great filter. Okay, H. Reedwork asks, is objective reality possible for humans? What does objective reality mean for artificial intelligence, if AI evolves and incorporates objective reality and pushes it into everything, would that squeeze out a human understanding of reality? I guess it's some cybernetic type question, Nick. If we do achieve, you know, these levels of superintelligence, will that have a back reaction, a kind of effect on humans themselves?

Yeah, how to interpret the question. I mean, it might be like one version of the question. I'm not sure that's what he was actually wondering about, but it's okay. I can imagine if we have AIs that become more epistemically reliable than humans as sources of information about the world. Would that then eliminate the opportunities for humans to form their own crazy beliefs about all kinds of things? If you have these kind of oracles officially vetted and confirmed like AIs, you can test them on a lot of different topics and wherever we can actually objectively check whether they were right or not, they are always right. at least way more often than any human. And then we can ask them about other areas where we can't easily check.

We might assume they're probably right on those topics as well, if it's the same mechanism producing the answers. So you might then have access to this kind of epistemically superior source of information about anything you might be wondering about. I wouldn't underestimate our human ability to ignore that evidence that we would get from the AI. I think we have ample resources to create our own illusions when we are motivated to do so.

But even in the context, I'm thinking now of the recent Google Gemini debacle where we had become so attuned to and accustomed to asking questions so much so of Google that we trust it more than any priest, minister, rabbi, Oracle, mentor, professor, that some of that caused a lot of cognitive of dissonance when you asked it to make a picture of, you know, a George Washington founding father,

and it came back with a, you know, black woman with a wig on. So do you think that, I guess, to take the question a little bit further than my audience member is asking, you know, I find myself speaking differently, at least to AI. And I know what kind of prompts will get me bad responses or responses that are stilted or cloying, as you said before. But so I'm adapting the way that I ask questions. And I guess if you did have, you know, 99.99% capable, you know, superintelligence, would there still, would there be an effect on maybe evolutionarily how we ask questions? You know, you don't have to really look it up anymore on a static site. You look it up on an LLM in your ear or a humane AI pendant. So I'm asking, I guess, well, I'm modifying my audience

member's question. Could it have a back effect on the, on the human species in a sense? Yeah, we shape our tools and then our tools shape us. It has to be the case that, say, you know, the Internet and email and social media shape the way we think and the questions we ask and our attention span, et cetera. And AI will also do that. I do think, though, the evolution of AI systems is very rapid. And so if it takes time to really change human modes of thinking and culture, like many years to sort of really sort of really. soak in and for children to grow up with it to really be shaped by it, there might just not be enough time for any particular generation of AI systems to really have its full ramifications

take place. That will be like a sequence of different impacts on human cognitive styles, but each one might only last relatively briefly before we have the next generation. And we shouldn't only think of these as like language models, right? Like we start to have more multiple things now where we will be able to have like these kind of conjured up video sequences of imaginary realities and then presumably 3D virtual environments and and then like increasingly capable and smarter knowledgeable AI systems that will shape us just by even if they're just like they're helpful to you as an individual that already would naturally shape your thinking over time. But even more so, if you imagine some of these are tools in the hands of various human

groups with agendas that specifically wants to shape your thinking. And so I think we're starting to see, but we'll see much more of these big AI systems as points of contention where big political pressures will build up to try to get them to say or refrain from saying various things. And different people in different groups will have different views about, you know, what they should say, but particularly about questions of black. politics or religion or ethics or culture that are contentious, right? And as these systems become more and more influential in society, it's natural that more and more people will start to care about exactly how they are trained. That's kind of... Do you have a few more minutes, Nick,

to answer? Maybe one more question. We can do. Okay, one more question. Yeah, buddy, 752 asks the following question. I'd like to hear his candid thoughts on the response in the Western world to a global, a global germ, perhaps in the context of utopian thinking and what went wrong or right, in particular, from your vantage point in Europe, what lessons could we learn from such an event? Yeah, what I would say to, yeah, buddy, is I think our civilization certainly didn't cover itself in glory in how it handled such an event in our recent past. And the fact that it's still a little bit difficult to talk about that you feel anxious about talking about it for fear of being kind of, I don't know, like penalized by the Google algorithms or something is quite outrageous. There should be some sort of post-mortem analysis of what went wrong and the herd mentality of people kind of rushing to come on board with one particular official narrative.

and then immediately de-platforming anybody who had a different view. Some of those views now in hindsight become clearly quite plausible. But one thing to be aware of is when you have a situation where there is, for whatever good, bad or random reason, some big filter in what opinions you're allowed to hear, you need to take that into account in your own thinking. Like you might only be hearing one side of the question presented with one side of the evidence. But if the other side is just filtered out, like analogous to the great filter, we don't, you know, then you might need to yourself try to make some corrections for that. So, yeah, there are so many different lessons that could be learned from these recent episodes of the global public health landscape.

But we don't really seem minded to learn any of them. it's like an amazingly big disaster with very little effort to try to prevent it from reoccurring compared to if there were like, say, a nuclear incident as to have been, like with Chernobyl and other things like or even smaller industrial accidents or if like a Tesla car, you know, runs over somebody like then there's like a massive effort to try to figure out or an airplane drops from the sky and 200 people die tragic. But then there is a systemic effort to figure out exactly what was responsible for, you know, the engine's failure. And like what was responsible for that thing and backtrack and learn and implement the lessons.

We don't seem to have had any similar learning experience from this recent event. Well, one thing I've learned a lot is how enjoyable and hilarious and delightful your writing is. I already knew that. But certainly Deep Utopia is no exception. Nick, I hope we meet in person, you know, before the singularity or before 2029, according to upcoming guest, Ray Kurzweil. He's updated it and maybe he thinks it's even sooner than that. It keeps getting near.

His books have to keep having new titles that make it more relevant. But I want to thank you so much for this wonderful book and another great conversation. And I wish you all the best. Thank you. Great talking to you, Brian. Ambition comes in all shapes and sizes. At First Citizens Bank, we roll with your goals because we're built for what you're building. Fit for your ambition, First Citizens Bank.