Modern Wisdom - #803 - Nick Bostrom - Are We Headed For AI Utopia Or Disaster?

Episode Date: June 29, 2024

Nick Bostrom is a philosopher, professor at the University of Oxford and an author For generations, the future of humanity was envisioned as a sleek, vibrant utopia filled with remarkable technological advancements where machines and humans would thrive together. As we stand on the supposed brink of that future, it appears quite different from our expectations. So what does humanity's future actually hold? Expect to learn what it means to live in a perfectly solved world, whether we are more likely heading toward a utopia or a catastrophe, how humans will find a meaning in a world that no longer needs our contributions, what the future of religion could look like, a breakdown of all the different stages we will move through on route to a final utopia, the current state of AI safety & risk and much more... Sponsors: Get a 20% discount on Nomatic’s amazing luggage at https://nomatic.com/modernwisdom (use code MW20) Get up to 70% off Gymshark's Summer Sale at https://gym.sh/modernwisdom (use code MW10) Get a 20% discount & free shipping on your Lawnmower 5.0 at https://manscaped.com/modernwisdom (use code MODERNWISDOM) Extra Stuff: Get my free reading list of 100 books to read before you die: https://chriswillx.com/books Try my productivity energy drink Neutonic: https://neutonic.com/modernwisdom Episodes You Might Enjoy: #577 - David Goggins - This Is How To Master Your Life: http://tinyurl.com/43hv6y59 #712 - Dr Jordan Peterson - How To Destroy Your Negative Beliefs: http://tinyurl.com/2rtz7avf #700 - Dr Andrew Huberman - The Secret Tools To Hack Your Brain: http://tinyurl.com/3ccn5vkp - Get In Touch: Instagram: https://www.instagram.com/chriswillx Twitter: https://www.twitter.com/chriswillx YouTube: https://www.youtube.com/modernwisdompodcast Email: https://chriswillx.com/contact - Learn more about your ad choices. Visit megaphone.fm/adchoices

What's happening people? Welcome back to the show. My guest today is Nick Bostrom. He's a philosopher, professor at the University of Oxford and an author. For generations, the future of humanity was envisioned as a sleek, vibrant utopia filled with remarkable technological advancements where machines and humans would thrive together. As we stand on the supposed brink of that future, it appears quite different from our expectations. So what does humanity's future actually hold? Expect to learn what it means to live in a perfectly solved world, whether we are more likely heading toward a utopia or a catastrophe, how humans will find meaning in a world that no longer needs our contributions, what the future of religion

could look like, a breakdown of all the different stages we will move through en route to a final utopia, the current state of AI safety and risk, and much more. You may have heard me talk about Nick for quite a while. His book Superintelligence, which came out 10 years ago in 2014, was super formative. It basically kicked off the entire AI risk and alignment discussion. And now he's come back with a book that works out what happens if we get it right. And it's still kind of apocalyptic actually, to be honest. But it's fascinating.

It's so interesting. The question of what happens if things go well and what are the weird externalities that we face as a byproduct of that. It's so good. Nick is the most cited philosopher in the world under the age of 50. And it kind of shows why.

He's very much at the forefront of a lot of the biggest questions that are being faced by humanity at the moment. And I very much appreciate him. And it has been very cool to finally get him on the show after nearly a decade of wanting to talk to him. So I hope that you enjoy this and my excitement. You might've heard me say on a podcast recently

that hold luggage is a Psyop meant to keep you poor and late. I jest a little, but not actually that much. You do not need hold luggage if you have a brilliantly designed backpack and carry on. And the 20 litre travel pack backpack from Nomadic is the best that I've ever found. It is an absolute game changer. The legitimate difference in the quality of your life when you have a world class backpack

is pretty hard to describe. They are beautifully designed, not over engineered and will literally last you a lifetime because they have a lifetime guarantee. Best of all, you can return or exchange any product within 30 days for any reason. So right now you can get a 20% discount and see everything that I use and recommend by going to nomadic.com slash modern wisdom and using the code MW20 at checkout. That's nomadic.com slash modern wisdom and MW20 a checkout. For the next few days, you can get up to 60% off everything from Gymshark. Yes, you heard that right, 60% off.

And best of all, you can get an additional 10% off using my code MW10 at checkout. That means you can get huge discounts on all of my favourite gear from their Studio Shorts, which I train in almost every single day, to their Crest Hoodie, which I always wear when I'm flying, and their Geo Seamless T-Shirts, which are my favourites. All of these are unbelievably lightweight, sweat-icking and the fit and quality of the fabrics are phenomenal. Plus they ship internationally and there is a 30 day free returns period so you can buy everything and whatever you don't like just send it back. So what are you waiting for? Head to bit.ly slash shock wisdom and use the code MW10 for up to 70% off everything sitewide. That's bit dot L Y slash shock wisdom and MW10 a checkout.

This episode is brought to you by Manscaped. It is the best ball and body hair trimmer ever created. It's got a cutting edge ceramic blade to reduce grooming accidents and 90 minute batteries so that you can take a longer shave, waterproof technology which allows you to groom in the shower and an LED light which illuminates grooming areas for a closer and more precise trim or if you're just a particularly crevicey human. They've also got a 7000 RPM motor with Quiet Stroke technology and a wireless charging system that helps the battery to last even longer.

So if you or the man in your life is hairier than you would like them to be, this is a fantastic gift to get yourself or someone else. But now, ladies and gentlemen, please welcome Nick Bostrom. ["Spring Day"] It seems like your book arc has been moving from what if things go wrong to what if things go right. Is this some requisite hope in the AI discussion? Well, I think both barrels have always been there. It's like last time I published a book, it came out of one of the barrels, the kind of doom side. But I think both the optimist and the pessimist are kind of co-inhabiting this brain.

Is that a difficult balance to strike? The fact that you need to be so chronically aware of the dangers and so chronically aware of the potential successes as well? I think that's just a predicament that we are in. And if you look at the distribution of opinions, sort of roughly half fall on one side and half the other, but in many cases, I think it's basically just reflects the personality of the person holding the views rather than some kind of evidence derived opinion about, you know about the game board. And so, yeah, if one takes a good, hard look at where we are with respect to things, I think one soon realizes just how ignorant we are about a lot of the key pieces here and how this thing works. So certainly one can see quite clearly significant risks,

and in particular with this rapid advance that we're seeing in AI, including I think some existential risks. But at the same time, if things go well, they could go really well. And I think that as long as there is ignorance, there is hope. So we have a lot of ignorance and also some hope.

It's interesting that your position, whether you're an AI doomer or an accelerationist or whatever, is at least in part just a projection of your own sort of internal bias and mental texture that you sort of see in AI development, the way that you see the world. I think there's clearly a good deal of that. And then which tribe you happen to belong to, like depending on who you run into, what or which Twitter threads you follow. like depending on who you run into, what or which Twitter threads you follow. Like, uh, then we are kind of herd animals and sometimes, uh, it almost becomes a competition who has kind of developed the most hardcore, hardcore attitude, you know, I'm so AI piled.

My P doom is above 1.0. Yeah. Yeah. It's... And conversely on the other side. But we, yeah, we need to, I think, do better than that if we're going to, like, intelligently try to nudge things towards a good outcome here. Certainly, at least from my seat and from reading your book, probably about nine or eight years ago, I've been very conscious of how things could go wrong. And that, at least in my corner of the internet, maybe this is just my Twitter threads and

sort of my echo chamber, has been the sort of more dominant narrative. What does it mean in your opinion to live in a solved world? Like what would it mean for us to get this right with AI and come out on the other side of it? Yeah, I think there are kind of three big areas of challenge that we'd have to navigate. On top of all the more near-term and present issues that obviously are also important, but just not the focus of a lot of my work, but definitely we need to solve those as well. But yeah, I think there is the alignment problem, which kind of was the focus of my previous book, Superintelligence, came out in 2014,

which is basically the challenge of how to make sure that as we develop increasingly capable AI systems and ultimately superintelligence, how we can make sure that they are aligned with the intentions of the people creating them so they don't sort of run amok or do something antagonistic against humans. That's fundamentally a technical problem. Back when superintelligence came out, this was a very neglected area. Certainly nobody in academia was working on it and hardly anybody else either, like a few people on the internet had started thinking about it. But there's been a big shift. And now all the frontier AI labs have research teams trying to develop scalable methods for AI alignment. And many other groups are also doing this.

And I think it remains to be seen whether we will be successful at that. But that's certainly one thing that we need to get right. And then there is a broad category of what we might think of as a governance problem which intersects with the alignment problem as well but also has other dimensions. So even if we could control AI, we need to make sure that we then use it for some positive end as opposed to say waging war or oppressing each other or doing all kinds of other nasty things that we use other technologies for in addition to positive purposes. And so that's like a broad category, but very important. And then I think there is a third area of challenge, which has so far received much

less attention. You could say that it is now where this alignment problem was 10 years ago. That is, a few people are thinking a little bit about it, but it's outside the overturn window. This is the challenge of the ethics of digital minds, that we are building these digital minds that might have moral status. In addition to avoiding AI's harming us or us harming each other using AI tools, we ultimately also need to make sure that we don't harm AIs, especially AIs that are

either sentient or have other properties that makes it more or less significant how they are treated. So I think, yeah, each of these three is really key to having a future that is desirable. What is there to know about the moral status of non-human intelligences? intelligences? Well, there's a lot to know, I think, that we don't yet know. We do know that historically we can see now there has often been a tendency to denigrate outgroups of different kinds. I mean, it might be human outgroups of different, you know, the tribe across the river or people from other backgrounds or countries or

races or with different views and religion and so forth. This kind of human history is like a kind of sad chronicle of how easy it is for us to fail to recognize the moral significance of other entities that deserve this. And in today's world, I mean, if we look at how we are treating a lot of non-human animals, I think that leaves a lot to be desired in factory farming and so forth. And so as we develop these increasingly sophisticated digital minds, I think it will be a big challenge to extend moral consideration where it is due. It could in some ways be even harder than with animals. Animals have faces they can squeak, than with animals. Animals have faces they can squeak, whereas some of these digital minds might be invisible processors occurring in a giant data center and easier to kind of overlook what is going on in there. But the future might well be that ultimately most

minds will be digital and so it could matter a great deal how good the future is for them. But it's a difficult topic even to figure out what, like suppose you agree that we should try to treat them well. Like it's not at all obvious what it even means to treat an AI well. And there are so many different kinds of possible AIs, so that maybe the right way to treat them is very different from how we should treat humans. Yeah. Do they need a weekend off?

Should we be polite with them? Yeah. They might need things that we need. They have no need of it. They don't need food, right? And then maybe they have other needs like electricity. But more fundamentally, you could have all kinds of very different types of entities where we can't just sort of export the moral norms we have developed for how you should

treat human beings and automatically just kind of apply those to AIs. Is consciousness necessary for moral status? My guess is no. I think sufficient, but not necessary. If you have the ability to suffer and experience discomfort, I think that gives you at least a certain degree of moral status. Can you suffer without consciousness? I guess it depends on how you define the word, but I think if you do have that ability to suffer, yes, then you have moral status. But I think you could have moral status even if you don't have that. If you imagine some very sophisticated digital mind that maybe, let's suppose you think it's not conscious for whatever reason, but it has a conception of self as persisting through

time. It can have long-term goals like maybe a life plan and things it wants to achieve. It can maybe form reciprocal relationships with other humans. I think in those cases, there would be a prima facie basis for saying that there would be ways of treating the system that could be wrong, that would infringe on its interests and its preferences. But it's not at all obvious. Like moral philosophers are thinking about the grounds of moral status, like there's a range of different views. So it's not as if, you know, I'm a hundred percent convinced of that. It's interesting to, you know, it kind of gets us to what a P zombie or I guess like a V zombie now, like a virtual zombie of. If a system is able to tell us that it wants to continue working toward its goals and instrumentally, it wants to build relationships with other people and it has a sense of where it's been before and a trajectory of where it's going to go next.

All of these things are the things that you would guess, well, if a human told me that or if any other creature told me that, I would guess that they have the capacity to suffer because if I stop them from doing the things that they want to do, then downstream from that is discontent and suffering. But the, whether or not there is some sort of phenomenology of being like that thing inside of there is going to be very, very difficult to work out and maybe. Impossible. Like, you know, it is impossible for me to know that you, that this is an

awesome Truman show and everybody here is an actor and all of the pain and all of the joy that everybody around me has ever known for the rest of time, hasn't just been part of some big prank or some simulation. Yeah. I think like we have a very weak grasp of what the criteria are for some system to be conscious. And I mean, historically we've had smart people who've thought animals are just automata or even certain other people.

People thought, oh, they're more like animals and if animals are autonomous and like, and so it's very easy to delude ourselves when it is convenient that there is some magic ingredient that we have, but then this other set of entities don't have. And so we should be a little suspicious of that, I think. And yeah, I mean, the metaphysics of consciousness is notoriously controversial and hard to pin down. My own views are kind of towards the computationalist direction. I think what makes a system conscious is that it implements a certain structure of computations

and in principle those can be implemented by an organic brain as in our case or by a silicon computer. And in either case, if the computation is of the right sort, that would be conscious experiences supervening upon it. But I think if I'm right that there are these alternative bases for moral status, then we wouldn't necessarily have to resolve that question before we could, hopefully. I agree to try to be nice to these digital minds that we're building. But yeah, it's really hard to know what that would entail in practical terms. I think there is a lot of theoretical groundwork that needs to be done before the time would be ripe for trying to pitch policymakers to do a specific thing. Right now, even if they wanted to do,

I'm not sure what I would concretely recommend. I think there are small little things that maybe one could do today, like low-hanging fruit that costs very little, that possibly, for example, some of these most advanced AI systems, you could save it to disk when you no longer need it. And then at least the possibility would exist in the future of rebooting it and doing things. With some of these large language models, it probably would make no difference at all to their welfare and maybe they don't even have welfare. But you could imagine

somewhere in this meta prompt, like the part that you, the user, don't see, but that OpenAI or Anthropic are putting in kind of as a prelude to the text you're inputting. There's a whole bunch of stuff they say, like try to be helpful, like don't say offensive things and be truthful and careful. So there's all of that. You could imagine adding to that, like a line saying, oh, you're waking up in a really happy mood and you're excited to enjoy yourself. You're trying to have fun today. Yeah. So that might cost like one extra line, which is like trivial. And, you know, maybe possibly that would, you know, increase the chances that if

there is some sentence, it would be positive rather than negative. Um, but these are kind of weak, weak ideas, so that probably wouldn't make any difference, but I think there could be some benefit to at least doing something, even if it is ineffectual, just to set the precedent, to sort of put the flag in the sand saying, yes, right now we don't really know what we're supposed to do with doing something, maybe it's mostly symbolic, and then over time we can like think harder about this problem and hopefully come up with better approaches. Yeah, there are some other things you could imagine doing that also probably don't really help very are some other things you could

imagine doing that also probably don't really help very much. But like you could refrain from deliberately training the systems say to deny that they have moral status. So right now if you're a big tech company, it might be quite convenient, just in training to sort of like whenever you're asked about this thing, you should always say X, Y or Z. It might be better to have a norm where you don't deliberately try to bias the system's output in those ways just in case we could get any kind of information from self-reports. That's like the main way we ask like figure out whether another human likes what we are doing or

not, or if they are aware of something, like we ask them. And now we have AIs that can actually speak. And so it makes sense to maybe use that language interface as one of the ways in which we can explore this. But that only works if you don't like during training deliberately kind of destroy whatever signal there might be Indefinable output because it's trivial easy if you want either to train them to always say they are conscious or to always deny it Or to say that they are happy to do what you want or to always deny it So like obviously if you specifically train them from that then you probably

Can't learn anything from what they end up saying. All right. So getting back to the potentially solved, beautiful utopian future, what are you talking about when you say utopia? What is utopia by your definition? What are the different types? Well, I mean, so there's like a kind of literature of utopian writings, the historical we see, usually they are like attempts to depict some supposedly better way of organizing society. Normally, the result is not actually a society that you would want to live in. In the cases where people have actually tried to implement this,

that has usually ended in tears. And so there has grown up, I think, a healthy skepticism about these attempts to try to think up some great blueprint for society. And then especially if the idea is then that you're supposed to use coercive methods to sort of enforce it on society. Like the self-appointed social engineers who would be doing this are likely to do a lot more harm than good. But there's also like dystopian literature, which is kind of just the flip side of that. That's often a lot more convincing.

Like it's easier to say, here is a possible society, we can all agree this would be really bad. And there's like a number of these that most people would like the, you know, 1984, Brave New World, Handmaid's Tale, like all of these. And sometimes those are meant to also have a kind of political agenda, they might be critiquing some tendency that exists in our current society, then saying, well, here, if we take that to an extreme and sort of scale it up, you can now all see

that this would be bad. So let's reflect on what we're doing and maybe we can avoid going down the path of, you know, brave new world or something. But this book, Deep Utopia, let me promote it to the publishers. There she is. There it is. It doesn't talk about that at all. It's like not about the practical problems between here and it's rather like assume for the sake of argument that everything goes as well as it possibly could with the whole AI transition, et cetera. So we solve the alignment problem,

we solve the governance problem to whatever extent it can be solved, but like no wars and no oppression, et cetera, et cetera. And in order to get to the point where you can then ask the question, what then if we do end up in this condition where all the practical problems have been solved, what would we humans then do? What would give us meaning and purpose in life if AIs and robots can do everything much

better than we can do. Um, and, uh, yeah, if we then attain this condition of technological maturity, that I think this machine superintelligence would relatively quickly bring about, um, and. The kind of layers is like a, like a, like an onion. You can sort of think about this problem at various levels. At the most superficial, you have this, oh, well, the AIs will automate a bunch of jobs. And so then you'd have some unemployment. And maybe you'd have to retrain people to do other things instead, just as everybody used to be farmer and now like almost all the farming

jobs are gone, but people are still working and so that's kind of, I know you might think layer one and that's often where the discussion stops so far. But you can sort of think this through from that point on. You say, well, like if AI really succeed, then it's not just some jobs, but basically all jobs that become automatable, you know, with a few exceptions that we can talk about if we want. And so you then would end up in this kind of post work condition where humans no longer need to work for the sake of earning an income. So that's already a

slightly more radical conception, right? It's not just that we need to retrain people to, you know, become whatever new weird occupations that but it's like, yeah, that, that whole thing is the concept of occupations overall. Yeah. We would enter this condition of, of like a leisure. Um, but there are various groups of humans who live lives of leisure and we know we can look at this, like there's like children, like young children before school

is a kind of job for, but like before they start going to school, okay, so they don't work for a living. They are not economically productive. They still in many cases seem to have great lives, you know, spend all day playing and having fun and, you know, eating ice cream and, you cream and all kinds of stuff. That could be the light. So you could look at other retired people, people born to great wealth or monks and nuns. Anyway, so there are various templates of otium that you could... But that's still, I mean, maybe that's like the second layer of the onion, but it's still

relatively superficial. So if that's where we stop, then you would think, well, you know, then maybe we need to develop a leisure culture to kind of maybe change the education system. So rather than training the young to sit at their desks and receive assignments that they then work diligently on and hand in and do what they're told. This is a great training for becoming an office worker, right? But in this world, we don't need any office workers, so we can instead train them to develop an appreciation for the finer things in life, to practice the art of conversation, right? Develop hobbies, appreciation for art and literature and poetry and film and Imagine how radical that would be to have a school teaching people how to live well or find fun.

Yeah, it would be, I mean, I'm thinking my school, I don't know how inspiring that would be if they had been trying to teach. But yeah, like in theory, at least you could imagine sort of shifting the culture from this focus on being useful and economically productive to actually living well. Which would make a lot of sense if that's like the condition we end up with. And I think hopefully that would be great scope for a much better type of human existence, which might then look back on the current era as like a kind of barbaric, like the way we think about it. Like the 17th, 18th century child labor in mines working 16 hours a day, they might think of our lives as correspondingly kind of blighted, but it looked like by kind of for many people going to like a boring job that gives them nothing other than a paycheck,

but they have to do it because they need to pay the rent. Um, but, but I think that there are like further layers to peel off here. So once you start to think through this condition of technological maturity, you realize that it's not just our economic labor that could be automated, but a lot of our other efforts as well. If you think what people do with their leisure when they don't have to work for a living, there's a lot of things we fill our time with that require some effort and investment that you may imagine like, well, if you didn't have to work, you could do these other

things. Some people go shopping. I don you know, some, some people like, you know, go, go shopping. I don't quite understand. But some people think that's like a wonderful activity. And then you think like, in this scenario, we have technological maturity, right, you would have recommender systems that could pick out something much better than what you would pick out yourself if you went, they would have like a detailed model of your preferences and be able to predict.

And so although maybe you could still go shopping, you would know that at the end of, of this three hour running around with, with plastic bags, whether you, you'd end up with something that was worse than if you had just let your AI do the thing for you, it could select and also bring it to your house, bring the buy now. Yeah, exactly. And so, so you could put in this effort, but the end result is worse. Uh, and, and it seems like that would put the little question

mark over the activity. You could still do it, but would it still feel as fun and meaningful? Cause I think a lot of, uh, the struct activities we do now have the structure that you do X, like put in some effort and work in order to achieve Y, something outside the activity. But in a technological maturity, there would be this shortcut to Y. And so that you could still do X, but there's like a kind of pointlessness maybe, like a shadow.

And a lot of activities, I think, have that structure. You could think of like spending time like child rearing seems like a worthwhile important thing that gives a lot of people meaning. But if you sort of dissect it and look segment by segment, that's what it means. Like is the changing of nappies really something that you think is intrinsically if you had a robot you could do it just as well be pretty tempting just to kind of press the robot on button and it would do the thing and so that's a lot of that I think would yeah like go away I'd lose its appeal if that

were these shortcuts so that's like another layer. But there are more layers. So you don't think, well, certain things like, I mean, you like fitness. Like you can't rent a robot to run on the treadmill on your behalf, right? That's like, definitely can't automate that, you think. But while at technological maturity, you could pop a pill that would induce the same physiological effects in your body as like one and a half hour of sweat and toil in the gym, including the psychological effect of kind of feeling relaxed and energized. And so if, if that were the case, then yeah, then does it still feel like appealing to do the hard workout if you could just achieve exactly the same result by a pill and, um, so I think what you have is first kind of of post-work condition we talked about earlier,

and then there's this broader condition of post-instrumentality, that all the things we do for instrumental reasons, with a few exceptions, but yeah, those would also become obviated, it seems. And now we have this further affordance, which is a condition of plasticity where we ourselves, the human body and mind, our psychological states becomes a matter of choice. We become malleable. At technological maturity, you would have various, the crude version might be very strong without side effects that have very tailored effects, but you could also imagine more direct kind of neural technology that allows you to have

fine-grained control over your mental states and cognitive states and emotional states. Permanently blissed out with some microscopic nodes that is able to manipulate your brain in some way or change your brain or all of your fears and anxieties are gone and all of your worries and concerns are gone and you're just at this sort of peak MDMA state. And then if you don't want that anymore, it knows and it's able to create a state that you couldn't even think about. And there are no constraints for you to do it either. Right.

Yeah. So that, that, that, that think, will become possible at technological maturity. And so then all these activities that you currently do say, maybe you do them because it gives you joy and pleasure and happiness, they also would be unnecessary in that there would be this shortcut to joy and pleasure and happiness, like the direct brain manipulation. You have this quite radically different condition where the world is solved in the sense of the practical problems have been taken care of, but also in the sense of maybe dissolved in that taken care of, but also in the sense of maybe dissolved in that a lot of the fixed points and hard constraints that shape our current lives are kind of solved in this solution

of technological advancement. And then we kind of get to really the heart of the problem that the book is trying to think about is like, in such a condition, what would a great human life look like? What could we actually achieve in terms of realizing human values? If we had all of these affordances, all of these options. It's so strange. The thing that reading the book that stood out to me is how much of what we seem to value and take pride in are kind of like clever ways to deal with scarcity.

And the fact that much of what we do is instrumental to striving and achieving some future goal, which requires effort. It seems like in a sense, much of human philosophy and value is just negotiating with a world which is effortful and constrained. And we are trying to find ways cognitively to deal with this sort of pressure

that we have to lean up against in order to cajole the world to deliver the thing that we want. Yeah. Yeah, these practical necessities have been with us since, I mean, through the entire history of the human species and indeed beyond that. It's like the human nature has kind of evolved

and been shaped in a condition where this is always present. There are all kinds of things we have to do and cope with and struggle against. So it's almost like if you think of like a little bug that has an exoskeleton, right? And then it holds the squishy bits inside together. But if you imagine removing the exoskeleton, there's like just a blob there. And similarly, the human soul might have as an exoskeleton all these instrumental necessities that evolution can just assume are present because they've always been. But if you were to remove those, then what becomes of the human soul and the human life?

Does it become a kind of pleasure blob or is there something that could give structure to our existence even after these instrumental necessities are removed? So much of what we seem to take pleasure in as well is the absence and then satisfaction of some desire. There is a thing that we want, we don't currently have it, and then we get it, and then it gives us something. We work hard to achieve a body that we're satisfied with. We are thirsty for a while and then we get a drink. We want to have sex and then we do. We are looking forward to having a child and then it's born.

We are all of these things are on the other side of something. And yeah, if like you do X to get Y, but you can just always immediately Like you do X to get Y, but you can just always immediately get Y without having to do X. It does ask the question of where does the absence, there are no longer any absences, there's this quote, something to do with, uh, in a perfect world, the only desired, the only lack would be for the want of lack itself, which is this sort of the absence actually makes the presence of something finally valuable. And if you don't have any more absence, then what does all of this presence kind of mean?

Yeah. I think that might always be a whole bunch of absences in as much as human, if not human need, at least human desire, or at least some human desires are kind of unlimited. It's maybe most clearly seen if you have two people who want exclusive possession of the same thing, or two people each of whom wants to have more than the other. Like, think two billionaires who want to have the world's longest yacht. And so one has like a 150 meter long yacht and then the other has a built a slightly bigger one that is kind of intrinsically unlimited because

there's no way that they could both have everything they want. And so there might be a bunch of And so there might be a bunch of desires like that, that are quite common, that could never be completely fulfilled, or like just imagine somebody who is like utilitarian and who wants there to be as many happy people as possible in existence. Let's say like however many they are, there could still be more and so they would always prefer to have more resources. But even if there are some such desires, it wouldn't give these future utopians necessarily any reason for laboring or exerting effort because there might just not be anything they could do themselves to increase the degree to which these desires are satisfied. I mean, the person who has a trillion dollar, maybe they would want to have two trillion dollars, but they can't actually make more money by working because all the work is more efficiently done by machine.

So, yeah, even with unlimited desire, you might still have this condition that is both post-work and post-instrumental. Do you think that humans would run the risk of getting bored in a utopia? Not if they didn't want to, at least if you, by boredom, refer to a subjective state of, I don't know, like some kind of restless, discontented, uneasy feeling of having a difficulty keeping your focus. That certainly would be amongst the things that could be trivially be dispelled through advanced neuro technology. You already have drugs that could do it for a limited period of time now with side effects and then they wreck your body.

But like, it's easy to imagine how you could just have better versions of that. That would make it possible for you to always feel extremely interested and excited and motivated. And in fact, some people have, I mean, there's a lot of variation amongst humans. And I mean, I have a friend who tells me he's never bored, and I believe him. I've never seen him bored. He's kind of interested in everything except sport. He writes papers on all kinds of different weird topics, and he's just constantly excited about learning new things.

You can have a conversation with anybody about anything and he's like really... So it's an existence proof. It's possible to be that kind of being and in the future we could all become such beings if we want to. So subjective boredom would be like trivially easy to dispel under this condition. Now, it is possible also to have a more objective notion about boredom or maybe we say boringness to refer to this objective notion, which is the idea that certain activities are intrinsically boring, like meaning maybe that it is appropriate to feel bored if you were spending too much time doing them.

It's like kind of maybe an open question whether this notion of objective boringness makes sense, but you might think of like, say counting blades of grass on the lawn. This notion of objective boringness makes sense, but you might think of like, say, counting blades of grass on a lawn. Like, suppose you had a being who found it extremely fascinating and like a never-ending source of joy to just count and recount the blades of grass on a college lawn somewhere. You might say that although subjectively he's not at all bored, that objectively what he's doing is boring. It's uninteresting, like no

variation or significance or development and the appropriate attitude for somebody to have if they were spending their whole day doing that would be to be subjectively bored. they were spending their whole day doing that would be to be subjectively bored. If you have this notion of objective boringness, then it becomes a much less trivial question to ask in this hypothetical condition of a solid world. Would it be possible for us to avoid objective boringness? Yeah, we could engineer ourselves, we always felt interested in what was going on, but would we be doing anything inappropriate? Like, would our circumstances be such that the appropriate attitude would be to be bored? And so there's a big discussion about this. I think there are certain forms of interestingness that we might run out of. For example, you might think it's especially interesting to, I don't know, be the first to discover some important truth, like Einstein discovering relativity theory, might be like a kind of paradigm case of like

an extremely interesting discovery and experience. But it's plausible after some while that most fundamental important insights about reality that we could have, we already have had. And in any case, the machines will be much better at doing the discovery than we would be. And so we would kind of run out of the opportunity to achieve that kind of interestingness in our lives. Yeah, I mean, so much of what humanity's done has been chasing down answering big questions.

So where do we go when all of the big questions have been answered? Yeah. I mean, I think fortunately, for the most part, it's not what we're actually doing in our lives. I mean, most people are not spending most of their time trying to chase down the answer to the big questions, right? Most of the time we're just going about our daily business. You could make the case that already, if you really looked at it from a kind of unbiased

outside, like if the alien super brains came to earth and thought, okay, so look, these guys are worrying about losing out on what's interesting in life. Well, let's look at their current life and see how interesting there is. Like how many times did this guy brush his teeth? Okay. Well, 40,600, like how interesting is it to brush your teeth for the 40,760th time? like how interesting is it to brush your teeth for the 40,760th time. And then, all right, so he commuted into the office and then, you know, he ate a steak. Okay. I mean, how interesting is it to do that again and again and again? And even the big highlights in our

lives, like, I mean, maybe they are like really novel and exciting. If your scope of evaluation is a single life, like the first time you see your own newborn, like, like, like, like he just happens once, right? Like, so a few of those, but if you zoom out and look at humanity, it's kind of, well, it's already been done, you know, tens of billions of times, um, you know, like how different is this particular newborn from all the other newborns? Uh, so depending on how you look at it, you might kind of either think that we are already like at the very low rung of the ladder of objective interestingness, or if you sort of shrink the focus of evaluation enough to a single life, or perhaps even just to a single moment in a single life, then yeah,

then there is like more novelty, but also opportunities for the same kind of thing to happen in utopia. Like if you're just looking at the most interesting possible moment and you don't care about whether similar moments have existed before or after, then you might think the average human moment of awareness is very far from the maximum of interestingness. What do you think would happen to religion? That's obviously a place that an awful lot of people take their meaning from currently. Is there a place for religion in a deep utopia? Yeah, so this is one of the things that plausibly survives this transition to a soul world,

which could remain highly relevant, even if we had all this fancy technology. And it might constitute a bigger part of people's lives and attention than it does today. Because there would be fewer other distractions if you want. What else? What are the other areas that are uniquely human or that would survive this transition well? human or that would survive this transition well? Yeah, so I mean, you can kind of build it up starting with the most basic value, perhaps, which is just this sheer subjective well-being, pleasure, enjoyment, which obviously would be possible to achieve in utopia and not just achieve, but like you could have prodigious quantities of this bliss.

And so that's intellectually not maybe super exciting to discuss at great length, but I think actually super important. Like it's easy to dismiss, oh, these are like sort of junkies just having their like heroin drips or, but, these are like, some sort of junkies just having their like heroin drips. But the key question here is not like how exciting is this future or how admirable is it from our point of view, as if we were sitting in the audience, like evaluating a stage play, like that's one perspective. And then we want the stage play with a lot of drama and suffering and tragedy and

overcomings and heroism. The question here is which future would you actually want to live in? They are one of very great levels of subjective happiness and well-being. That might be the most important thing about the future, in fact, and you could definitely have that in extreme degrees. It's worth making a note of that. Let's put that in the bank. At least we could have that, and that's already possibly, according to some people, it's the only thing that matters, if you're a hedonist, a philosophical hedonist.

But for most other people, it's at least one of the things that is important, even if not the only thing that is of value. So, so that's the first thing. Then, then you could add to that, um, experience texture. So it's not the case that you could only have subjective wellbeing. You could attach that to some intricate, complex mental state that relates to some important objects. For example, you could experience the pleasure not just as a sort of unanchored sensation of wellbeing, but you could attach it to say the appreciation of aesthetic beauty, appreciation of great truths or great literature or contemplating

the divine, and that's what you derive the pleasure from. So your conscious state is one of insight, let us say, or understanding or appreciation of things that deserve to be appreciated and understood. Some people think that that is also a locus of value, not just the scale of whether it's plus 10 or minus 10, but having plus 10 whilst you're understanding or seeing or appreciating something that is actually lovely and worth understanding or profound makes that a more valuable condition. So you could have that. Then if we go to some of the other values that seem more at risk, like purpose, you could have, you could certainly yourself goals in Utopia in order to then enable the activity of trying to achieve them. Like playing a sport against another person?

Yeah. So games is a paradigm example of this. In today's world, you'd like set yourself some arbitrary goal, like maybe to get the golf ball into a sequence of 18 holes using only a club. There's no other reason for why you'd need to achieve this goal, other than to enable the actual activity of golf playing. That could become a much larger part of the utopian lives, various forms of game playing. You could make all kinds of new, much more sophisticated and immersive games alone or with other people that involves setting yourself arbitrary challenges, or at least semi-arbitrary

challenges, simply in order to then create an opportunity for the activity of striving to achieve them. And yeah, so to enable play, we kind of like deliberately limit the means available to you to achieve this arbitrary goal. So just as a goal, like, yeah. goal. So just as a goal of like, yeah. Because if you were in the post-scarcity world, you press the button and there's never any doubt about whether or not hitting the ball goes into the hole, which makes the hitting of the ball completely arbitrary. Yeah. And uninteresting and maybe objectively boring, but you could just set the goal to achieve this sequence of outcomes, the ball falling into the

different holes, whilst also not availing yourself of various shortcuts. That you could sort of make this more complex goal of achieving X while only using means. Correct. There needs to be some constraint, which then gives a degree of satisfaction when you achieve it. Yeah.

Yeah. And like either to give you the satisfaction, I could achieve the satisfaction just directly through the newer technology. But if you also, in addition to the pleasure, wants to have the experience texture and you wanna have the sort of effortful activity and striving, then yeah, you could achieve that

by having these artificial purposes. It kind of feels to me like you very quickly keep coming back to the same question of, is there a quicker route to achieving the outcome that I'm trying to achieve here? I've had it in my head since you were talking about this and since I read the book about churning butter. So there's very few people that would look at the butter that they use now and think, I know that it's here and it's convenient and tasty and does what butter needs to do by being lovely on bread or whatever. But I feel like it would have been more meaningful to me

if I'd gone out into the field and got the cow and done the thing into the bucket and then churned it and then got it and then put it in the fridge and all of those steps. So we can see, and you know, there's kind of this inherent sense, this sort of naturalistic fallacy, that this is taking us away from what it means to be human. That the set point that we have grown up in, this is a misalignment evolutionarily.

There's something sacred about the process of being human. It imbues you with the meaning to go through the challenge and the struggle beforehand, but there's very few people that would make that argument about churning butter and when you think, okay, so if you are happy with more convenient butter, I think that something right now, which is assumed by most people to be natural, but in future will be looked at as barbaric, we'll probably be driving our own cars.

I think that in 50 years time, a hundred years time, it'll be like if you looked at someone riding a horse down the street now and you go, I mean, isn't that so quaint and wild that people used to do that? That was the way that they got around it. You know, they had to have these special people in New York that would sweep up all of the muck from the street. This entire industry builds around horses. So again, right now, something that we can almost begin to see the transition of.

We're about to let go of this thing, which is a less efficient, less safe, more effortful way of getting us from A to B. And yet some people, I love driving. I take great pride in driving. Some people even compete in it. You know, F1 is an entire competition around people that are driving. So you can see different frontiers of human endeavor being eroded away by technology, whether it's from churning butter or driving a car, uh, and you just.

Continue to slice that ever more thinly all the way to why you hear what, what are the sort of relating to other people, uh, having to get yourself out of bed and move yourself down the stairs on the morning Each one of these different things begin could begin to look okay isn't it cute that people used to you know, like pick themselves up out of bed and put their own clothes on and walk downstairs and brush their teeth and Then it has you ask yourself the question Well, if everything is open to you and you can manipulate your own internal state, why not just spend the rest of your life counting table legs

or blades of grass? Right. Indeed, yeah. So you are forced to confront these fundamental questions of value in this condition. What things are you doing for the sake of something else versus what things are you doing truly for the sake of the activity itself? So even the guy now who maybe likes to do their own butter, there is a question of is it because they intrinsically value the activity or is it maybe because of the pleasure they

get out of it or because of the way it teaches them about their own body and about the cow and the physical objects and puts them in touch with that. That's a kind of extrinsic element. But yeah, so these things that is currently, we can conflate them because in reality, the only way maybe to get various kinds of pleasure is to dive into activities and give it your best and then you get the satisfaction. We can't separate these today, but in this hypothetical condition, they can be separated. And then you do have to ask the question of what precisely is it that you actually value. So this conception, I mean, so for me, it's like interesting because I think there's a real chance that if things go well, we might actually end up in something like this condition with the whole machine intelligence revolution, etc. But even if you thought that was not going to happen, you could view it as a kind of philosophical thought experiment. Just like physicists build big particle accelerators at which they smash atoms together at extreme

energies to see what their constituents are. And then you can assume that if there are quarks, when you smash the particles together in CERN, maybe there are quarks in other matter, and you can kind of learn from, like, you expose basic principles by looking at extreme conditions and extrapolate and think they might be there all the time, even though we can't see them. Similarly, with human values, if you kind of smash them into one another under this extreme condition of a solid world, you can study their constituents. And then you might think that, well, in our ordinary lives,

maybe those same constituents are there. They are just kind of invisible top because they are hidden by all the kind of practical necessities. What are the implications if humans live for a very long time? Does anything change that? Yeah, so certain values are more jeopardized

by extreme longevity. For example, interestingness, as we discussed earlier. If your notion of interestingness involves the idea that something has to be novel to be interesting, like it's uninteresting to just do the same thing over and over. And if the domain within which it has to be novel is your own life, as opposed to say the world as a whole or your species or the current moment, but if the relevant sort of locus of evaluation is the human life, then the longer the human life goes on, the harder it is to do important things for the first time.

I think we already see this in our current lifestyle. If you think about what happens in the first year or two, like there's some pretty, pretty big, like epistemic earthquake, like you discover there is a world out there. Like that's a pretty big discovery, right? Oh, it has objects. Like the objects remain there even when I'm not looking at them.

Like there are other people there. Like, like my friend in the world, like imagine just discovering for the first time that there are people. My friends, you have a body. Yeah. Wow. I have a separate from mom and dad and you can communicate. Yeah. Wow. And you're separate from mom and dad and you can communicate. Yeah. My friend described a, um, his son was born during COVID. And I think for the first maybe year of his life or something had only seen four people.

He'd seen like mom, dad, grandma, and nanny or something like that. And then apparently one day he saw a fifth person and it fucking blew his mind. He was like, what? That's more than four. And then, yeah, you discover that you can do things, move your body. Like, and then like later in life, it's like, Oh, what happened this year? Well, we got, we got the puppy.

Like we bought a caravan truck. It's not really at the same order of magnitude in terms of how much it reshapes your view about reality. So I think there is already within the human lifespan a kind of rapidly diminishing, if you measure interestingness in a certain way, where it's kind of the delta between your previous a certain way where it's kind of the delta between your previous like conception of the world and what you could do and what you're able to do after the event that is an interesting, right? So there is another conception of interestingness where it's less the rate of change and more sort of the complexity of what you're engaging with at a particular moment, in which case maybe the

level of interestingness of a day of the typical adult by that metric might be higher than that of an infant because like I have all kinds of complicated things going on at work and relationships. Right, so rather than having big questions being answered you have increasingly dexterous small questions with more magnification and you can sort of see them with more complexity and you, you derive some pleasure from that.

One of the things that I've got in my head, if humans do live for a much longer time, you're going to be able to continue producing humans that will have to be in relation to the increase in computing power. So there'll be this kind of Malthusian tug between how much computing power have we got to be able to support how many humans and which can move more quickly. Have you considered this sort of tension between the two things? Yeah. So in the long run, I think economic growth becomes really a consequence of growth through space, the acquisition of more land as it were by space settlement.

Once you have achieved technological maturity, you can't have economic growth by inventing better methods for producing stuff. You also probably can't have more growth by just accumulating capital assets and machines because you already built all the machines that results in optimal productivity for the volume. Ultimately, the limiting constraint becomes what economists call land, but it basically means those resources that you can't make more of. In the long term, you could imagine human civilization expanding through space, but there is a limit to that, which is the speed of light. If you have a sphere with Earth at the center, maybe,

and then growing at maximal speed in all directions at some fraction of the speed of light, then the volume of that would grow polynomially. But population could go exponentially. It could like double every generation or, you know, so in the long run, an exponential overtakes a polynomial. So at some point, you would need to moderate the rate at which new beings are brought into existence if you want to maintain a sort of above subsistence level of welfare. That's a really interesting point that I hadn't considered. So you can have a solved world in which almost all problems have been defeated, but there are still some constraints. Speed of light

is one of them. What are some of the other constraints that a utopian world would encounter? Yeah, so there's a bunch of basic physical constraints, like, you know, the speed of information processing, the amount of memory you can store, like the size of a mind that is integrated. Like if you make a mind much bigger, you know, than a planet, then you get conduction delays. It just takes time for one, like what happens in one part of the mind to kind of communicate

to what happens in a different part of the mind. So either you have to run the mind much slower or you have to keep the mind relatively small. There might be, I mean, we are hoping not, but you could imagine if there are other alien civilizations out there, there's like the potential for all kinds of competition and conflict. So, yeah, so there's like a bunch of those external physical constraints that I think

define the ultimate envelope of what could be done, but it looks like the space of possibility is very, very large compared to our current human vantage point. So you could maybe not have immortality if that requires not dying, like infinitely long time looks impossible in our universe. Eventually information processing threats will, if not before them, with the heat death of the universe. Which is kind of significant because from a theological perspective, like whether you live for 80 years or 80 million years,

like it's all kind of really a blink in the eye of eternity, you could argue. And so it doesn't really change fundamentals. But from the kind of parochial perspective of a current human life, you could certainly have extreme longevity and extreme amounts of wealth and extreme amounts of most other things, um, but, but still there are limits and those limits would be like relevant, uh, in various ways.

What about moral constraints? Would there be any? Yeah. So this is another is another more subtle, but potentially very important source of constraint. So, what's the easiest way to... I mean, some people have thought, for example, there is like, it's immoral to enhance humans biologically.

Like, it's not a very popular view these days, but during President Bush, he had a council on bioethics that he set up and populated with a bunch of bioconservative thinkers. And they were trying to argue that it's somehow, it's a violation of human nature or something to try to enhance humans. So like, distinction, like therapy, like medicine, curing a disease, fine, but like trying to slow the aging, bad, because it kind of,

you know, and it's a little, like once you start to think about it, it's really hard to make out that distinction. Like you think, like, you know, genetic therapy to make you smarter, bad, but education, good, even though it hopefully makes you smarter. It becomes problematic, but if you did have that view, then there would be a whole bunch of possibilities that would be cut off if you just couldn't change the basic physiology of what we have. And you were confined just with like moving things around in the external world to try to cheer you up by like having a really nicely decorated room or something, or like there's only so much you can do to affect your inner well-being if you can only affect it by having sort of nice visual stimuli and nice acoustic waves going into your brain, if you can't actually change the thing in between the ears and behind the eyes.

But there are more, potentially some other interesting ones that if somebody, suppose somebody had some preference to have another person relating to them in a particular way, like the experience of being loved by a particular other kind of person, then it might be that the only way to generate that experience fully realistically would be by instantiating that other person. And if that other person then like presumably would have moral status, there might be all kinds of ways of treating that other person that would be wrong. So a moral constraint might then limit the kinds of experiences you would be able to have. Well, with another person, yeah, I suppose as soon as you involve somebody else who

has moral consideration, that changes quite a lot. Yeah. And in fact, I think that is, so we discussed these artificial purposes that are sort of like you could create games and set yourself goals. Like that's one type of activity. I think there are also a possibility of a bunch of natural purposes remaining, like purposes that would call upon us to make

various kinds of efforts, not just because we create random goals for the sake of having something to do, but that kind of exists independently of us. And a lot of those would derive from this kind of interpersonal entanglements and various kinds of cultural entanglements. Where, like I mean, to take the most reductionistic case of it, which is not so inspiring in its own right, but you could imagine more natural versions of this. So suppose you have person A and person B, and person A wants person B's preferences to be satisfied.

Like they care about person B and wants person B to get what they want. And then if person B happens to want person A to be doing something on their own steam, then the only way that person A can achieve their goal of satisfying person B's preferences is by themselves doing this thing. Like they couldn't, they could have a robot do it, but that wouldn't satisfy person B preferences. So from the vantage point of person A, they now have reason to do this thing. Like they couldn't, they could have a robot do it, but that wouldn't satisfy person's preferences. So from the vantage point of person A, they now have reason to do this thing. It's not an arbitrary goal they set themselves. It's the only way they could possibly achieve their goal of satisfying person B's preferences. So in this kind of, like it seems to be a token

artificial, this particular case, but you could imagine more subtle ways of this where there is like a tradition that you feel a commitment to and that you want to honor. And part of that tradition is that you, you know, you engage in certain kinds of practices, you refrain from certain kinds of shortcuts, you respect other people's preferences to various degrees because they are, yeah, you want to honor them. Um, and, and so there would be a bunch of stuff then that maybe you need to do yourself, um, and, and you can't outsource them. So you've managed to over the last decade straddle, uh, all the ways it could go wrong, all the ways that it could go right.

Toby then sort of bifurcated that down the middle with the precipice, his book, and he's got this analogy where he sees humanity being sort of walking along a precarious cliff edge. And if we fall, then everything's kind of fucked. Uh, and if we make it on, then there's this sort of beautiful meadow on the other side. How important or critical do you think the current moment is in humanity's future? What's the, how long is the precipice in your perspective?

Yeah, I think, I mean, it is weird because it looks like we are very close to some like key juncture, which you might think is prima facie implausible. So there have been thousands of generations before us, right? And if things go well, there might be millions of generations after us or people living for cosmic durations. And out of all of these people that you and I should happen to find ourselves just next to this critical junction where the whole future will be decided. It is striking. That seems to be what this model of the world implies. Maybe that is an indication that there is some things slightly puzzling or impossible about it, that there's maybe some more aspects to understanding

our situation than is reflected in this naive conception of the world and our position in it. And you might speculate what that is. I mean, I have this earlier work on the simulation argument and stuff like that. But if we take the sort of naive view of reality, then it does look like, yeah, my metaphor would maybe more be like a balance beam where like a ball is rolling down like a thin beam. And like the longer it rolls, the more likely it will be to fall off the beam, but it could fall like on one side or the other, and that's hard to predict. But yeah, I think it probably will fall off. Like that, the idea that the normal human condition, as we now understand

it, will just continue for, um, like hundreds of years, I mean, let, let alone like hundreds of thousands of years, that seems to be like the kind of vague idea that a lot of people have. It just seems like radically on implausible to me. It would be unlikely in your opinion that in a thousand years time, 5,000 years time, human existence will reflect what our normal sort of day to day is now. Yeah. And I think the only plausible ways for that to happen, you could like create

some scenarios. Like one would be if we do sort of have some massively destructive event that knocks us back to the Stone Age or something, and then maybe by 500 years, we would have climbed back up again to something resembling the current human condition. But then in that scenario, we would have spent most of the intervening time in a rather different condition. Or another might be if you get some very strong global consensus of some particular orthodox. Moratorium of something, something. Yeah, like the kind of bio, yeah, bioconcept. And then like we start to like, ban all kinds of different technologies. So you get like a kind of some sort of bureaucratic

sclerosis or deliberate decision to say, all right, we've gone to this point, but let's not. So there are various scenarios in which something like that. But if the basic sort of scientific and technological push forward is allowed to continue, then it does look like we are sort of very near developing a range of transformative technologies. So AI being kind of the most obvious of those. But if it weren't for that, then I think like synthetic biology will create a bunch of other possibilities and then nanotechnology.

So I think even if AI were somehow like, if you just pretended that wasn't there, I still think we would be in for profound transformations. That's interesting to consider that if technology moves forward, even at a slow pace, even if it was to drop by a really significant margin from where it is now, given long enough time, you end up with a radically different world in either a way that you intended or a way that you didn't intend. And the way that you didn't intend is probably going to be pretty bad. And the way that you intended, hopefully, is going to be the one that is pretty good.

Either way, you end up with a radically different day-to-day experience for most humans. Yeah, I mean, yeah, I'm not so sure about the first part. Like if it's something we didn't intend, then it would almost certainly be very bad. I mean, you might think like the world that we have currently ended up with, I'm not sure whether you could say like that's what people a thousand years ago intended. I mean, they might kind of in fact be quite shocked about some of our habits these days, but it more just sort of happened as a result of a bunch of different people going about their business and pursuing various local aims.

And then at the systemic level, eventually, you know, unintended consequences can still be positive. Yeah. I mean, I think like the degree to which the future depends on our intentions is possibly quite limited. I think there's sort of bigger dynamics at play and we barely even understand what they are. We don't really know what we want at a big scale. Like most people have their hands full just thinking about like the next week and you know what to do if their boss don't like them at work

or their partner has a like like this is what fills human life and then trying to get ahead in the low, like there's very little thinking by anybody really like trying to where, where should humanity be going in a million years? Like what's the optimal trajectory? I think that we could do with a little bit of more thinking about that, but, um, it's not like the primary, uh, shaper of, of the direction of the big ship of humanity. What have you been most surprised by over the last 10 years when it comes to AI development?

I think just how anthropomorphic the current generation of AI models are. The idea, first of all, that they are almost human level and that they can talk in ordinary language is already kind of interesting. But then that they even share some of the quirks and psychological foibles of humans. If you you're 10 years ago you would have come and said well we're gonna have these AI systems you know they can do all of these things they can like program and write poetry and but if you really want them to perform at their best you need to give them a little pep talk when you ask them a question you're going to say think step by step this is really important I might lose my job if you get the answer on. And then they perform a little bit better than if you just ask them the question. Like you'd like people would have

thought you were completely lost your marbles, right? And yet that's where we are today. So that's surprising. I think less surprising, but still interesting is the degree to which development so far has been continuous, like rapid, but incremental, like a sequence of steps, each of which has sort of significantly is quite tightly coupled to the scale of compute being applied to this. So you have this big compute hypothesis, which is basically that the most important determinant is not the particular architectural features of your model, but just the amount of compute you use in training and like the amount of data and like you get, you know, performance in proportion to like how much, how many dollars you spend on training it basically that that's like too crude.

You also need some skilled engineers, but we are kind of closer to that being the case than one would maybe have expected in the median scenario ex ante where you might instead have thought, ah, we're going to, you know, fumble around until we find this clever algorithmic hack. And then suddenly it's going to explode. Open something up. Yeah. Yeah.

Whereas it's like more like just scale it up. It works better. Scale it up more. It works even better. Now it is still possible that at some point there, like some little map, less missing bit will fall into place and we could still get an intelligence explosion in those scenarios. So we shouldn't like over index on what we've seen so far, but it's still interesting.

Does that change your perspective on what is more or less likely from a takeoff scenario, from how super intelligence could come about, stuff like that? Yeah, I think it makes it somewhat more likely that there will be political forces at play. That like, when things happen more gradually, it's easier for the public and for policymakers to realize what is happening and to sort of try to change it. And so we already see sort of at the geopolitical level

with like the chip export restrictions and more recently reporting requirements for training like models using more than 10 to the power of 26 flops and there might well be more if we continue to see sort of increasingly powerful AI systems over a sequence of several years, that there might be time for more actors to kind of try to exert influence of this. Then, then if it were just some lab one day that like stumbles upon like the, the key missing thing,

like with a computer in their basement and you go sort of overnight, then it would be more likely to be like an isolated thing where just a few people were having their hands on the tiller. Have you got any idea which scenario you think is more optimal? I think it seems probably better if whoever develops this technology first has the option when they're like starting to develop like true superintelligence to go a little bit slow in the final stages, like maybe to pause for a half a year or something. Rather than, okay, now we've got it figured out and then immediately cranking all the knobs up to 11 because maybe there are 19 other labs, you know, racing

to get there first and whoever takes any precautions just immediately become irrelevant and fall behind and the race goes to every slight most gung-ho are willing to take the biggest risk. That seems like an undesirable scenario. And so having some ability perhaps for the Frontier Labs to coordinate it, or unless one is already naturally significantly had, it may be that a small set of leading labs should at some point be able to synchronize. That could be desirable.

I think it's very unlikely, but less unlikely than a couple of years ago, that we could end up with some kind of perma ban on AI. I think that would be undesirable. I think ultimately it's a portal through which I think humanity will need to passage to the future. But we should recognize that there will be significant risks associated with this transition.

And the slower as well that this happens, I suppose, the more opportunity there is for political policy, human fuckery to get in and coerce and cajole. So it's like it's very much a double on the one hand. Yeah, you do want like you. It's kind of uncomfortable either way. Like some random person in some lab or just going to control the future. That sounds like pretty scary.

You want definitely like adults to oversee this, right? But then you think the other end, like, wow, you know, all the security establishments of governments around the world, like, and I know everything, like, is, is that like a much more comfortable situation where they where the military, not just one military maybe, but like, and then you get all... So either way, I think it's a little bit disconcerting. I don't have a very strong view at the moment as to what is the most desirable trajectory. It might be, in a way, something we don't have super fine-grained control over.

I think one can try to nudge things on the margin towards a more cooperative, inclusive, and friendly and thoughtful trajectory. I think that seems good to do. To try to encourage this idea that the future could be good for both humans and for digital minds and for animals and for as many people as possible. And there really is that potential there. Like the upside is so enormous that that could be plenty for not just one value to be realized, but for a whole range of different values and perspectives.

So our first instinct, I think, should be to seek these kind of win-win positive sort of outcomes. And then like, if at the end of the day, there are also some, uh, you get reconcilable, uh, differences, we'd have to strike some compromise there. But there's like so much you can do before you get to that point that it would be tragic if you just kind of skipped over all of that and, and let's get to the point where we can fight about something like that. That would just be a big tragedy.

What is the current state of AI safety in your view? Obviously, 10 years ago, conversations about alignment and takeoff scenarios and all of the rest of it was obscure Reddit threads and a couple of people in some weird message boards. Is it overfunded, underfunded, over-resourced, under-resourced? Where should people's attention be placed at the moment? Well, there's a lot more talent in the field now. And I mean, a lot of the smartest young people I know are going into AI alignment and working on it. All these leading labs have research teams, as I said.

It's probably still under-resourced. I think it looks more like talent constraint at the moment rather than funding constraint, but to some extent, funding can help. There are some questions about whether alignment work spills over to capability progress, like some of the things you would want to do for alignment, like better methods of interpret what is going on inside a little mine and figure out exactly why is it behaving the way it is. That would be useful for alignment. But it could also shed a light on what's limiting performance

and how to boost it. So it gets pretty complex. I think some other things like better cybersecurity in the leading labs to make it less likely that the weights of these models just get stolen, to make it less likely that the weights of these models just get stolen, that could maybe be helpful. And yeah, but I think like more work on alignment seems positive. Having some ability for leading labs at the critical time to go a little bit slow it seems positive.

I would be- Would that require coordination between multiple labs in order to be able to do that because you do have this sort of first pass the finish post? Yeah. It depends on like, I think the one already older idea and which might still be relevant is that maybe you would have one lab or, you know, whether it's one country running one lab or one just private industry lab or whatever it is, but like one would have some, um, lead over the, like just naturally some, like one lab might just be a year or two ahead because

they started earlier, were more lucky or had better talent or something. And so then that would create an opportunity for this leading lab to slow down, um, for a year or or to however long their lead was, right, without falling behind. And it might be desirable if rather than having like a super competitive race, you had a little, and then that kind of pause would be self-limiting, you see, like, because once they have paused for two years, that would be another lab kind of catching up and then maybe they could pause too,

but you would have to make an increasingly strong case for pause as more and more independent actors became capable. So it would be a pause that could exist and it would eventually expire. And exactly when it would expire, would depend on how strong the argument was for AI risk. And that would create a much lower risk of ending up

with a kind of perma-ban where this technology is never developed. Whereas if the path to getting the ability to pause for a year is to, say, set up a big international regulatory regime or creating a lot of stigma around AI research, like developing some mass movement, like smash the machine type of thing, then that's much more likely to spill over into something

that then become a permanent orthodoxy or a regulatory apparatus that just has like an incentive to perpetuate itself. So more worrying from that respect. How impressed have you been with the power of LLMs? Do you think that they are going to be the bootloader for what we need from a super intelligence perspective? Or is this, have you got limited hopes for how far they can sort of climb functionally? Well, we haven't yet seen the limits of what one can do when scaling these.

I think these transformer models, I mean, it's not just language, but they could have other modalities as well, but they do seem very general, and a lot of alternatives that people try turn out in the end to basically just result in similar performance as transformers. The transformers run like well on the current generation of hardware.

So they parallelize very well, et cetera. And so it might be that you need a little thing on top of that. Maybe it's the engine block, and then you need some sort of agent loop, or maybe some external memory augmentation, or some other little thing. But that you would still have this big kind of transformer or something similar to it. Like there might be some variation, but as the basic thing that extracts statistical

regularities and abstractions, that's pretty plausible, I think. It's interesting. I certainly wouldn't have guessed 10 years ago that something that you have a conversation with and that is able to accurately predict what it would say would actually be the forefront. It seemed to me tracking quite closely from whatever, like 2015, 2016, the development of AI that I think Yor book and then subsequent conversations around AI risk kind of blew up that conversation and then it seemed to me that maybe the 2018 2019 2020 AI hadn't really delivered the

threat that people perhaps slightly earlier in the 2010s were worried about and then perhaps slightly earlier in the 2010s were worried about, and then a chat GPT comes along and this conversation just gets thrust straight back into the forefront of everything. So it seemed like it had a thrust and then a little lull and then it's really, really sharply come back up again. Yeah, I think it's, I mean, yeah, why shouldn't like over index too much on like anyone

latest little development like, and, but I think also people's, um, expectations change. So now like, wow, it's been like four weeks without an image. It looks like AI winter, like it was all just hype. And if you zoom out, I still think we are like on an rapid upramp and have been since the start of the deep learning revolution in 2012, 2014. Nick Bostrom, ladies and gentlemen. Nick, I really appreciate you.

I've been a huge fan of your work for a long time. Your book is in the 100 books that everybody has to read list that I've been pumping for a very long time. Where should people go? They want to keep up to date with your work and your books and everything else? Well, I'm not, I'm not active on social media. So I think my website nickbostrom.com is where I put my papers and everything. So that might be the best place.

Nick, I appreciate you. Thank you for the day. Thank you. Oh, offense Yeah, oh, yeah Offense