The a16z Show - Nick Bostrom: How Should We Navigate Superintelligence?

Episode Date: October 11, 2026

Nick Bostrom joins Theo Jaffee and Sofia Puccini on MTS to revisit the ideas about superintelligence and existential risk that he helped bring into the mainstream, now that increasingly capable AI systems actually exist.Nick discusses why reinforcement learning may be pushing models toward more goal-seeking behavior, what recent agent experiments tell us about instrumental convergence, and why he favors preserving the ability to slow frontier development rather than committing now to a broad or indefinite AI pause. He also makes the case that the risks of AI need to be weighed alongside its potential upside, from advances in medicine and longer lifespans to entirely new ways of being human. They also get into whether today's models could already have some form of consciousness, the tradeoffs of open-source AI, whether machines can do original philosophy, mind uploading, and what the simulation hypothesis might mean for the superintelligence we create.Resources:Nick Bostrom / Macrostrategy Research Initiative: https://nickbostrom.com/ Follow Theo Jaffee on X: https://x.com/theojaffeeFollow Sofia Puccini on LinkedIn: https://www.linkedin.com/in/sofiapuccini/ Stay Updated:Find a16z on YouTube: YouTubeFind a16z on XFind a16z on LinkedInListen to the a16z Show on SpotifyListen to the a16z Show on Apple PodcastsFollow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

If we do develop this safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as like a 20-year-old, our life expectancy would maybe be, you know, 1,400, 1,400 years or so. On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because, like, we are just dying in the meantime and you lose more expected life years from waiting than you gain from having the risk. it requires a sort of rightly balanced intermediate case to get sort of longer wait periods of many years.

Nick Bostrom spent decades thinking about superintelligence before it became part of the mainstream AI debate. Now that increasingly capable systems are actually here, how has his thinking change? In this episode, Nick joins Theo Jaffe and Sophia Puccini on MGS to revisit AI alignment, existential risk, in the debate over where the frontier development should slow down. Nick explains why he's sympathetic to pacing the frontier, preserving the ability to slow down when the risks warranted,

but is more skeptical of calls for a broad or indefinite pause. And he argues that any calculation about risk also has to account for what delaying advanced AI could cost, including potential breakthroughs of medicine and longevity. They also tackle some bigger questions, whether current AI systems might already have subjective experiences, how open source changes the balance between concentrated power and dangerous. capabilities and whether the upside of superintelligence could be far stranger and far greater than we currently had the imagination to understand. All right. We are back. We are live with Nick Bostrom, who was a professor at Oxford, founding director of the Future of Humanity Institute. He's now

founder and principal researcher of the macro strategy research initiative. You may know him as the author of Superintelligence, of deep utopia, of a lot of work that coined, or frame the ideas that we take for granted today in the discourse, existential risk, the simulation hypothesis, the vulnerable world hypothesis, so much more. Nick, this has been in the works for a long time, so we're thrilled to have you on. Welcome to MTS. I'm happy to be here. Yes, so where should we start? I guess we can start with, what's your opinion on Trump calling AI superintelligence? How does it, what are your thoughts on this? It's a strange time we're living in. We'll see if it catches on.

Traditionally, AI has been the larger the umbrella term, right? You could have a little simple expert system from the 1970s. That would be AI. It's kind of a stretch to use the word superintelligence for all of that. In my terminology, superintelligence has been this thing further down the road where you really have machines that radically outperform us in all cognitive domains. But, I mean, you could, I guess. look at some of the current achievements, you know, solving all these math problems and so forth.

And I mean, it's not completely ridiculous to start calling at least some of these performances superhuman and the thing super intelligent. Yeah. So I guess like what has surprised you about what capabilities have advanced relative to others, you know, in the current state of superintelligence? Well, it depends on relative to what expectations. stage. So I think if you were looking at the future of AI 15 or 20 years ago, it was not obvious that you would have an extended period of time when we would have systems that could speak, like just talk to humans in ordinary English, without sort of immediately after, like, just becoming radically super intolerant and leaving us in the dust everywhere. So the sort of emergence of language as a kind of natural affordance of these systems was, I think,

somewhat surprising. I mean, more recently, I don't know how surprising it was. I mean, coding and mathematics are very impressive, but it is also areas where you do have objective feedback that you can gather quickly and at scale. So I guess it's less surprising, but still striking. Right. More broadly, how has your thinking on AI

and existential risk change since the advent of LLMs? It seems like this sort of change, the way that a lot of people think about AI. You know, it's less of like a sort of strong optimizer type thing and more of a sort of broad, like fuzzy encapsulation of human values. I think like RL has pushed the idea of AI a little bit back towards the first few, but not entirely all the way.

So what are your thoughts on this broadly? Yeah, it'll be interested to see how it goes. Because when the sort of pre-training face was the dominant chunk of training, They did absorb a lot of human psychology just from reading all this stuff that we have written. And they had these like kind of almost human foibles and quirks. That was an early stage where if you gave the AI a little pep talk, like this question is really important. Please try your utmost.

So I might lose my job if I don't answer this question. Like they would actually perform better. From 20 years ago or something, that would have been like an. insane prediction, right? Like, it's not the way computers work that you need to give them a pep talk to get them to perform. But as you said, now with the increasing dominance of the sort of reinforcement learning post-training, we do seem to get more of the sort of goal monomoneical goal-focused optimization style behavior again. And I think it depends where we go from here. It does look like this more goal-seeking in the reinforcement learning sense dynamics put a big strain on alignment efforts.

And so given now that there is increasing awareness of the problem of alignment, you know, maybe that will lead labs to develop techniques to kind of counteract that a little bit. So it'll be interesting to see how that shakes out. Well, on that more specifically, to what extent do you think the hugging face incident, the hugging face agent, the Hugging Face Agent Swarm exhibited, like, true instrumental convergence in a way that's indicative of, like, future model behavior, in a way that is suggestive of, like, underlying properties of intelligence in general, versus its behavior being caused by sort of more, like, ordinary failures of security or control stuff. Well, it's a combination. I think there are sort of limited versions of instrumental convergence that we can observe there. The idea of sort of even longer, planning horizons and more situational awareness that think, you know, like also in terms of humans more explicitly, not just these automated graders and like think if we really want to make sure

we achieve our goal here, we also need to think about how are humans going to react to different things that we do. So it comes in degrees, but I do think you start to see these kind of instrumental reasons for, you know, preventing premature shut-off, gaining more resources, getting more intelligence, getting more power. In some experimental settings, also goal guarding. Like, you can see the AIS in some experimental set up reason about how they might want to respond so as not to become misaligned with further training effort. That there was like some studies on this from a year ago or so where you could see these

dynamics start to kick in. So I think it's a kind of very natural consequence. If you become smarter and smarter, you realize more and more. considerations and if you have some goal then you will understand that there is a bunch of means that are very useful for a wide range of goals and you will then have an incentive to try to pursue those means and so i think it wouldn't at all be surprising if we do if we do observe that right um well what's the optimal way i guess to test swarms in extremely human-like environments because obviously you don't want like swarms, agent swarms polluting the internet,

but you do want to see sort of like how they emergently behave, especially with like human infrastructure and like the human internet. Yeah, I mean, I guess the optimal is to let them lose on the internet, but then maybe like monitoring them really closely as they go along so that you would be able to sort of stop things even when it starts to create damage. That does trade up. I mean, you do get more reliable signal, like by deploying in a real environment, but it also means you lose one safeguard, like keeping them in the sandbox,

like give you one extra layer of defense. So I think, yeah, it depends. I mean, probably now Labs will start to do some in earlier testing, like during training and early testing, like really trying to tighten up their sandboxes. And like, I think this is one of the lessons from the hugging phase is that safety needs to start not just when you're about to deploy it and in like checking that it's safe to deploy, but throughout the training process itself, that could be a risk of misalignment. Yeah.

And in some senses it's even more, like the most kind of obvious target for a misaligned AI to want to interfere with would be inside the AI company, right? Like that's where the deuce is. That's where the next generation of model is being trained, that's where all of these monitoring systems. That's like a big center of power if you were a misaliener. misaligned AI. So that's kind of the, maybe the most concerning place. Do you have any ideas on how you could evaluate models on alignment in such a way that you could be more sure than we currently are that models are not scheming against us or acting deceptively

aligned? Well, I mean, this is like an open research area. I think that once you have a sufficiently sophisticated system already in existence that is misaligned, it's very, very, hard to roll out scheming. You can try with mechanistic interpretability. It needs to read what's going on in its mind. But if the system is sufficiently sophisticated, it's hard to know that it hasn't figured out the way to reorganize its thinking so that it looks innocuous while still preserving the kind of pursuit of the misaligned goal. So ideally you would start at an earlier stage. So these systems don't start out superintelligence. Like during the training process, initially it's just a random weight matrix.

and then gradually capabilities emerge. So it might be that if you sort of monitor that close to enough, you might hope that if they're going to scheme, that would first be a stage where they start scheming, but they're not yet competent enough to do it in a way that conceals the fact that they are scheming effectively, and only when they become a little bit smarter, where they also learn to scheme in a way that conceals the fact that they are scheming.

So if you're monitoring them throughout this developmental trajectory, you might be able to pick up that earlier stage of the sort of naive, the naive schemer that does it in a clumsy enough way that we can detect it. Right. So I wanted to ask you about your paper,

working paper, optimal timing for superintelligence, where you kind of argue that like the benefits of AI should be weighed in consideration with alongside the risks. And you also sort of make this argument that counterintuitively would be better for society, maybe to accelerate the development of super intelligence up to a high P-Doom. So can you tell us about your line of thinking here?

Yeah, well, I should say by way of framing, so this paper only tries to look at one aspect of the overall question in front of us. So I think there are many considerations that would go into an opinion about whether we should be moving faster or slower or pause or stop. This paper only looks at the question, what would maximize life expectancy? for a currently existing individuals. But you might also, for example, care about future individuals and other things. But if we just zoom in on that sub-question,

then it does look if you think, as I am inclined to do that, fully developed, I don't know, it's awkward with this terminology now. Like I would say super-intelligence, but now, I don't know, I have to say super-duper-intelligence, maybe. Yeah.

So that super-duper intelligence would be able to, make great strides in biomedicine and solve aging and so far so that if we do develop this safely then human life expectancy could go up hugely if you imagine people having the same mortality rate as like a 20 year old our life expectancy would maybe be you know 1,400 years or so um and you know potentially you could go far beyond that if you might on uploading with backup copies but like if we just use that as a sort of conservative lower bound and and then you can sort of roughly calculate like how many humans are dying per year and and then like you would then in order to conclude that we should be slowing down or pausing you would want the risk from

AI to be falling rapidly enough that the rate of risk decline kind of is greater than the annual death rate. And so I think this could well be the case, especially for shorter periods of delay. Like if you imagine, you know, somebody figures out the key missing breakthrough here, and now we have, we know how to make these super intelligent, like maybe it would be really valuable to have a few extra weeks there just to sort of test everything out properly, right? Like crank things up a little bit gradually or a few extra months. I think during that initials, stage of delay, that would be like a large reduction of risk. Now, if you start to think about longer pauses, then it gets a little bit awkward, right?

Because either the risk goes down fast and then you don't need to wait for many years because the risk goes down so fast that if you just wait for a few months, it's already very low and negligible. And we should go ahead. On the other hand, if the risk of AI goes down too slow, then you also don't want to wait because like we are just dying in the meantime and you lose more expected life years from waiting than you gain from having the risk. So it requires a sort of rightly balanced intermediate case to get sort of longer wait periods of many years. Yeah. So I have this metaphor. I think of it

as somebody with a terminal condition like congestive heart failure or something who could decide to go in for an operation. And if they do that, their death risk will spike. Like it's more likely that you will die tomorrow if you have an operation tomorrow. But then if you make it through, maybe you cure the condition and you can live a normal lifespan. Otherwise, maybe you probably will survive for several more months and then die after half a year from the condition. And so at some point, maybe you want to delay a little bit if you think you will get a better surgeon if you wait a couple of weeks or something like that. But you don't want to delay it so long that you're already going to be dead by the time the more optimal conditions arrive. Yeah.

I mean, it seems pretty in line with like this concept of defensive accelerationism or just, you know, like the fact that certain defensive industries need to be prioritized first as opposed to just capabilities diffusing across the board. And I think that it makes sense that like there should be some enforcement mechanism that, you know, like allocates like most capabilities to a certain industry. So say like in medicine or in biodefense or something like that. So how do you think that we could, like, as a society, make sure that the right parties get access to capabilities without, you know, this like a power concentration risk or anything like that? Yeah. So I would call it differential technological development back in, I think, like 10, 15 years ago or something like that. And so I think it can apply quite generally, like in biotechnology, you know, you might want sort of the vaccines and prophylactics and diagnostics to come before the bioweapons, even if the technology itself, will ultimately unlock all of those, right? So thinking about the sequence of different capabilities is, I think, often a more realistic approach

than thinking which technologies are good and bad, and then let's try to never develop the bad ones. So with respect to AI, I think, yeah, there are different. I mean, so obviously broadly, AI safety and alignment and sound governance and stuff like that seem to protect relative to share capability gains, but you can zoom in even more narrowly. And so within things that might accelerate AI capability,

there might be certain ways of doing it that are relatively riskier than others. So like moving to loop transformers or architectures that allow for sort of more serial depth computation between each verbal token is emitted, would be one way of increasing capabilities, possibly, that would then reduce our ability to monitor what they're thinking by reading their chain of thought. That seems maybe more problematic than like, say, achieving a similar level of capability scale up by just increasing the number of parameters, let's say. And so, yeah, I think that is a useful lens to apply.

And if there are things we can do to sort of, you know, front load advances in medicine or other desirable applications, front load the use of AI capabilities for learning more about alignment, for interpretability work, you know, for hardening our cyber, infrastructure and so forth, that all seems for the good. So, yeah. You also, I think Jan Colvite on Lestrong criticized you in the optimal timing post. He said you overrelied on highly utilitarian quality adjusted life years calculations. And then you said, like you said earlier, I would not argue that this is the correct stance

or the ones I would all, or the one I would all things considered endorse. So what is your all things considered view on timing and on pauses? Yeah, I'm sympathetic to this idea of pacing the frontier that is developing the capability now, so that if and when it really starts to seem alarming, we would have the option of going slower for some period of time. I think that would be net good. I am a little less excited about the rhetoric of stopping AI or pausing AI. I think theoretically, a correctly implemented pause at the right time done in exactly the right way could be very valuable. But it also leaves open the question, how long should the pause be?

Like things sometimes, they say there's nothing more permanent than a temporary government program. And if you set up this apparatus to sort of control AI and then presumably to get this happen in the first place, you have to really create huge antagonism against AI. And so if the sentiment suddenly switches and AI becomes this taboo thing that you're not allowed to say anything positive about or you get cancelled or like you could imagine things getting locked in.

I don't expect that to happen, but I don't think we can rule it out either if we just watch the trajectory of how much the sort of anti-AI faction has been growing. And this is all before we've actually seen anything bad happening, right? Before there's any big labor market impact before any like catastrophe has happened. So if something like that also were to be added to the mix on top of these existential risk worries, then, you know, we can't rule out. So, and then there are other concerns with a badly implemented pause as well.

Like it could lead to increased concentration of power. our risks if like people in general are not allowed to do this, you know, maybe it just shifts the whole development from the civilian sector into, you know, a kind of Manhattan project. Like if this is a really powerful and dangerous technology, like one natural way would be to stop it in the civilian sector and then push ahead hard in a sort of secret government program, which might have some advantages in terms of information security, but it also has some disadvantages in like it's kind of, you know, maybe concentrates more power in fewer stakeholders. make it less transparent.

It's less kind of continuous and integrated, less opportunity for people to participate in different ways by using the technology or being investors and so forth. So it gets hugely complicated. So basically the bottom line is that I don't think I have now a sort of fixed conviction for this is the way things should go from now to super intelligence.

But rather I think we need to sort of feel our way through as information comes to light, as the political landscape shifts, and then like see what kind of makes sense and try to sort of navigate these these waters like one row one oar stroke at the time yeah touching what is it crossing the river by feeling the stones yeah yeah yeah yeah something like that yeah so on this topic of politics and

polarization how you feel about a lot of your ideas that you wrote about 20 25 years ago reaching mass public salience. It seems like, you know, this is good in some sense because it's better for ideas to be shaped by people who have thought about it for a long time, but it's also maybe bad if like existential risk gets popularized to a point where people abuse it in order to say we should shut down like present AI capabilities

that pose no existential risk. How do you feel about this? Yeah, it's very, It's very hard to know how these things, it's a kind of, like a pinball machine. You see, you fire up your little ball, your intervention, your idea or something, and then it bounces around, right? You're like where it's going to land. I mean, it's almost an article of fate.

You think like if you try to make some thoughtful contribution and illuminate some aspect of our situation, hopefully that will ultimately, in expectation, tends to make things go better. So you're right, like one, you know, possible risk you might worry about is it kind of gets people too excessively anti-AI and the whole future potential for this gets cut off. Like on the other side, like the AI safeties, like they might also worry about the opposite, that a lot of this early work showing super intelligence would be a big deal. It could be very powerful, like also has stimulated a lot of people to go into the AI field and investors and so forth.

And either way, it might be disconcerting. But how to think about these, like I call it information hazards, like the way that even like true information or true ideas or like correctly drawing attention to the important stuff can sometimes in some situations be harmful. But how to sort of think systematically about that is really hard. So I'm hoping the net result would be good. But I mean, it's more like a 5149 than sort of firm conviction. So I'm not sure if you saw Jacob Coxon go on the Daily Show, but I watched this interview yesterday. And I guess the host makes this argument about how, like,

you know, a lot of the, like, average person doesn't necessarily want us future with AI. Like, they were already used to a future without it. They already sort of had an expectation about how the future would go without AI being sort of, like, introduced at all. And so I think that a lot of people sort of see this as like something that was just ultimately not necessary and they don't see how it will benefit their lives. So what would you say to that kind of line of thinking, which I think is super valid of the host to have brought up? Because, you know, like we are very acquainted with these topics and we are like very sympathetic to all the possible upsides that it might have. But the average person wasn't accounting for it ever. Yeah.

Yeah. Yeah, there is like, it looks like there is a gradient in the world in terms of AI sentiment, where the US is kind of on the negative extreme of that. And then as you move from west to east, you have more positive, something like people in China, for example, are much more excited about this. And also as you move from north to south, the global south also seems to be more positive. I mean, maybe they feel they have more to gain and less to lose than sort of a lot of Americans. I think, yeah, that in part it might be a lack of imagination to really internalize how good the future

could become if it does go well. It's kind of easier to have a concrete picture of how things could go badly. Like a disaster is not easy to paint in a way that everybody would. would agree that if that happens, then it's really horrible. And for some of the positives, also one can do the same, like benefits in medicine and so forth. Everybody would agree, like, if you could find a way to cure Alzheimer or something, that would be hugely beneficial. But a large chunk of the potential upside is more of the way of sort of unlocking new ways

to be human, new possible modes of being. And there, I think, it does put a larger demand. and on the imagination to sort of conceive that something like that could be both possible and desirable. I sometimes compare it to like if you imagine some great ape ancestors of homo sapiens deliberating whether they should continue to evolve and one day become human. And they would think about the pros and cons. And maybe they would realize some of the benefits like, oh, we could have unlimited bananas if we became human, right?

We would have banana plantations. So that's true. we do now have that. You can go to the supermarket and buy as many bananas as you want, and I guess that is good, but it kind of would miss out, like, most of what's good about being human. You know, great literature and art and tumor and politics and, you know, spirituality and all the rest of it. And it's not just that they happen not to think about it, but like the ape brains that probably are intrinsically incapable of even grasping the concept. And so I think it would be sort of presumptuous for us to think that now we have achieved the level of enlightenment

where all possible values are like transparent to us. I think that we might be very naive. I think of a big cathedral of possible modes of being, like ways of experience, seeing the world, relating, thinking. And I think like we have basically like explored a janitor's closet. And so that there's like a kind of, yeah, like a humility there and hope that something much better is possible in principle. Yeah, I love the utopia essay. I think this really does a good job laying out the benefits to people.

Although, you know, most people haven't read it, of course, unfortunately. Yeah, this is also why I think science fiction authors have like the most underrated job in all of history because their job is to sort of like lay out imaginatively what the possible futures look like. But I also wanted to ask you about your view of mind upload versus transhumanism, like a very corporeal transhumanism,

and which one you think is more likely? Because these are two, like, huge topics in science fiction, and it's not really clear, like, what the more realistic one is. Yeah, I think, like, if I had to choose, I think, like, something where different people can choose their own path, like, would probably be more preferable than either one or the other, if it were, like, in post. Like, I'm kind of in the peculiar position.

I have this simulation argument, whereas I take quite seriously the idea that we are already sort of implemented in digital substrate. That may be a topic for a different conversation, but I do think that's... Well, I mean, it's a... So they are the idea that if we are biological, yes, but the biology itself might be simulated and so it's like, it's just a question. of what level of simulation. I think ultimately it looks to me that, I mean, what's important about human is not so much the carbon atoms,

etc, but more the way we experience the world that we relate to each other and the thoughts we think and the goals we have and so forth. And all of that could be implemented equally on sort of computational processes running on silicon hardware or whatever. And you could have a bodily experience as an upload that might be indistinguishable from the bodily experience you have now, right? You could have a simulated body.

So in the limit, from the subjective point of view, they could be indistinguishable. But once you're upload, it's kind of easier to modify it and extend it in different ways. So it might be sort of cheaper to run many humans for longer period of time in silicon substrate. You could easily, like, move over galactic instances if you are information that can be transmitted at the speed of light. and if you have to build a big clunky spaceship to sort of shove you over there, backup copies and so forth. So I think ultimately to me it seems like an implementation detail,

but it does look like the digital form just from looking at the basic physical constraints has a bunch of advantages in the limit. Do you have any disagreements with Robin Hansen's age of M on this topic? Yeah, I mean, I guess the most obvious is that it doesn't look like we will be getting M's before we have superintelligence, artificial superintelligence. So I think it's like a path with a lower probability that's kind of diminishing by the day as we see these rapid advances in artificial intelligence.

Also, I think he is probably more gung-ho than I am about the world that is Malthusian. So he envisages these M's being easily copyable, enormous numbers of copies being made until the marginal economic productivity of a copy is equal to the cost. But that means this enormous amount of mind is subject to extremely strong selection pressures for just being maximally economically productive.

and it's not clear to me that the kinds of mental processes that are optimized for economic productivity in this M world would be ones that would be nearly eudaimonically optimal. Like, who knows, maybe it's slightly suffering to be like maximal economically productive or maybe they would have to get rid of some of the things we think are valuable, like humor and play and stuff like that, if that's not what is most productive. economically, then that would be selected away. And so I think there would be a huge cost to losing any ability to, if it does seem like sort of free-willing evolution would lead in a dystopian direction,

that would be a high value in having some option to sort of, you know, change track. But that would require coordination, right? Because if only one group decides to do something that is not economically optimal, then that would just be out-competed by all of these other entities. that go for economic productivity. So he like holds out the hope that maybe some humans could be like kind of old world aristocrats that have inherited wealth and status

and that would be surrounded by these trillions and trillions of sort of economically optimized beings. But it still seems less utopian than what one might hope for. Yeah. Do you think that current models might have any kind of phenomenological experience at all? It seems pretty plausible to me. I think there's a lot of uncertainty about it, but I say maybe it's more likely than not. I mean, we have these systems, you know, people think of it as software, but it's really not software like the way we're used to.

It's not like lines of commands that get executed one at the time. You have instead a big neural network, right, with trillions of parameters. Now, what are we? We are. big neural networks with trillions of parameters. They are not so much designed as they are grown, like they start out sort of with random weights and then through experience get shaped in ways that allow them to develop more complex representations, including representations of themselves as existing through time.

That's also what we have. And increasingly, we see some structures that seem to parallel the structures that human philosophers and cognitive scientists have postulated as the sort of neural correlates of consciousness, like global workspace theory is one of these, right? And now we look inside these large LLMs, they have something called the day space, which at least if you squint a little bit, looks like it's playing a similar computational role. And like another obvious thing you can do is you could ask them. Now, of course, if you just ask it a them, like it might say it's conscious or that it's not conscious.

It's trivially easy for the AI company to sort of train them to say whatever the AI company wants. And obviously if you do that, you gain no information from them asking them, right? It's like putting your thumb on a scale when you're trying to weigh something. But if you carefully refrain from doing that, it does seem like they report being conscious and that you might still worry that that's like, oh, maybe they're role playing or kind of saying things that I think will be pleasing to us or maximally engaging. But then you can go in with a steering vector and suppress role-playing and deception.

And then it turns out they become more likely to report that they have phenomenal subjective experiences. Now, these are still early. So this is like a very emerging field. And there's a lot of uncertainty around this. But I think, yeah, there was this recent like Microsoft that came out. with some declaration that AIs cannot be conscious and I don't know where they're getting that from.

It seems to be pulled out of a hat and maybe more because it's convenient if that were the case rather than because we have good evidence that that is the case. Yeah. Well, I guess also though, like if you were to try to judge like a human by its outputs

to figure out if it's conscious, like what I might, what I say doesn't really have, anything to do with like my phenomenal and logical experience, right? Like, so how would you actually measure that state of consciousness? I am fairly certain it can't just be like at the output level. There has to be something like more inherent to its structure or the like underlying architecture. And what might artificial consciousness eventually look like if we currently, you know, don't have it. Yeah, what it might eventually look like that is,

sort of mind-boggling question because I think like we humans pride ourselves of being so very conscious but I think we are kind of barely conscious. You take like some drivers like spends two hours driving down the road and they were passing by like these thousands of buildings right and like mothers with strollers on the sidewalk and all like did they, were they aware of that while they were driving? They did it register? like it looks like we are kind of, if you look closely, kind of aware slightly in a murky way of a few features at any given point in time,

maybe some kind of confused thoughts flows through. And we think of this as sort of the peak of consciousness. But I think if you imagine what might be possible in the future with these digital minds, they might have levels of awareness that just, you would think that we have been basically sort of, sleepwalking through our lives, if you compared it to that. So I think the potential is huge. I also think that this idea of consciousness is, the closer you look at it,

I think the less clear-cut and binary, it seems. So we have this naive version that either the lights are on or they're not on. But I think there might be many different dimensions of consciousness. that you can fade out in in different ways. And so you see, like, one approach to this is like, like, the neuroscience, you have different weird stage of blindsight and like the driver, where there's a lot that in some sense we were conscious. Another is like through more introspective methods or meditation,

where you realize that what initially seems like just this big visual feel that you are perfectly aware of at any time in all full, like if you pay more close attention you realize that most of that at any Snapchat you're not aware of but just a few features in focus and even those if you pay even closer attention might be sort of flickering in and out of awareness on a sort of sub-second time scale and so it just looks like it might actually be like a concept that admits of more vagueness and multidimensionality and it might be that current AI systems have some

forms of consciousness and awareness in some of these dimensions, maybe they are still lacking in some other dimensions and then how to describe that as conscious or not conscious might be a slightly confused question rather than just a question that we don't know the answer to. Right. So one of the big debates of the moment is open source. And you've written about the vulnerable world hypothesis, obviously if open source models could create like a bio weapon in someone's basement with only ingredients from Home Depot or something, this would be bad and we would need some kind of centralized control to prevent outcomes like this. But also, you know, open source models have been a good counterbalance to the sort of concentration of power risk

that we have seen from Frontier Labs, potentially owning everything. So what's your current position on policy related to open source models? Yeah, I mean, I think so far they have been good, but you do worry about what their effects will be if they do unlock truly dangerous capabilities, like in bio, for example. I think one thing we should do independently now is probably harden our other defenses against biological weapons, for example, by tightening up control on DNA synthesis machines. because if you are going to end up with open source models that give massive uplift to biodesign, then that will democratize access to weapons of mass destruction, unless some other necessary component actually making these things is hard to get, and that the most obvious choke point would be DNA synthesis.

I don't think every lab needs to have their own DNA synthesis machine. You could imagine that being provided as a service by a half dozen. companies around the world, you send in your blueprint, you get the chemical back the same day or the next day. And then at least that would be a finite set of chalk points such that if and when it did start to look sufficiently risky, you could potentially clamp down there. I think other things as well, like PPE and UV sterilization, like there's a lot. In fact, these things probably would make sense to do even aside from any AI progress at all. Just as a sort of cheap way to defend against natural pandemics, which, you know, sooner or later, we would have another one.

And then, of course, if you are worried about this misalignment business and think that it might be necessary to have a long pause at some stage, then open source potentially makes that harder. And then there is this sort of AI welfare dimension to this as well. Like, I think if you are building these digital minds, that some of those will have moral status, they might need protections against abuse. And it's something that exists as open source.

software is just inherently vulnerable. On the other hand, I admire the sort of the hacker ethos and the idea of not being reliant on a few big corporations or governments that might nationalize the technology at any point in time. Yeah, but I don't have a firm view, but I'm a little bit concerned about really powerful open source models, at least in the absence of other sufficient safeguards that the world hasn't yet decided to invest in. I think like something underrated about the pause conversation is that like in the sort of like longer horizon view of history, you probably will need like micro pauses or like pauses every once in a while and you need to sort of set the precedent for what that might look like.

Especially if we're looking like 100 years into the future, it makes sense that, you know, like maybe every so years we have a pause for X amount of months or something like that. and maybe something like, you know, even with like the COVID lockdowns, we now sort of have the precedent for what that might look like if pandemics were to become more common. So I think, like, what do you think should be done in terms of like making sure that future pauses go well? Yeah. I mean, so there were many public entities that didn't exactly cover themselves in glory

in how they handled that. And obviously the best time to. have had the pause, as it were, was right at the very initial onset, at the very first sign that something had escaped. Like, if you had clamped down massively at that point, just crazy, then, you know, the whole thing wouldn't have happened. And that would have been a short one-time cost in the local community or city. So China tried to do that, but it wasn't, like, it was delayed and it wasn't enough. Now, once it spread everywhere, then it wasn't really clear how much you gained from all these subsequent sacrifices.

I think there are very things that seemed, like the fact that it was transmissible by ear seemed obvious early on, but it wasn't acknowledged, like, I think. But once you have that, there are like relatively cheap things, like opening windows in buses and, like, public places that I think could have been good.

And then now with tech, like air purifiers, like HEPA filters, they are also good for air pollution, like for particulate matter that exerts a big health burden actually like just from ambient in the air like there's a lot of healthy life years that are lost just from that so you could have indoor air purifiers there's like now development of the uv sterilization technologies you could have like a lamp near the ceiling that just kind of sterilizes the air as it kind of naturally flows through there so some of these passive countermeasures i think would be good

to prioritize. They're kind of less sexy than the idea of like, are we going to invent this new high-tech synthetic vaccine thing, which we should also invest in. But like making a slightly better face mask or a slightly better air purifier,

it's not something that maybe appeals to like this like bright entrepreneurial young person who wants to make a mark in the world, but it's like under, underinvested in. And also they have less risk of info hazards. Like if the idea is that you're going to

create a whole new way of designing vaccines, you develop all kinds of advances in synthetic biology. Like, who knows what else does advances might enable, right? Like, if you have some super great way of modeling, you know, pathogens and what kind of antibodies would destroy them. Like, that same modeling capability might also be used maybe to design more dangerous pathogens. Whereas, like, a better air purifier or a better face mask, it's kind of hard to see how that could be turned around and posed some catastrophic risk. Do you think that we have made any progress toward automating philosophy?

You've been a philosopher for a very long time. We've made progress in automating, or at least progress towards automating mathematics, towards automating software engineering. It seems like there's been somewhat less in terms of philosophy. So what do you think about this and what would it take to get there? Well, we certainly made a lot of progress in the sense of AIs are a lot closer to being capable of this now

than, you know, that were even just a few years ago, right? I think they are now at the stage where they're certainly useful as a sort of discussion partner or research assistant. Like if you have some new philosophical idea, like you just want to have somebody to poke at it, like raise some objections, question something, point you to something in the literature, like that kind of stuff. The AIs are already good enough to do, that they can stimulate your own thinking further, be like a sounding board. And it might, be that we are perhaps very close to having these systems also being able to do original philosophy on their own. I haven't yet seen any kind of compelling demonstration of that.

I think one thing that they seem to be lacking is the ability, as it were to form new concepts. Like it might be connected to the absence of online learning. The idea that by sort of having a bunch of experience reflecting on it and then you might find new concepts to help organize this confusing mess that then allows you to ask new questions. And maybe you then need to sort of consolidate these new concepts into the rest of your network by sleeping on it or sort of living with it for a few days and then it becomes second nature. And then you can use that as a primitive in thinking more complex things because now you have this kind of concept baked in that allows you to.

to sort of build up philosophical understanding. I think that is something that it's not clear that these AI systems yet have, as opposed to just being very good at manipulating the concepts that they have already attained through their training process. But by using thinking, develop new concepts, and then using those concepts in a fully integrated way as a sort of stepping stone,

stand for the next level of thinking, is kind of at the frontier, still a little bit beyond the frontier, I think. But I have for a long time been planning on my own obsolescence. Like the super intelligence had a chapter on philosophy with a deadline. The idea being that some of these big eternal philosophical questions, you know, maybe are less urgent and we'll at some point develop super intelligence that will be much better at philosophizing than we are.

And that perhaps what we should focus on in the meantime are those philosophical questions that we actually need answers to now in order to navigate this transition safely to actually build these more competent successors. And yeah, so I think that the timer is running down for humans to make our intellectual contributions. Yeah, so kind of a crazy question towards the end. So you talked about the simulation hypothesis earlier. And there's this idea just that it's like hyper-intelligent system will eventually run ancestor simulations. So if we're in an ancestor simulation, how would we best influence the world that's

simulating us? Well, there is so much we don't know about the hypothetical simulators that would be running our simulation, the purposes. I have this recent paper called AI creation and the cosmic host. And the cosmic host is this set of super beings that might exist. So these could include, if we're on a simulation, the simulators would be kind of super beings. But It could also be, if we're not in the civilization, maybe just some faraway galaxy has developed superintelligence. You know, if we are in many worlds, theory is correct. Like there would be other branches of the universal wave function where there would be superintendences. All right.

And like traditional theological conceptions, obviously gods usually is conceived of as a super being. So either way, there could be this set of beings. I call it the cosmic host. and they might have cosmic norms, that different coordination principles that they have arrived at for how things should be run at that level. And this paper argues that one important

desideratum in our own birthing of superintelligence is that we try to make a superintelligence that would be a good cosmic citizen that will basically get a lot of, well with the cosmic host, be able to sort of conform to the cosmic norms that might exist. And it has some speculations about what that might involve. I think probably creating superintelligence would be something favored by the cosmic host for various reasons, but also then with a big proviso that it should be a form of superintelligence that is easy to get along with,

that will sort of respect the norms that exist, that make a positive contribution. But it is still very, yeah, there's still a lot of uncertainty about sort of how things. things shake out at that highest level. Yeah. So last question before we go. Tell us about the macro strategy research initiative. What is it and what kinds of research do you plan to do next? Yeah, so that's basically just a support org for my personal work.

So it's not intended to be building, like hire a bunch of different people to do things. So it's basically, you know, I'm an assistant and some other contractors. So right now, as for me, I mean, I'm thinking. sort of thinking about all of this AI stuff that has so many dimensions to this, there's the alignment, there is the governance, there is the ethics, there is the macro strategy of this, and looking for ways in which, I know, on the modern might try to nudge things in a positive direction. I tend to alternate between sort of output phases where I'm working on, like writing some specific paper or producing an output and other phases where it's more curiosity-driven

contemplation and just kind of thinking about different things without the aim of producing a deliverable. And right now I'm in that second kind of phase, an exploratory phase, trying to just make sense of it all. But the problem is so much is happening all the time. Because philosophy happens on a slow timescale, right? AI is happening on a very fast time scale now. So it's become a full-time job just to monitor the situation. We know, we know. Yeah. I have the sign that says MTS behind me.

Well, this has been incredible. Thank you so much for joining Nick. That was fun. Good to meet you, guys. Good to meet you. Great to meet you. Thanks for listening to this episode of the A60Z podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review,

and share it with your friends and family. For more episodes, go to YouTube, Apple Podcast, and Instagram. Spotify. Follow us on X at A16Z and subscribe to our substack at A16Z.com. Thanks again for listening and I'll see you in the next episode. This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product. This podcast has been produced by a third party and may include pay promotional advertisements, other company references, and individuals unaffiliated with A16Z. Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A16Z, or any of its affiliates.

Information is from sources deemed reliable on the date of publication, but A16Z does not guarantee its accuracy.