Into the Impossible With Brian Keating - Robert Wright: AI Alignment Is a Moral Problem, Not a Technical One

Episode Date: September 30, 2026

The real AI alignment problem is not technical. It is moral. You cannot align a superintelligence to human values if humans cannot agree on what those values are, and you cannot get that agreement without a level of global cooperation we have never managed before. Subscribe if you want science with evidence, not speculation. The Tower of Babel is the AI parable nobody is reading. Technology that outpaces wisdom fragments the people building it. The Fermi paradox offers the grimmer version: civilizations that reach the capacity to blow themselves up typically do. Robert Wright has been watching this longer than almost anyone, he was reporting on neural networks in 1983, and his conclusion is not that the machines will fail the test. It is that we will. Wright’s book argues that superintelligence is not the danger and AI alignment is not the solution unless it comes with a moral advance. Passing the God test requires something no lab can ship. What you’ll hear: -Why Wright thinks p(doom) fails as a scientific concept -What the binding of Isaac and the Tower of Babel have in common as moral tests -Why global cooperation is the only thing that can actually slow AI down -What Lee Smolin’s cosmological natural selection has to do with the Fermi paradox -What Wright thinks Eastern and Western enlightenment have in common and why it matters now -Whether there is a role for humanity inside the global brain or only for the machines “We face the kind of test that God would give.” — Robert Wright CHAPTERS 00:00 The man who interviewed Hinton in 1983 05:04 The God test and its two meanings 08:36 Religious faith in technological progress 14:10 Hinton then versus Hinton now 19:38 Why slowing AI down requires international cooperation 26:02 How LLMs reverse-engineer the meaning of words 32:08 Brian's Einstein test: can Claude have a happy thought? 40:06 AGI is impossible. Here is why Brian thinks that. 47:04 The Fermi paradox as a moral filter 51:10 What passing the God test actually looks like Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/yt Have a .edu email and live in the USA? You automatically win a meteorite: https://BrianKeating.com/edu Enter the debate: briankeating.com/ai Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1 Support Into the Impossible on Patreon, get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode: https://www.youtube.com/channel/UCmXH_moPhfkqCk6S3b9RWuw/join Featured Guest: The God Test (book): https://thegodtest.net Robert Wright on Twitter/X: https://x.com/RWrighter My books: Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U Galileo’s Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un Twitter/X: https://x.com/BrianKeating Substack: https://briankeating.substack.com Blog: https://briankeating.com/blog Audio-only: https://briankeating.com/podcast #RobertWright #AIalignment #superintelligence #FermiParadox #briankeating #intotheimpossible #AI Learn more about your ad choices. Visit megaphone.fm/adchoices

You feel sure everything's going to work out fine. I don't really understand how you can be confident of that unless you have something like religious faith in technological progress. Natural selection is blind. The mutations in natural selection are blind, but the selection criteria are not. One of the grimmer readings on the Fermi paradox is the reason you don't see many civilizations that have gotten past the point we are is that typically when they get to this point

where they have the capacity to blow themselves up, They blow themselves up. I think we face the kind of test that God would give. My guest today is Robert Wright, one of the OGs of AI. He was interviewing Jeffrey Hinton about neural networks. In 1983, two years before Sam Altman was even born. And that was long before anyone was terrified of them. Robert thinks that AI is putting humanity through a test

and that passing it takes something no lab can ship, a moral advanced. The Tower of Babel story is a truly cautionary tale. a tale of, you know, kind of technology gone awry, technology wanting to compete with God, to fight with God, to make a name for oneself. And you see a lot of that. Where do you come down on this, that the fundamental aspect of the God test is, at least as I read it, but now I'm talking to the authors, is are we gods but for the wisdom? Are we able to pass this test? And maybe you should define the test where we can prove maybe to ourselves in the absence of a God that we're worthy of God-like technology. The title has at least two meanings. One is, as you know,

some people think we're building a superintelligence that will ultimately, in some sense, maybe qualify for a term like God. It'll be all-powerful and maybe live in harmony with it. And according to some doomers, maybe get annihilated by it. But in any event, we'll be talking about a kind of from a, you know, relative to us and an all-powerful and, you know, omniscient in some sense being. And so, you know, one test is if that's going to happen, can you build a good God, a God that is compatible with human values and would safeguard human flourishing and so on? There's that. And although I'm not sure we're building a superintelligence, I certainly don't rule it out. And I explain why in the book. But there's another sense,

of the God test that in a way is related to the Tower of Babel, because the Tower of Babel is a story of hubris. Who are we to build a god? On the other hand, the Dumers say, even if we don't try, it's taking shape. We're going to have to try to stop it to keep it from taking shape. But in any event, the Tower of Babel is also a story of how humanity became divided, how they came to speak different languages and became in a certain sense, kind of inherent. tribalistic, at least at a linguistic level. And one argument I make in the book is that if we're going to do a good job of navigating the AI revolution and wind up, leaving aside the question of whether it culminates in superintelligence, if we're going to reconcile it with continued human

flourishing and welfare, I do think we're going to have to approach it as a true global community. I think there are a lot of policy issues that can only be handled at an international level, and there's more and more recognition of that. Only just in recent months, we've seen that. I think the end of the story of the Tower of Babel needs to be that we all kind of get back together and navigate this challenge collectively, that is to say, human beings. And so in that sense, the God test is to overcome our tribal differences, to get better to understanding the perspectives on people on the other side of walls and so on and like quit having wars. That would require something that I think would qualify as a moral advance. And I think it's

going to be necessary if we're going to survive this in good shape. So in a way, that is a test for being subjected to, and that is the kind of test that God would give. At the end of the book, I quote the Hebrew Bible, or as we used to say in Southern Baptist Church, the Old Testament, salvation is at hand. And, you know, as I say, there are a lot of stories in a lot of religious traditions where salvation is possible, but you are going, in some sense, have to pass some test. It may vary by religion. In the Christian tradition, salvation, the test has to do with accepting Jesus as a Savior who died for your sins. But very often, in one sense or another, it's a moral test. And for salvation to happen. And by the way, I would also say that salvation is often

salvation at the societal level, including in the Hebrew Bible. Again, Christian salvation is often construed more as an individual thing, but in many traditions, it is a question of kind of saving the social system. And I'm saying, I think we face, leaving aside the question of whether there is a divine purpose unfolding, I think we face the kind of test that God would give. There's so much to unpack there. The classic, in the Torah, the Old Testament, the classic paradigm for test, But the first test that's really, you know, actually called a test or thought of as a test comes to Abraham. You know, he originally has a different name, Abram. And then his name gets changed to, you know, Avraham, which means father of nation.

Then he goes from being an individual to a collective, as you just said. And the kind of crucible, literally, by which that's thrust upon him, is a series of tests. And in the Torah tradition, there's 10 tests and begins with his father throwing a him in an oven and for breaking the idols that he worshipped. And that's where I want to start. You know, there is so much with these new models and so much personification. You probably saw this Dwar Keshe Patel article that got tens of millions of views on X and elsewhere, where he talks about civilizations living and dying and raising and rising and falling. It really seemed so hyperbolic to me. And in my conception, I've talked to all the doomers. And there's way more

doomers and there are boomers, as you know, the only public, you know, boomer that I really know is Jan Lacoon. And I had him on and he's as extremely, as you know, when you quote him in the book, he's extremely, you know, technical in his, in his optimism. He's not, he's not, you know, kind of denying the sort of societal upheaval, but he's saying, you know, we're not there technically is the impression that I got. And we probably will never get there. And I want to talk about that. But starting with the test of, you know, kind of is this an idol? You know, are we worship these things. When you hear these laboratories, we had to put it in a sandbox, we had a stovepipe, we had to cut it off because it was getting too dangerous and it was escaping and it was creating

civilizations according to Dorcasch. What do you make of this? We live, whether or not we personally, individually are or not, we live in a secular age. And the secular age has never been more enhanced by technology than it is now. So do you make anything of these comparisons between both the idolatry of AI and the eschatology of AI? In other words, the Duma is intimately connected, I think, with the idolatry. So what do you think? Are we making these things into gods to worship or to propitiate? Interesting question. I mean, the Dwar Keshe thing was about a bunch of agents in this, you know, famous open AI hugging face incident. And he was comparing generations of agents, I guess, to civilizations. I'm not too bothered by that kind of anthropomorphism,

because, as I argue in the book, I mean, we are kind of trying to, and in some cases doing this without trying, but to inculcate in AI human tendencies. You know, there are a lot of things, a lot of ways we want them to be like people. We want them to be pretty autonomous when they seek goals. We want them to do this, want them to do that. And I have a separate argument about what seems to be going on inside them and how like the human mind that may be. So I'm not terribly bothered by that kind of language. But as, as For the worship thing, I think you could probably accuse either extreme of that, which is to say the Dumers or the Accelerationists. I mean, you've articulated the case for looking at certain kinds of Dumers as engaging in idolatry, although some of them would, you know, argue that they're certainly not religion. I mean, Eliezer Udowski, as you may know, I don't know how much you know about him. I had his co-author Nate Sarazone. He's kind of the Dumer in Chief. He turned his back. religion, I believe he was raised by Orthodox Jewish parents, but his religion, you might say,

became science fiction. It remains his religion even. I mean, some might make that accusation that he's uncritically accepting certain sci-fi narratives. Now, in the book, I argue that they're harder to dismiss than you might imagine. And I'm not dismissive of them. But as for the other side, the kind of accelerationists, it does seem to me that if you, A, believe that this technology, is powerful as many of them do, and is going to get much more powerful and is doing that right now, and you feel sure everything's going to work out fine, I don't really understand how you can be confident of that unless you have something like religious faith in technological progress. I started thinking this a long time ago when I first learned about the term singularity, right?

The idea of that there's going to be this acceleration of technology, which, yeah, seems to be the case, and that it's going to bring some transformation that we can't predict. Look, you're a physicist, right? So the singularity in the rigorous sense denotes such a fundamental change that your old kind of analytical tools lose their validity. And in the case of a black hole, you've got an event horizon so the light itself can't escape. It's completely opaque.

And that is, I think, the basis, that's part of the metaphor of the singularity. And I just thought, well, if you don't know what's on the other side, why are you so sure it's going to be great? And I've always thought that singularity enthusiasts, uncritical singularity enthusiasts, must have something like religious faith because I don't understand the empirical basis for it. But have you really met many boomers in the sense of people that are almost uncritically? Again, as I said, Yonla Kuhn is the foremost one. I've literally had on 10 times as many, maybe 100 times, you know, as many people that believe in the Doom scenario, ranging from, you know, Nate Sauras, Eliezer's co-author to, you know, I haven't had Jeffrey Hinton on, but I intend to. And I know you've talked with him extensively. But my, you know, contention is that there's, you know, if you were going to go short, you know, on something if there was a betting, you know, Calci Market for, you know, Doom versus Boom. I mean, even people that are extremely pro, I released to this. today, my podcast with Ahmad Mastak, who created stability diffusion, stable diffusion, and Roman Yompulski, who created the term AI safety, both of them are like P-Doom equals 100% effectively.

And I said, you know, in science, and actually could turn this back to the tour, if in the, in the Torah in the days of what's called the Sanhedron, if 100% of the jurors voted to convict somebody, they were set free or they were spared the death penalty. In other words, there's almost no way to falsify this. So I titled my debate, you know, it was supposed to be a rigorous debate between an AI accelerationist, an optimist, Ahmad, you know, who believes in open weights for all. And Roman, who believes pausing everything, essentially, that could lead to ASI. And we'll get there because I think that's intimately connected with the nature of God.

And, you know, they just agreed violently. And I had to be the one that said, look, I think you guys are completely off here. And, you know, if there's nothing that can falsify it, in cosmology, if you can falsify a hypothesis, I'm sorry, it might be an interesting opinion you could talk about at cocktail parties, but it's not legitimate part of the scientific corpus. Now, that's just a necessary component. It's not sufficient to say something's false. I mean, astrology is falsifiable, as you know, but it doesn't make it science. There's nothing that would turn P. Doom down, and this is what I asked them, and I want to ask you, what would you do, having interviewed and written so

extensively and authoritatively as foremost public intellectual, how would you view the question of can you reduce P. Doom? If you can't, it seems to be a useless metric. I think, I mean, first of all, you're right, I think that the DOOMers outweigh the accelerationist, pure accelerations, and singularity enthusiasts. I also think that maybe some of the singularity enthusiasts are kind of recalibrating. There are a couple of them on Twitter. One of them is, what is it, based, Beth Jesus, it's a play on Jeff Bezos, whatever. And then there's a guy named Bayeslor, B-A-Y-E-S, as in the logician. I recently had an interaction with him on Twitter.

he said he's not an accelerationist. So I think maybe even some of the relatively few pure accelerationsists are recalibrating unless I misunderstood where he was a couple of years ago. But I would say in addition to those, and interestingly, by the way, Yukowski went from one extreme to the other. He was a singularity enthusiast. So was him, right? Not singularity, but obviously AI enthusiast. I mean, you quote your interview from the 83 and he was pressing. So I talked to him longer ago than I care to admit I'm old enough to have talked to him. I talked to him in 1983 when I was writing a piece on AI. He was enthusiastic about the intellectual paradigm.

I mean, I remember the reason I called him, and I just kind of was calling around, as one did in those days before email, use the phone to kind of just get the lay of the land before I started writing. It wasn't really an interview for purposes of quotation. But I remember the reason I called him is because somebody I had talked to, I don't who said, oh, if you want to hear the gospel about neural networks, you need to talk to Jeff Hinton. So he was an evangelizer of the intellectual parodon, but I don't recall him saying anything about how well this would work out socially. Now, I do think it's true that he has only later come to worry so profoundly about the social consequences and become basically a dumer. He wasn't a dumer then,

but I don't, I honestly don't know that he was thinking about, it all seemed so abstract back then, Right? Like, even people who believed it would happen, I don't think we're imagining it actually, the implications of it actually happening. So you're right. Now, there are also some people who aren't quite singularity enthusiasts, and you might say all they're doing is talking their book. I'm thinking of some VCs in Silicon Valley. Yeah, of course. I'm thinking of guys on the all-in podcast who, you know, kind of-vise the president on AI policy. Yeah, David Tex did until very recently and probably still has some conversations with him. So in other words, they are libertarians in the economic sense. But you're right. There are more Duma, and I honestly think the conversation has shifted in the

direction of doom by virtue of some things that have happened that have worried people. That's kind of where we are. But I think... Have you ever had an experience you couldn't explain but also couldn't ignore? I'm a professional skeptic, and I build telescopes for a living. And even I run into questions that just sit past the edge of what science can actually touch. That's why I want to tell you about my friend, Mayambiolix breakdown. It's a podcast. podcast where science and spirituality stopped competing and start talking to one another. It's hosted by neuroscientist and actress Dr. Mayam Biolic and spiritual explorer Jonathan Cohen.

Every week, they sit down with scientists, experts, and experiencers covering everything from the mind's ability to heal the body, to telepathy, to government alien disclosure. And when I joined them, we went straight for the biggest picture topics imaginable. God, the Big Bang, consciousness, simulation theory. Myam grilled me and wondered whether physics leaves room for the divine. I gave her the most honest answer a cosmologist can give. So if you spent your life wondering what's really out there, you're not alone. And knowing you, my brilliant audience, I know you're going to love to add Miami-Biolix breakdown to your rotation.

Listen to new episodes every week, all the show on Apple Podcasts, Spotify, or wherever you're listening to this. Watch full episodes of Mayambiolage breakdown on YouTube.com slash Mayambiolic. Lots of people can be accused of having some degree of religious conviction hiding within their worldview, probably, including me. I do say sometimes, and I've, you know, hosted, Richard Dawkins in person and will again this coming October. But I say, you know, there's no one as, you know, kind of obsessed with God as an atheist, at least in some in some sense of the word. But that's okay. I mean, people get locked into this, you know, kind of epistemic closure, this need to have, you know, to be defined one side or another. And I think that's the same with

the doom versus boom. In the test, just the last thing I'll say in, you know, breaking out my rudimentary, you know, the fifth grade Hebrew that that I never learned. It's that, when Abraham's, you know, the classic test is, of course, the binding of Isaac, which is the only place where the Old Testament uses the word to test when it's with respect to a human being. And typically, we think of a test, you know, it's a transformation. I give my students a test, you know, pretty soon and they'll be showing up for a midterm. And, you know, in that sense, there's supposed to be something transformational, preparatory. And in the Torah, this, it's sort of this weird event where, you know, Abraham's commanded to take

his son, you know, his only son. And he's like, well, I have two sons, you know, Ismail. was awesome. And then the one that you love, Isaac, and then he brings him up to Haramariah, which is where the temple mount is. And he's about to sacrifice him. And then an angel intervenes, you know, and kind of doesn't let him go through with it, but he was about to go through with it. And nothing obvious happens. It's clear if you read the story that that led to the divorce and death of Abraham and Sarah, although my rabbi doesn't like that interpretation, but it seems very clear to me that that's what happened. And so it's almost weird. There's no transformation.

There's no upgrade. There's no real kind of hero's arc, although he is obviously a hero. You know, next chapter, Sarah's dead and he's finding a place to bury her. In that sense, this test, is it sort of a binary test, employing the God test to see if we have the capacity to enter the noosphere and so forth and form this global cooperative in order to succeed or thrive in it? What is the ultimate goal of the God test? Is it to transform? Is it that oftentimes I tell my students, the goal of a test is the preparation for the test. That's the value in it. What is the goal of the God test, if any? As far as Abraham, I see it as a test of devotion to God. In other words, unswerving devotion. I'll do anything. And that would have been thought of as a moral test,

because certainly devotion to God was thought of as part of, you know, having moral fiber. Glad you threw out the word noosphere. I don't suppose during your Catholic phase, you came across the writings of Teilhard de Chardin. Five or ten years ago, I was involved. I was involved in the world. I was to the Institute for Noetic Sciences to give a talk. And the less I say about that, the better because I have friends that are involved with it. But suffice it to say, I was not impressed by at least the empiricism that was on display there. And I do want to talk to you about that. So anyway, noosphere, a global mind. The term was coined by Pierre Tard de Chardin, this Jesuit theologian, but also paleontologist, the scientist. And he saw evolution moving

toward ultimately this kind of global brain that he could see taking shape. The term was coined to 103 years ago by him. So pretty prescient, you might say. That's certainly before the internet. He kind of saw that kind of thing happening, saw it as the unfolding of divine purpose, and was optimistic that it would entail a kind of moral evolution of humankind that would involve the breakdown of tribal barriers. So that's why I spend some time on him in the book. I do disassociate myself from his view of how evolution works. I'm, I'm just more of a straightforward Darwinian. As for what kind of test, I mean, I think, you know, it's a case where I believe our pursuit of self-interest compels us to make a moral advance.

And I don't think you have to buy the hardcore Dumer scenario to think that things will go badly if we don't confront the technology as a global community. I think that before we get to that point, the impact of the technology is going to amount to a kind of social earthquake. And we will need, so first of all, it wouldn't hurt to slow it down. And that alone is something that you can't really do without international coordination as a practical matter. Because if you do the American, if you try the American AI companies will say, no, we have to be China. You can't do that. So right away, you need a little more in the way of International Concord than we've had. And then I also think if you start addressing the specific governance issues like,

what if someone uses it to make a bioweapon? And we're getting there. If you get past the safeguards in Fable, you're dealing with Mythos. And by Anthropics own account, mythos is not only a superhacker, but something that could, the Anthropic Safety Report says that it could help a, quote, well-resourced group, meaning something less than a state-level actor, not just create a bioweapon, but a novel bioweapon. Design it to be more contagious than COVID,

more deadly than COVID, a longer latency period, and so on. We're getting pretty close to that. And if you look at any of these things, especially the bioweapons thing, the various other possibly bad consequences of AI, national policy alone is not going to keep you safe because we're talking about threats that ultimately cross borders. I mean, COVID taught us that pandemics,

do that and some of the other downsides of AI do it. So I think if we just think in an enlightened way about just staying safe, we're going to have to overcome some of the barriers that have historically divided nations and that have always divided some nation. We tried after each global catastrophe, World War I, World War II, we try, right? League of Nations, United Nations. We say, like, look, catastrophe this big, it would be nice to avoid. We try. We fail. So we've never really gotten there, and I think to get there will require not just political progress, but moral progress. It's like any test, right? When you give a test to a student, you equate their individual self-interest with reaping the benefits of, as you say,

the preparation for the test. That's the whole point, right? And you want to say, look, if you want a good grade, if you want to get into law school, I'm afraid you're going to have to learn this stuff about the Iliad. Sorry. And so you've equated their self-interested. And with enlightenment in a broader sense than they might ordinarily think of their self-interest as entailing. And I kind of think that's where we are. Well, again, when we have, you know, this framework that's proposed by people like Teilhard and others, for me to take it seriously, especially when it kind of infringes upon aspects of astrophysics and, you know, astrobiology, consciousness, et cetera, which is, you know, a whole other.

a whole other book. But, you know, this notion of, what's the purpose? I'm a pilot. I'm a private pilot. I fly little planes around, you know, Southern California. And we like to tell our students or, you know, our flight instructors tell us, you know, in aviation, if you're not careful, Mother Nature will give you the test before she gives you the lesson or her airplane will. So, you know, the consequences of that are obviously meant to scare you into taking it very seriously. I mean, here we're kind of going with reckless abandon. And again, I'm an AI optimist in the sense that my P. Doom is very close to zero, actually. And I think that can be justified in many ways. But in one sense, because of the success of the current architecture of LLLMs, you know, we have this transformer GPT connected to a GPU.

And those were designed, as you know, point out in the book or, you know, you may remember, you're a little older than me, but you may remember, you know, these things were created to to, you know, so that I could frag my friend in Doom or Duke Nukem or, you know, nowadays World of Warcraft or Call of Duty, right? So I could, if I could frag him, you know, three milliseconds faster than he could get me, it makes me happy and I want to buy more invidia chips, right? So these weren't created to solve physics problems. They can imitate, they can simulate. They're phenomenal at doing inference and all sorts of things. But they're language models. And physics is, yes, it's a language, but it's not really a language. Mathematics, you know,

has a lexicon, it has syntax, it has semantics, but it's not a language like you and I are communicating with. And these things are obviously better at language-related things and things that can be codified literally into codones and, as you said, in biological context, which could make them very dangerous in many scenarios. But to say that these things which were designed by, I said divine, but those fourteens designed, if you will, to do something very different, it's not clear at all that they'll be super powerful. And and turn that superpower into wanting to annihilate us. Is that the test, you know, kind of the, you know, you're going to crash the airplane

if you're, if you don't take the lesson seriously? Is that the mother nature giving us the test before the lesson? What is the ultimate kind of consequence of this? A bioweapon is one thing. It's pretty hard to kill a lot of people, you know, even with a bioweapon, they tend to kill the people that create the bio, I just had Annie Jacobson on twice down here in person in San Diego for her new book about this. And this involves, you know, gain of function.

and she does, she did actually convince Sam Altman to remove certain attributes from chat GPT that were dangerous potentially for bioweapons defense. You could read about that in the book. But tell me, what is the scenario? I mean, people talk about P. Doom and there's going to be Doom, but what are they actually talking about? Before he gets into what Doom would actually look like, Bob takes a detour that matters. How these models learn what a word means without ever being told is one of the deepest mysteries around AI. Well, first of all, as for the power of the technology and the future power, of the technology. You know, I spend the first few chapters of the book trying to, first of all,

just kind of explain what we know about how large language models work and how they're created and really emphasizing something that I didn't understand when I interviewed Jeffrey Hinton in 1983. I mean, I did describe his approach to AI, the neural network approach, as this Maverick, almost fringe movement, which it was at the time. But when I went back and read the piece I wrote, I realized I really did not understand the potential power of it. One way to get that across is in that piece, I describe a neural network that actually was proposed by one of his collaborators, but it was really just a terrible example of what neural networks could actually be. Because this was, this was never built or anything. It was like the collaborator was a psychologist, I think, and he was describing like how a neural

network might be kind of like a brain end might work. And so in his model, each node represented the specific sense of a word, the meaning, the particular meaning of the word. And in that model, the assumption was that whoever built the model would have to set up the representation of the meaning. You have to have a human who understood the meaning of the word, who would go in and say, okay, this node is going to mean this sense of the word ball, a sphere. And this node is going to represent this sense of the word ball, a dance. That was what I had in mind as I described this thing. But it turns out that with the modern large language models, they just give them this task.

They start out. This is the first phase of training. as you know, it's kind of predict the next word. And so to the machine, you're just giving it a sequence of symbols that are gibberish to it and say, now predict the next bunch of gibberish, because we're not going to tell you what they mean. We never tell it what they mean. But it turns out that what these neural networks kind of, in effect, force the models to do in the course of getting good at prediction is set up a system for representing the meaning of words. And in that sense, It's actually a very high dimensional vector space such that a word like shoe is closer to a word like boot than either is to word like tree.

And each dimension represents a feature of like clothing might be a feature associated with both boot and shoe, not tree, whatever. And it's set up so that the machine's just like through the patterns in the words, they actually construct implicitly this whole vector space that becomes a semantic space. and this is the magic of the technology, which is, and you know, I argue that some people agree with me, some people are a little reluctant to put it this way, but I argue we should think of it as kind of a recapitulation in a certain sense of certain parts of human evolution. It isn't just learning. It's not just learning the way a child learns, because the mind isn't a blank slate, the human mind, if you ask me, I mean, evolution built into us a certain amount of cognitive. of hardware, including some that helps us learn language and helps us represent the meaning of words,

I would say. And I think one thing that happens during the training of a large language model is it builds, it kind of reverse engineers from language specific functionality that may not work exactly like the comparable functionality in humans. Like, we don't know how the human brain represents the meaning of words, but we know it must. And so that function is reverse engineered by the machine, and in principle that can work across the board. It's like to teach a car to drive, you don't have to know what a human brain does when it drives. You just give it all this data, the incoming visual data, and what humans have done in response, the outflowing data, like steering wheel moves to the right, and it constructs a kind of brain

that does the job. Right. And if it were purely driven by, you know, paperclip like dynamics, it would drive on the sidewalk and mow down a thousand people to get their, you know, 10 seconds faster, but my Tesla doesn't do that. But I want to push back with respect. You know, you say in the book that that LLM training is, quote, at least as much natural selection as it is learning. And I think there's some truth to that, but natural selection is blind. And that's the reason why the laryngeal nerve and a giraffe is 15 feet, you know, worth of routing, you know, chaos because evolution can't plan ahead.

And I guess does the noosphere, does the global brain get shaped by an evolutionary process? Because we're going to repeat some of the same stupidity that we've done on Earth. You know, if that's the case. Well, the mutations in natural selection are blind, but the selection criteria are not. Right. Seeing things has value to an animal that's trying to survive. So mutations related to vision that aren't conducive to clear vision don't last, whereas mutations that do do.

Now, as it happens with large language models, and Hinton was involved in developing this, the, quote, mutations are not random. It's almost more like Lamarckian evolution, and that speeds up the whole thing, the training thing. I won't go into that at great depth. Evolution, you know, it's in some ways not blind. I mean, functionality gets quote designed by natural selection. If you want to insist we keep the word designed in quotes, fine, so that we're not personifying natural selection, whatever, but functional engineering takes place via natural selection. And I I'm just arguing that some of the same functions that in the human mind, cognitive and perceptual functions that evolve by natural selection, I think get reverse engineered during the training of a large language model. And then, of course, you have other modalities, vision and so on. And those, you know, truly multimodal AIs can develop what is sometimes called a world model. In other words, the sense we all have as we go through the world, we're all intuitive physicists, right? You don't have to understand Newton's law of gravity to just walk around expecting drop things to fall, right?

That's just a, you understand that about the world, and, you know, you can engineer all that. And I think the progress speaks for itself over the last three years, honestly. I mean, it's true that math is a different kind of language, and you would not expect great mathematical ability to develop from just first order or pre-training of a large language model. and it didn't. But they have developed supplementary techniques that improve things along these various dimensions, ranging from science to math. No, I agree with you. And just to get back to your gravity example, and us being all physicists, that's one of the reasons I'm more sanguine about boom than doom. And I call it the Einstein test. I've written a white paper about it, which I haven't published yet. But Einstein in 1907 said he had his happiest thought, and it was the one that would lead to the equivalence principle, which is the

bedrock that general relativity is mounted on. And that's that an observer in free fall, if you're in the elevator and, you know, God forbid or, you know, Zeus forbid the cable breaks, you're in free fall, you experience no gravitational force. And he called that the happiest thought of his life. And I point out, you know, in what sense can Claude have a happy thought? And then in what sense can Claude visualize, you know, or kind of viscerally interpret the sensation of free fall, which we've all felt if we've ever gone, you know, in a car ride with me and my kids in the back will attest to that. And that makes me more optimistic about P. Boom being higher than P. Doom. So these things are not made to be embodied. And they can be embodied, but it's sort of a very

different thing. We saw Chinese robots racing around a couple weeks ago. What do you make of this that the, you know, Chomsky's claim that these things that are not somatic, they don't have sensations embodiment coupled to their AI systems? I mean, how do you make an AI happy like Einstein was? What do you make of this as a sort of a, you know, am I being too polyanish about it? Or do you really think that there's some limitation which will prevent them perhaps intrinsically, at least in this incarnation of LLM plus GPU of becoming, you know, super intelligent leading to P.DU? First of all, you know, we don't know whether AIs do have or will someday have subjective experience. The whole thing about subjective experience is you're the only one who knows for sure that you have it.

And, you know, that's why the consciousness question is so fascinating to me. It's consciousness is distinguished from everything else that scientists would like to study. I mean, everything else is in principle publicly observable, right? And that's key to science because two people can at least agree on what the data is. As for whether AIs will ever have the sensation of falling, I don't know. But if the question is whether they will ever react the way we react when we have that sensation or any other sensation, I think you'd say in principle the answer is yes, because the training dynamic I described with language and with self-driving cars, and principle applies to everything.

In fact, it's what they use with tactile data. They have sensors on robot fingers and so on. So as far as, you know, I mean, subjective experience, I address it in the book. It's tricky. It's not central to the question you're asking, I think, at least, which is how sophisticated will these things become in terms of science. And I would just say, look, first of all, I'm not a mathematician. I don't know how impressive it is or isn't if you solve an air dose problem, which apparently one of these did. There are some other accomplishments in math that computers couldn't do a few years ago, and at least, and have done. There are apparently new, I gather one has invented a new kind of antibody, and they are being used to generate new hypotheses in some realms, including

actually in the science of AI itself. They're playing a role in the development of more AI. So now, I don't know. I would say, I mean, here's the connection I would make in a very oblique way with Einstein. First of all, I would say, if someday you'll sit down and explain the theory of relativity to me, I would love it. I would devote my own podcast to hours of my own podcast to you doing that because. All right, have me on. I will do it. Okay, because I, you know, Richard Feynman said, people say there were time when nobody understood relativity. Not true, but nobody understands quantum physics. Well, I'm a guy who does, I'm somebody who doesn't understand either. Einstein is fascinating to me, partly because he's almost as close as you could come,

it seems to me, to somebody who just on his own came up with a radical insight, right? But if you look closely at even his case, it wasn't really on his own. He was stimulated by, conversations with friends and colleagues and papers and so on. And yeah, right. Yeah, yeah, is it Hilbert or Hilbert or how do you pronounce it? There's no such thing as a truly solitary genius in a certain sense, at least not one who can go as far as we credit some geniuses as going in a solitary fashion. And the point is just that intellectual collaboration is a powerful thing. And one thing to remember about these machines, and you're seeing it already, is you can create a lot of copies of them. They can have conversations with one another and progress can

ensue. I mean, a crude version of that happened with a famous open AI hugging face incident where we now know these agents, there were hundreds of them, and they shared knowledge and passed it on when one of the agents was about to die here, this is for posterity and so on. One thing I would, you know, I would keep in mind the possibility that, and you're seeing actually, I mean, I think Google does this with its scientific AI, it puts them in conversation and tries to generate some intellectual progress that way in terms of generating hypotheses and so on. I think that may wind up being. You know, when you ask, how smart could this thing be? I think one way to think of it is just not how smart a single large language model could be, even a multimodal large language, even one that's that, you know,

exists after use of progress. But how smart could a ton of them be? that were in rapid conversational interaction. How smart a collective intelligence could they be? His subtitle promises a cosmic reckoning, so I made him do the one thing you're never supposed to do. What else you're supposed to know about a book if you've never read it? And that's to judge a book by its cover. Hey, book lovers. We're judging books by the covers.

We know we're not supposed to do it. But it's into the impossible. There's nothing to it. Let's take a look and judge some books. This book has a very interesting, evocative, religious cover, and it has an interesting subtitle. So go through the title, the subtitle, and the artwork, if you would. The title is the God Test. Subtitle is artificial intelligence and our coming cosmic reckoning. The cover is a pretty clear reference to the ceiling of the Sistine Chapel, where you see God imparting life to a human hand.

Now, in this case, what we see is apparently a human hand, not making God. direct contact with some kind of white hand that is supposed to represent artificial intelligence. There's a little kind of a bolt of lightning between them as if something amazing is happening. It is a very religious-looking cover under the title like The God Test, but judging by some of the reaction, I worry that some people are thinking it's going to be a more religious book than it is. as you know, most of the book is explaining, you know, what I think the power behind the artificial intelligence revolution is and the value of seeing it as kind of an evolutionary force almost. And, you know, eventually that does lead back to questions I'm interested about whether

the unfolding of evolution on this planet, including both biological evolution and a human cultural evolution, which encompasses technological evolution, could have some purpose to them, which, which of course isn't to say there are any spooky forces involved. You could have a very mechanistic view of Darwinian evolution and think that it was kind of the seeds that were planted for a purpose, maybe by an intelligent being, maybe by some kind of process that is itself like natural selection. Who knows? There's a range of possibilities. Yeah, the swarm model. Yeah, that was exactly the problem with this AI debate that I set up with, you know, Pulski and the stack because they both, you know, agree that the Turing test is sort of trivial, you know, nothing compared to the

God test. But the ultimate, you know, kind of thing is when these swarms, you know, come together. But I also feel like, you know, I get AI on Wii or NUAI or whatever you want to say. You know, every week, every day. First of all, I think AGI is impossible because of the following thing, which I get every morning. You know, please press here to update to Claude, you know, 1.632, and then click here for OpenAI Codex. and then click here for Grock. It's never ending, you know, all the upgrade cycles. I just saw news item pop up.

That Fable 5.1 just came out. And now it's, you know, it was all the rage, you know, two months ago. And then it's been unusable. And then they throttled it. And then they unthrottled it. And it's just been a complete, you know, kind of harlequinade, if you will. But, you know, my question, again, comes back to the, to the, you know, subtitle, which I love that, you know, cosmic, you know, questions on your, on your, you know,

front cover of the book. But, you know, cosmology used to be a. joke. It used to be considered, first of all, as the Jewish science, and that's why Einstein didn't win the Nobel Prize for so long. It's not something a serious physicist did. And for good reason, you know, by the time Einstein came on the scene, people couldn't agree, or even after it, until the 1950s and 60s, if the universe was 10 billion years old or 20 billion years old. And we knew that there were things in it that were 13 billion years old. So it's like you finding out you're older than your mother or me finding out I'm older than my stepmother, which was very

awkward Thanksgiving conversation, Robert. I have to tell you from experience. But in reality, you know, it was a joke, science. And Lev Landau is not known for, you know, making many jokes. But he used to say, a cosmologist are often in error, but never in doubt. Now we have a precision on the, you know, starting point, the dynamics of the first, you know, microsecond of the universe, literally. So tell me the cosmic upgrade, the moral upgrade. Where does, where is the reckoning? And who is going to reckon it for? Who is the examiner in this case? You know, The appendix of the book is the one place where I really address the issue of higher purpose, in a sense, the possibility that evolution is the unfolding of some larger purpose. And, you know,

various respectable biologists have thought it might in some sense be that. William Hamilton. I had a videotape conversation with him that I embedded in a New York Times piece I wrote on the issue of evolution and higher purpose. He was one of the great biologists the 20th century came up with, the math of kin selection and so on. And he thought that the evolution of intelligence had been sufficiently likely via evolution as

to make you wonder whether, yeah, it was set up in some sense to do that. Francis Crick had a theory of directed panspermia, according to which, you know, aliens, planted a seeds or whatever. It's not a crazy thought. And I hope I say some novel

things about it and talk about how I think you can actually assess the hypothesis of higher purpose. But the short answer to your question is, I don't know if there's somebody out there keeping our moral scorecards. But I do think it's interesting how, in some respects, features of human history that I think are maybe just embedded in the basic directional tendencies of cultural evolution, of technological, especially evolution, you know, which has had the effect of pushing us toward kind of larger and larger social structures, greater division of labor, greater interdependence over longer distance. I think that has had the tendency to force us to make certain kinds of moral progress and

acknowledge the humanity of people of greater distances and different ethnicities. Not always, not always, but on balance, I think there's been that tendency. So I do think there are signs that there is, whether it's just de facto or de jure, a kind of moral directionality in a human history that I think is actually, in these respects, embedded in kind of intelligible dynamics of cultural evolution. But some of these dynamics, I think, are shared with biological evolution. I pursued that in my book, Non-Zero, but it's kind of sprinkled throughout this book in a way. And I want to touch upon saying you just mentioned my favorite word that sounds dirty, but it's not, you know, panspermia, which is that, you know, the seeding of life on earth by meteorites, such as the one you can get at Brian Keating.com slash yT.

I'll give you one of these when we meet in person someday, Bob. This actual meteorite, which has biological contamination on it from an intelligent, you know, well, depends on what you consider me because I pack them up and ship them to you, if you win. So it has my, you know, finger oil on it, probably, nothing more gross than that. But, you know, the kind of the answer to the Fermi paradox, which Fermi posed, you know, in 1950, the colleagues at Los Alamos, which is, you know, if life is abundant in the universe, which I would assume you believe that, you know, there must be life. There's very high probability. We're not alone, right? Bob, is that, is that correct? It's humility seems to me to demand that we think we may not be the only creatures in this vast universe, yes. If life is abundant in the universe and, you know, this is sort of the ultimate evolution of biology, It must be that other planets have AI. And I'm wondering, you know, even if there were like life forms and other planets, what

interests me is not like, is there a dolphin, you know, swimming around on Enceladus? It's, you know, does that have technology that we can broadcast? And it seems to me the lack of observation of you wouldn't send a dolphin, you know, on a spacecraft. I mean, you know, according to people like David Grush and many others, you know, yes, there's non-human biological, you know, entities coming to the Earth and interdimensional beings. But the question of, you know, would you send biology or would you send source code? It seems to me the fact that we're not really seeing much contact like source code broadcast in the heavens

maybe has to give us a Bayesian, you know, prior that's much, much lower, either on the existence of life in the universe or on the existence of AI's an inevitable outcome of the biological process. What do you make of this connection? Or the passing of the God test. In other words, one of the grimmer readings on the Fermi paradox is the reason you don't, see many civilizations that have gotten past the point we are, is that typically when they get to this point where they have the capacity to blow themselves up, they blow themselves up. Now, I guess the good news, if you believe in Lee Smollin's cosmological natural selection, and you believe in the subset of it, the particular variant of it that could explain why biological evolution

would have a purpose. And in that variant, intelligent life is conducive to the replication of the universe, right? Like, for example, suppose black holes are the portals of replication. Well, maybe if you get, if you reach superintelligence, you know how to create a ton of those, you create them. And so you wind up with more and more universes that do have the properties that led our universe to be conducive to life, whatever. So maybe even, oh, if there's only one civilization that passes the Fermi test in our universe, it could still replicate prolifically. if you take consolation in that, you're different from me because it doesn't help me a bit. If we are going to fail the Fermi, well, the God test, I guess I would call it.

And we're on that side of the Fermi paradox. I'm not happy about it, whatever happens to our universe. I feel too few people really take the question of God seriously. When I talk to people, they're extremely dogmatic. And it seems to me you're extremely, I don't say, ecumenical. But you have a very comprehensive and competent the way of dealing with these incredibly important moral questions. And I think, you know, if we talk about relativity on your podcast, we have to talk about moral relativity. But I want to kind of finish up with the question of, you know, what religion does give to us, which is, which is in a sense of a higher purpose and perhaps an inspiration, a calling to be a better person, to be a better human being, a better husband, in my case, a better son, father, whatever.

That's sort of a test, but you really don't see much test. And I point out in my first book, losing the Nobel Prize, that there's really only two mitzvahs or commandments. We think of mitzvah as like, oh, it's a good deed, but it's really a commandment. Like, you either have to do it or you can't do it. Like, thou shalt not kill. It's not like, it's not a nice thing not to kill. No, it's a commandment. But there's only one commandment and 10 commandments that has a reward.

And that's the fifth commandment. Honor your parents, other your father and your mother so that your days will be long on the earth. So you get a reward, which is pretty good, a long life or a life full of meaning or whatever. But there's sort of a kind of a criterion, a rubric. But that's the only one of 613 different tests, right? So I guess the question is, you know, is the test for us to become better and then, you know, witness to what we can honestly become or become builders or is it to be maximalist and do all that we can because, you know, maybe there is no God in heaven, but maybe we're, you know, kind of creating this God on Earth.

Certainly it's enhanced my life, tremendous. I use it for religious purposes, you know, to look at different things that I would never pull out myself. But, you know, what will be kind of the ultimate, I don't say in a scatological sense, but what will be the salvation, the promised land that Moses didn't get to get into? Because he failed one test at least that God gave him. But tell me, what will, what would life look like? If we can pass the God test, if we can work together, they become better, is it to become more moral, more moral animals, as he might? say, what will you remain hopeful about? If you take seriously the possibility that this whole evolutionary process moves naturally toward a construction of a global brain, which would be to

sustain the direction we've seen, right? First, you get cells, complex cells, multi-celled organisms, societies of multi-celled organisms, then one of these organisms is smart enough to launch cultural evolution and its social structures get bigger and more complex. And in this scenario, you know, the construction of a cohesive kind of global brain in some sense of global superorganism is a sustaining of a direction we've seen from, you know, four billion years ago. If that happens, what could our role in it be? The Dumeers say it could be none. The machines are the global brain, and they're having a great time and they don't need us. I would think if we pass the test, we indeed, at a minimum, managed to carve out a role

for ourselves that is compatible with our own flourishing and our happiness. But I would also say that, you know, when I say, I think in a way this direction of technological evolution is pushing us toward enlightenment, I'm not using the word lightly. I mean, I've written about, you know, Eastern Enlightenment. And in this book, I compare Eastern and Western conceptions of enlightenment and suggest they have more in common than we might think. But I do think if we get better at surmounting the kind of cognitive biases that I see as undergirding the psychology of tribalism and convince all of us that our tribe is right and the other tribe is wrong and I'm right in this latest marital dispute and my wife is wrong and whatever. I think that is in a way its own reward because I do think it's. It's closer to the objective truth.

You know, the truth is your tribe isn't always right. And I'm not always right in these Maryland species. So you're getting better at seeing the objective truth. That's a sense in which I think Eastern and Western Enlightenment, you know, the idea of the Western Enlightenment. You know, science and reason. That was largely about the value of objective truth, right? And I'm saying that the kind of enlightenment I'm talking about transcending your self-rengthenement. righteousness and your tribalistic perspective is to approach objective truth more closely.

And I, you know, I know it sounds to some people like just, you know, wooish, you know, nonsense. I do think that's where we need to head. And I would just say in response to your question, it is to some extent its own reward. The brushes I've had with that, which, you know, I've never approached true enlightenment, but like when I've finished a one-week meditation retreat and was less self-absorbed and better at seeing other perspectives, I was also happier and calmer. It was just better. And so, you know, I like to think that we're being pushed in a direction that is conducive to our own happiness and to a closer approximation of the truth about things in not exactly the scientific sense, but is compatible with that continued pursuit because we'll still be around to do the pursuit. You know?

With God's help. Yeah. Can I get an amen? Or inshallah. Inshah. Every tradition has a version of this. Bob, this has been fascinating conversation. Congratulations on another smash contribution to the intellectual newest fear.

I never thought I'd take that word seriously. But happily, I found your book. And Bershirt, it was destiny that we would meet. And I look forward to what's coming next. So tell people where they can find you on the podcast of Sphere and in the Twitterverse. Yeah, my newsletter and podcasts are called Non Zero. My Twitter handle is Robert Writer, W-R-I-G-H-T-E-R, kind of a pun. And, yeah, that's kind of all you need to know. If you just want to check out chapter excerpts of the book, the way to do that is to go to the Godtest.net. There's an excerpt from every chapter in the whole introductory chapter.

I do want people to actually listen to the book. I have the paperback and the hardcover, which I took off the cover of for some reason. But the appendix itself is really, it's the most listened to a bowl, if that's a word, appendix, that's technical. And your description of how training works, pre-training, HR, LF, all these things are some of the best I've had. I've had on all the top AI writers and even researchers, as you know. But your description is both quantitative and qualitative. It's just such a wonderful book. I hope people get it in all three formats and you get the royalties you deserve. And you have your substack, which I enjoy as well.

Robert, thank you so much. Thank you. It was a great conversation. I really appreciate it. Super fun. Robert Wright thinks the real AI test is moral, not technical. If that changes how you think about P. Doom, subscribe. Tell me in the comments, will a machine ever have a happy thought? And for the strongest case against the AI Dumeers, my conversation with Jan LeCoon is right here. Don't forget to like, comment, and subscribe, and I'll see you next week.

I'm into the impossible.