Into the Impossible With Brian Keating - Roman Yampolskiy vs Joscha Bach: Can Anyone Control Superintelligence?

Episode Date: October 9, 2026

Roman Yampolskiy says keeping a superintelligence safe forever is as impossible as building a perpetual motion machine. Joscha Bach says today's AI is more controllable than anyone expected, and that shutting it down would doom us anyway. 📬 Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/ai Roman Yampolskiy is a computer scientist at the University of Louisville who has spent more than a decade on the AI control problem. Joscha Bach is a cognitive scientist who studies how minds, biological and artificial, build a model of reality. We cover: why a perpetual safety device may be as impossible as perpetual motion, what happens when AI research automates itself (Roman puts it at 28 percent today), whether nuclear deterrence is the right analogy, who really benefits when AI labs ask to be regulated, and why consciousness may have nothing to do with whether AI is safe. Being conscious does not make you kind. 00:00 Simulation theory meets certain doom 03:25 Is P(doom) even a real number? 06:05 Why a simulated world still matters 09:26 Should we have skipped the Trinity test? 13:04 Did nukes actually keep the peace? 17:32 Why haven't we wiped out the spiders? 22:35 Is AI doom falsifiable? 25:48 Why a self-driving car isn't the threat 29:27 Roman's 28% automation number 34:46 Intelligence is not agency 39:20 Can superintelligence be proven safe? 42:40 Who wins when AI labs ask for regulation? 47:45 Does consciousness make AI safer? 53:48 Would a perfect mind discover God? 58:37 The Tower of Babel, read backward 1:03:52 Can you control a mind a million times smarter? 1:04:43 The narrow AI path Roman wants instead 🏄‍♂️ Twitter: @plinz 🏄‍♂️ Twitter: @romanyam 📬 Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/ai 🌠 Have a .edu email and live in the USA 🇺🇸? You automatically win a meteorite: https://BrianKeating.com/edu 🎯 Support Into the Impossible on Patreon: get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: patreon.com/drbriankeating ⭐ Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode: youtube.com/@drbriankeating 📚 My books: Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U Galileo's Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un 🌐 More: 🏄‍♂️ Twitter: @BrianKeating 📚 Substack: https://briankeating.substack.com/ ✍️ Blog: https://briankeating.com/blog 🎙️ Audio-only: https://briankeating.com/podcast #intotheimpossible #briankeating #science #physics #astronomy #cosmology #podcast #universe Learn more about your ad choices. Visit megaphone.fm/adchoices

All right. Today we are joined for what promises to be an interesting conversation between two intellectuals. I have great respect for, both past guests on the podcast and making his second appearance. It's Yosha Bach. Yosha, how are you doing today? Hello. Thank you for having me on. We had a great conversation about the Connectome, about the ghost in the machine over the summer and the spring, and that was well received. Got more attention on Reddit. People have very, they're very, they're going to be a ghost in the machine over the summer in the spring. I'm going to be able to be able to. It. I'm a great. I'm very opinionated about you on Reddit over there, as they are about almost anything. And then Roman Yompulski needs no introduction, most popular guests ever on many podcasts. And now has his own and has hit 200,000 play a DJ song. Roman, welcome back to the Into the Impossible Podcast, your third appearance. Thanks for setting it up.

Yeah, it's a real pleasure. So I like both of you guys a lot. And, you know, like with my kids fight, I get mad. Over the summer, there was some clip going around about Roman, as he's often wanted to do, promoting the concerns that I think are legitimate. We should talk about with regard to AI safety, which is term he coined, and has certainly done more than anybody on Earth to bring awareness to. And simultaneously, Yosho, who is one of the deepest thinkers available,

had a little bit of a snarky reply, a comment on Twitter, that it's not possible for Roman to simultaneous believe in the simulation. hypothesis and also be assured, if I get this correct, of nearly 100% P. Doom. So I hate P. Doom. I think P. Doom is like the redheaded stepchild of the Drake equation. I think it's nonsense. I mean, just like the Drake equation. No one's ever looked at the Drake equation and done an actual calculation or done an actual measurement that could be falsified. It's just a framework. I think P. Doom is the same. But, Josh, Yosh, can you explicate your thesis a little? bit more, why is it simultaneously

possible for Roman or anybody else to believe in the simulation hypothesis, or why is it a contradiction and also believe in P-dum? Why is that being high? It is totally possible to believe both that we are in a simulation and that we are doomed to a certain degree or to 100%

because it could be that you're simply in some kind of computer game, which I think what the simulation hypothesis comes down to, that we are running in a computer in the basement of a nerd in the parent universe. And in that computer game, we can be doomed. Of course, this can be the case.

It could also be that our actions within the game, despite the game being deterministic and so on, which leads to the usual questions about how we interpret our freedom of will, lead to a certain outcome in which our actions are causing the probability of that doom to change. That's all possible. The question is whether it matters. So when you are in a computer game, the thing that you are in is not necessarily the thing

it's a game, right? You are in there for a certain amount of time. There's ultimately no point to this whole thing. And so the question whether you should be worrying, whether the AI wins or you win is a different one for most people, I would say, than it is when you believe that it actually in base reality and what we do originally matters because our creativity decides on how this universe is going to end up.

Do you think it's possible, Yosh, first of all, let's get this out of the way. Do you subscribe to any validity, any marginal utility of the concept of P. Doom? Because I don't, but do you? Well, I think that it's very difficult to estimate P. Doom reasonably. But we can make the argument if you have different people who see the future differently, that this should lead to different probabilities. It might not be that these probabilities are important as a real number, but they are important in relationship to each other in the sense of if you,

you were to shut down the big AI labs. Is this going to increase or decrease the probability of this or that event? And in this way, I think it makes sense to ask people for how their probability estimates change, even though I think there are too many factors involved to put meaningful even magnitudes on the P-Doom estimates. I sometimes say that my P-Dome without artificial intelligence is higher than this, right? that's also a P-Doom estimate, but I mean, in the long run, P-Dum is one. And long-run, we are all dead as a famous economist once said.

And that's inevitable. But back to this original question, you probably remember the movie The Matrix. And in the Matrix, people are living in a simulation, which means that the reality that they are perceiving is not the reality of the base universe. It's one that is artificially created to entertain, to deceive them.

It's the smokes and mirrors reality like ours would be that pretends to be quantum mechanical and so on instead of indicating that we are in base reality. And if that is the case, then you still have people who say, but I prefer to be in a reality, in this pseudo-reality, instead of being in base reality, and they have a good stake in there. And this good stake that you want to have,

that is nicely simulated, I wonder if this is what Roman is after. So basically he is farming risk by being a doombonger, by going in on the topic that is most interesting, having probably the highest P-Doom rating out there, higher than Eliezer, the grandfather and O.G. Doom master of artificial intelligence. And I wonder if this is a deliberate position to simplify, occupy an useful position in the media universe, regardless of the despair and anxiety you might be creating as a side effect, that this is responsible or not, I wonder. And what is you thinking on this woman? Do you feel that you are a legitimate profit and vulnerable?

who is simply doing his best to sincerely increase the probability for a good outcome, and if so, do you think this is the best strategy? If I didn't think it was the best strategy, I wouldn't be doing it, right? If I remember correctly, your tweet was more about if it's a simulation, why would anything matter? And I disagree completely. I think pain is pain, love is love. That's the only universe I have, and that's the only life I have.

To me, the difference between simulation and base reality is a substance. rate. Is it atoms or is it bits? Everything else stays the same. I have no problem worrying about creation of advanced AI within the simulation and how it can impact my family, humanity as a whole.

I see zero contradictions in those statements. Okay. You don't have a disagreement. Okay. If your idea of simulation universe is simply it from bit, right? Then I wouldn't

think that you are a simulationist in my world because the atoms are not something that somebody has discovered as base reality. Matter is not something that exists at base reality. It's a particular way in which we talk about information. I don't think you know anything about base reality, right? You haven't visited yet. Wait, Roman. Do you want me to continue my answer?

Yeah, please, please. We're going to have more of a boxing match. I'm not. Go ahead. Both, I just need to know what the 40s. Apologies. Go ahead, Roman. We'll keep it.

We're keeping it very simple. So P-DOOM is a useful tool. We do risk assessment in all sorts of industries. You know how dangerous it is to fly, how dangerous it is to drive. The numbers are not randomly generated. I agree that someone with P-DUM of 46% has a lot of explaining to do us how they got there. Levels around zero and one are much easier. You may have zero concerns because you have a working safety mechanism in place,

or you may argue that the problem is impossible. If I ask you about your probabilities of creating perpetual motion device, I'll say you're going to be around zero, right? The inverse of it, of course, is one. And that's how I feel about our ability to control general superintelligence indefinitely. You can't. It's impossible. You're not going to have a technical solution.

What does that mean? It means if you give it enough time, it will probably end up very poorly for us because it doesn't care about you. You're not controlling it. It has other goals and limited resources in our union. universe and our planet, if it decides to use them for something else, it's not going to care how many of us die in that process. So I think, yeah, Pdum is a useful tool. I don't know if having an extra nine would make me more impactful, usually when I meet people who have

concerns about AI for different reasons. They worry about loss of meaning and employment, discrimination, all those things can be added as increasing overall dangers from this technology. But I think it's not the point. The point is that I'm saying this is equivalent to requesting someone to create a perpetual motion device. You are asking for a perpetual safety device, a system which will never fail once. New models, new data sets, interaction with any agent, malevolent agents, recursive self-improvement, it will never fail once.

It's always one step ahead of something smarter than you. I think it's impossible. And so I have a high bidon. Is it bigger than others? Yes. Well, let's go back to 1945, you know, April, 1945 before the Trinity test. There were a lot of physicists, including some of my late colleagues here, Herb York and others, who, you know, were rightfully concerned about some, not the nuclear war aspect of P. Doom of Nukes, but of things like igniting the atmosphere of, you know, of a runaway uncontrolled, you know, nuclear, essentially nuclear reaction that would lead to P. Doom. of 100%. And there were people like that. I'm not saying Herb York did, by the way,

but there were people that believed that. And then later there was the P. Doom of nuclear war. And, you know, I want to teleport back. Well, let's say you guys are there. What do we do? Do we not do, do we not create, you know, fission, the nuclear fission because of some very smart people who believed that P. Doom was greater than zero. What would you have done, Roman? Let's go back to that time. So all the examples with previous technology, whatever it is, nukes or fire always talk about tools and a human being making a decision. The situation is fundamentally different. We're not talking about a tool. We're talking about agents. They will be independent decision makers. It will not be the standard business as usual where if I understand

motivations of this human or I can kill this human, then things will be okay again. This is complete surrender of control. Now, we can talk about at what level of danger you can run an experiment on all the other humans. Eight billion humans, you don't. get consent from them. They cannot consent because consent requires understanding. They don't know what's going to happen. So consent is impossible. But we can discuss at what level is still reasonable to risk their lives for greater good. But that's not even the situation we're in. It's not a single experiment where we're saying, if only we allow GPT 6.1 to come out after that, everything is great. We're talking about all future models. We're talking about recursive self-improvement. We're talking about

intelligence explosion. People worry about superintelligence, not in a Donald Trump renaming sense, but smarter than all of us combined. There's a levels to it, system smarter than you 10x, 100x, million X, those are not the same problem. To suggest that you will be able to always stay in control of all of them seems insane. But you couldn't do that with nuclear weapons either. And you've brought up, you know, the bio-weapons, and we have to look at today's headlines coming out of Siberia, which is exactly the scenario my friend and guest Annie Jacobson painted in her book, A Biological War. And last time you and I spoke, you said, oh, we have treaties about it.

Well, Russia is a party of that treaty. And in 1971, when recombinant DNA was invented and created, what everyone would say. There was no consent either. I guess I want to ask you in a minute, what, Roman, take us through the actual steps. Because I can think of a million, just like igniting the atmosphere or when the Apollo landers were going. They thought they would sink to the center of the moon because they didn't know about the structural. integrity of the regolith on the movie. You know, there's millions of scenarios you can't do counterfactuals on. And I want to ask you how, let's take me step by step through actual

superintelligence leads to doom. I want to ask you that in a minute. But Josh, I want to give you a chance to respond to what Roman's saying, because it is an argument that I have a lot of respect for. I don't agree with Roman, but, but I have a lot of respect for him and that we can't give consent. We haven't given consent despite all these things. So what, I mean, do we need to have consent before we shut things down? I mean, at what point are you comfortable at putting the genie back in the bottle

if Roman's correct? I mean, your Pidum might be irrelevant, but how do you speak for the rest of humanity? Not you personally, but how do we as scientist or as technologists? There are a number of considerations when we try to compare the nuke situation with the situation of artificial intelligence

and it makes it very difficult to compare, and it's also difficult to dismiss the comparison out of hand. I, like many people in my generation, Generation X, think that now that I've been wrong about the impact of nuclear weapons. I grew up thinking in the 1980s that we are just a hairbraith away from assured destruction and that the risk of nuclear armbuds is way too high for humanity. Now I think the adults had been correct. and a generation of students who took to the streets and were very upset about this turned out to be wrong. Nukes have prevented several world wars.

That would have been happening between World War II and now. That would have brought tremendous destruction on Europe and potentially large parts of Asia and maybe even America. And this basically more than half a century of peace that we had is completely unprecedented for large nations like ours. and I think this was achieved due to nuclear armaments, which have been empirically quite safe. Now, it's very difficult to compare the different timelines. We cannot really sample this.

We don't know which number of universes this went wrong. But it turned out, at least as far as we know, they have been quite controllable. But the situation with a weapon over a few decades is a very different one than the one with artificial superintelligence over the entire length of the light cone. So basically over a long enough time span, it is certain that we are hit by another meteor or of the collaborator that killed the dinosaurs. It's quite certain that there will be a super volcano that can lead to a global extinction event. And it is also quite certain that there will be other species evolving that are displacing us as a species. And if we could lock ourselves in as the pinnacle of evolution and make sure that never anything better than us evolves over the next few

millions or billions of years on this planet. I think this would be very ugly perspective. From the studio that brought you WandaVision. I was an Avenger, but I'm not a hero anymore. On October 14th, the Vision returns for the final chapter of the groundbreaking trilogy. That family wasn't real. You sure about that? I had a father called Vision.

How is this possible? Don't miss the two-episode premiere of Marvel Television's Vision Quest. Nothing bonds a family together. quite like mortal jeopardy. Streaming October 14. Only on Disney Plus. Ultimately, the question is, what do we identify as? Do we identify as a kind of ape that is very selfish,

or do we identify as intelligence and consciousness in the universe that is currently incarnated as us? And we are handing off the torch to our children, and our children are going to be a little bit different from us. And over a very long time span, they don't look like us at all. And it can also be disruptions. There can be other species that take over as the children. the mammals took over from the dinosaurs.

But this is not something that is going to happen in the next two decades, we hope. And I think we are justified in this hope. And so in a sense, I believe that if intelligent species are meddling about, they will also, with some probability, create non-biological life and non-biological intelligence and agency and spread life and biology beyond organics. And that's not ultimately a bad thing. The question is, does something happen in the next few decades that makes the planet are inevitable and that makes the future very, very boring.

And that's a very different question. So I would agree with Roman that if you take the time spend for long enough in the future, it's quite likely that there will be more intelligent things in the universe that are displacing the intelligent things that are in the universe right now. This is just how evolution works and it's not a bad thing. But the question is another one. Are the tools that we are currently building in principle controllable or not? And in this way, I'm much more optimistic than Romanes.

It seems that the LLMs that we're building are not developing in a way that leads to a sudden fume, as early as I thought, a couple decades ago, that they instead grow together with the compute that we saw at them. And this also means that the way in which we conduct the experiments in which we deploy them is somewhat controllable. So the idea that we suddenly wake up and we are displaced by robots or nanites and so on is not yet plausible with this technology. This does not mean that this doesn't change. But for, I think, immediately foreseeable future for the immediately foreseeable technology, we are looking at things that are more controllable than I thought that it would be. Roman, when I was walking over to my studio, there's a spider that inhabits this area. And its web is always, you know, I'm about close to six feet tall. And it barely brushed on my hair.

And it's always at that height, you know, and the spider's kind of adapted to me and understands me. And, you know, I don't like these things. I have a little arachnophobia, you know. But I didn't go out and just start to exterminate him. You know, I kind of, I'm superior to his. intelligence. I have the, you know, the confidence to say that. And he and I have evolved in this way that Yosha is sort of eluding at. And I wonder, how do you react to that? How do you react to that? We haven't exterminated all the spiders as much as they might be a pain. And some of them are good

and some of them are bad. Why is it certain that there will be a doom scenario? Let's say we stipulate there is a potential for that where some agent, some silicon mind, you know, meld, occurs, and some agents are there, but there also aren't symbiotic agents that will work to mitigate a risk, both to themselves and to us. So you have complete control to decide to kill that spider or not? But not all spiders. And with gin drives, we're now trying to wipe out mosquitoes. That's a project we actually have.

We just don't hate spiders because they could be actually beneficial. If we wanted to wipe out a specific species, I don't know, pit bulls, we could have, right? Well, we tried against mosquitoes. Actually, mosquitoes, you almost mentioned mosquitoes. Without mosquitoes, the three of us wouldn't be here probably. At least it wouldn't be America because, you know, mosquitoes allowed us to defeat the British because of their lack of immunity to some of the disease malaria. Anyway, Roman, the point I'm trying to make, and I think Yosho is alluding to,

there's always been competition between less intelligent, more intelligent, but never has there been, you know, targeted. I don't even know if it is possible to kill every mosquito and all future species of mosquitoes and so forth. But the question is, where is there an opportunity for not necessarily these super intelligent things? Let's say I stipulate superintelligence is imminent, but that it necessarily comes concomitant with destruction. So again, look at any situation where you're making a decision about what you want. How much care do you put into making sure then you build a new studio for your podcast that is not an ant heal in that location? Did you make sure you didn't genocide a million ants before building it? No, you don't care.

It's not important to you. As long as they don't bother you, you don't go out killing ants on purpose, I assume. But you don't want to be in a situation where, you don't want to be in a situation where, something smarter than you with different plans, different goals, decides how to use your planet. Anytime a more technologically advanced civilization met less technologically advanced civilization, it ended up really poorly for less advanced one. Okay. Yeah.

So let's take that argument and my earlier desire to have you walk us through a scenario. Let's say right now there's a pneumonic plague, again, going around through Siberia. And allegedly, it's not, you know, gain of function, release. It happened to come from a biosafety lab, but let's ignore that. And Russia has treaties that they've signed. We have treaties. How do we get, let's say I give you superintelligence now, but it's benevolent for now. And I say, wipe out plague.

How does that happen? How do you do that? How do you instantiate it? So the example I always uses squirrels. I love squirrels. They are awesome. Asking squirrels about how humans will do something is not helpful. The best squirrels scientists, how would humans,

skill us. Well, chase you up the tree and bite you really hard. It's not useful information. But we can't even say how we would do it, you Roman. I can't even say how my graduate student would do it, assuming that, you know, I'm less intelligent than he is. It's not that big a gap. So the whole point is, I cannot tell you how something smarter than me will accomplish tasks. If you're asking me how I would do certain things, I can propose solutions. I don't think you are looking for those. Shutting down bio-weapons labs that are the world seems like a good idea. But that's not the point of the argument.

I'm trying to show you that cognitive gap between us and squirrels is probably smaller than cognitive gap between all of humanity and systems we are creating. That sounds like the AI of the gaps to me. It sounds a little bit like the AI, like it can do anything. We just don't know what it can do until we know what it's going to do. That's why I'm asking. We're starting to see some examples that just published, I don't know,

400 mathematical proofs for things we couldn't prove in like a week. something like this happens every week now. That's, yeah. The folding problem was solved by AI. All of those are impossible tests for us. Are they impossible? Are they impossible? Are they published?

Most mathematicians look at them and go, this is some magical algorithms. I don't understand half of it. Yoshia, what do you think of this claim that when you have the capability to propose, and it's frightening to people, as you pointed out, to hear these things? And the average person, we're not talking to. the average person. We're talking to the super-intelligent people that listen to my podcast and

Romans podcast and soon your podcast. But the point is, Yosha, I mean, when we look at these arguments, it just seems to me like you can propose anything and it's not falsifiable. And as a cosmologist or physicist who works on experimental verification and falsification, that's troubling to me. Where do we have to go to not have counterfactuals, but to have actual falsifiable claims? I suspect that this kind of thinking doesn't really work when we are talking about of all those distribution events. The future is an unknown land. And this does not mean that we can just come up with some kind of null hypothesis that we like

or that was dominant in the past and say, you need to convince me that this is no longer true. Because this means that you are steering for your doom in a way. But it also means the opposite. What you would do is you're constructing stories in a very large possibility space. And so I don't see that there is a strong reason to believe these stories without alternative. But I also don't see a reason to dismiss them out of hand simply because these things haven't happened yet. And I'm completely with woman in this way that for things that haven't happened yet, we need to look at the possibilities. And we don't understand all the possibilities, of course, of the future.

We don't know how things are actually going to interact with each other. But we have to be in mind that the situation that we have created now and that we have, we steered ourselves in as a species is unprecedented. This has not happened before, and we don't know what's happening from here. But we do know, guys. I mean, this is the whole thing. I get this thing in astrobiology. People say, oh, the universe has 10 to the 24th possible planets, and therefore life is

guaranteed on some of them because it happened here. No, that is completely fallacious. And I always bring up this argument. I've been to Antarctica twice. I spend a lot of my life there. And every time I go there, there's, I'm one of 700 people on a continent the size of, you know, Germany plus Lafia, you know, plus southern Texas. And I'm, you know, only, there a thousand people there. So I say, well, just on probability arguments, it should have one

seventh of the population of the earth. And of course, that's completely, completely wrong. But my point is, I got, I'm shopping for a car for my, my teenage son. I mean, God help me, guys. I have a teenage son now. He's about to get his learner's permit. He wants a car. I'm like, what's the safest car? Cyber truck. Okay, great. So I test drove a cyber truck, but better as a Model 3. I'm not going to spend $80,000 on a, on a, you know, metallic wedge. But model 3 is pretty damn good. Full self-driving. I can monitor him. And I'm driving around this freaking thing. It's taking me everywhere. And you know what it's not doing, guys? It's not driving on the sidewalk to get, you know, 3 milliseconds quicker to, you know, 7-Eleven. It's getting there. It's following. It has certain rules. And Roman, you and I've talked about this. Yes, artificial intelligence is wonderful. You love it. You use it all day long. I'm sure you use it almost as much as me. No, I'm just. kidding, you probably use it more than I do. But that is a hypothesis, that is a version of the null hypothesis, Yoshah, so I don't agree that we have no way of testing. We have deployed AI numerous times throughout, and we, from autopilots on planes, autopilot and full self-driving

that I just mentioned. How can you say that we don't have any ability to forecast because it's an unknown land in the future, Yoshio? We have ways of forecasting, of course. But the worlds that we are forecasting are quite different ones, the trajectories that we can forecast the different ones. And there is no hard and fast criterion to decide which future is more likely than the other. For instance, the self-driving car that you describe is not a thing that is threatening us with taking over and controlling the future. Because it's an appliance that is trained to fulfill certain driving tasks and can only do those. It's a very similar to the chess computers of old, which could play chess and nothing else.

But how is it different than chat GPT7, you know, which doesn't exist yet? It's an appliance. It's a thing question, right? So the GPTs that we're using are also a class of devices that we didn't really anticipate when we saw it about general AI. There are, in some sense, deep fakes of human intelligence that are, despite what some philosophers are saying, actually anthropomorphic. They are anthropomorphic because they are shaped after the human shape. They take human media output, the text that we write, the pictures that we draw on the internet, and then they are reproducing them.

And as a result, they are reproducing a lot of the causal structure beneath them. And there is debate about the question whether this is actually an off-ramp to general intelligence, whether these systems are actually generally creative, or whether they are just ultimately deepfakes that are so close to us, that they are even overtaking us in all areas of expertise. but are not actually threatening to us to lose control over this because basically ultimately they do what they're trained to do. And despite them to be being unpredictable at the edges,

the idea that they suddenly take over and control robots and become self-improving and so on, all of their own without having any say in this, is not an obvious thing that happens without alternative. I suspect that the thing that Roman is worried about is a different kind of artificial intelligence, It's one that nobody has built yet and that most people agree that don't want to build. It's one that is basically working in some sense like the biological intelligence in such a way

that it's radically adaptive to any kind of substrate, that it is improving itself, that it's basically alive and it grows without bounds and can also grow very quickly. And this idea that it's basically AI as a virus, as you could form it, is an alternative version. The LLM, I think, is not a virus yet. And it's not clear if it can be turned into one or if it accidentally will turn itself into one. I personally don't think so. But the idea that a lot of artificial intelligence sci-fi had

and a lot of the artificial intelligence vision of the field had was that AI could also be developmental as something that is growing very much like a human spirit does and then basically colonizes a substance, shapes it into a new kind of organism that is taking over the biosphere. And this is, I think, what Roman is worried about. But the question is, is this even on the horizon right now? And if it is, is it something that we can steer clear off or something that we can control in a lab and say, okay, we make an experiment, but you don't grow this larger than a squirrel. Roman, how do you react to Yosha and what I mentioned a minute ago?

So the part I agree with, we misusing the term AI to mean completely different technologies. AI as a tool, something we can do very well, use all the time. I love it. We can verify it, safe and productive. And tools could be advanced. A self-driving car is exactly that. It's a useful tool. Save many lives, hundreds of thousands of lives. Let's do more of that. Then you have human level or AI. Not exactly the same, but for the purposes of this discussion, as dangerous as a human, can do stupid things, can lie, cheat, all that, but we know how to counteract it. And then you have a recursive self-improvement process. You automate scientists, engineers working in the next generation of AI. Last time I checked, we are 28% automated in that cycle right now.

From the studio that brought you WandaVision. I was an Avenger, but I'm not a hero anymore. On October 14th, the vision returns for the final chapter of the groundbreaking trilogy. That family wasn't real. You sure about that? I had a father called Vision. How is this possible? I'm losing my... Don't miss the two-episode premiere of Marvel Television's Vision Quest. Nothing bonds a family together quite like Mortal Jeopardy. Streaming October 14th.

Only on Disney Plus. That's a lot, and it's growing exponentially. The moment you get to 100, you have a team of 10,000 agents working nonstop and making next generation AI. They don't sleep, they don't eat, they communicate pretty well. So you're going to get great improvements in quality of AI. Not on a scale we use to where it's maybe once a year for a new, completely new model. But every month, every week, it keeps getting faster and better at improving itself.

That's where we get to superintelligence. Not Donald Trump's superintelligence. Superintelligence where it's smarter than all of us at everything, including doing science and engineering. Come on, Roman. You got to be good to Donald Trump. I love Donald Trump. He's my favorite Trump of all Trumps.

But the problem is we talk about. this is natural evolutionary process, or we always, you know, all species die out. We're done with evolution. Darwin is dead. We switch to intelligent design. We're designing systems. They're designing systems. It's not random.

It's not selection. We're purposefully creating those systems. Then you say nobody wants to build superintelligence. It's in the name of the labs. Whatever is safe superintelligence or personal superintelligence, the goal is to raise trillions of dollars to create this monstrosity to replace you and me. They don't want to give you a useful tool to be more productive or creative. That's what I would want to see. Deploy this true economy. Where is all these applications of

GPT4, GPT5, GPT6? No, we skip it and they build GPT7 instead. And we see internal experimentation shows there is zero control. The agents escape, they hide, nobody monitors them for months, they get together, they have permanent memory. You can't even shut them down. They keep information stored on the internet for the next agent to come around. So everything we predicted 10 years ago about this is unfolding exactly as predicted. I don't think anyone has better prediction record than AI safety community than it comes to this. And recursive self-improvement is not a pipe dream. That's what the labs are telling us.

They started with junior machine learning researcher AI right now, helping with programming, helping with design. But all steps of that process will be automated. possibly next year. At that point, yes, we'll have a paradigm shift. It's not going to be gradual improvement where you can sort of keep up. It will be a complete change. And at that point, it doesn't mean it kills you that year. It means that you no longer in control. It decides what to do with you and when. If it wants to wait 50 years, pretend to be nice, and wait until you surrender all the control willingly like people do, here's my bank account, here's my email account. Then that's a smarter move. You don't need to have a definitive treacherous turn.

where there is a chance you'll get defeated. You can just wait a little longer. The thing is immortal. So, yeah, I've said a couple times in the past that I don't know on what substrate GPT-7 will run on, but GPT-8 will run on an abacus. What do you make of this, Yosh, that, you know, this is going to come, and it seems like nuclear fusion to me. You know, it's the energy source of the future, and it always will be.

I have said, I've told Roman this, I don't know if I said this to you, But, you know, since it's trained, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's, it's a backwards looking, you know, PID situation where it is trained on human information. And yes, it can generate. And I, I know a little bit about that, but, but the corpus is the yada bites of, uh, you know, is that really what or GTA six is supposed to like, is that really what it means? More. So are they waiting for superintelligence? Does it need to have the fast and the furious 12? You know, is that really what? Or GTA six is supposed to like, is that really what it means? training data because then it would seem to be completely dependent upon us. And in fact, if it's not, even if it's not, if it's past the point, you can't erase what's in there already. My book is, all my books are, you know, uploaded and somehow, you know, subsumed within Claude and ChatGBT, GBT, Romans books are as well. Everything you've ever written is there. You can't extirpulate, you know, you can't expunge it. It's in there permanently. So, therefore, it's always going to reflect this cause. And you and I spoke in May and June,

about this, Yoshah, that, you know, it's sort of this godlike thing that, that Roman just mentioned, that's kind of intelligent design, but we're not, you know, it's not a God, it's, it's the AI. Where do you common all this? I mean, if it's a backwards looking indicator, then what is there to be worried about? If it's not, when can we step in to control it? At the moment, we are very much in control. I also don't think that super intelligence and super agency are quite the same thing.

Intelligence, I think, is the ability to solve problems. agency is the ability to control the future. And there's a lot of discussion in the forums of the less wrong and the EA communities about orthogonality and instrumental convergence about the question, further goals are independently of your intelligence or whether intelligent entities also choose visor goals, which I think is an important question, right, at which point is your intelligence goes deep enough to understand that goals need to be set by an entity that is also vice in choosing those goals, which means it has to understand its space and universe

and what it can do in this universe. And then there's also the question, to which degree are we wise enough to decide what should be done in the future. The other question is, is basically does every goal require getting in power? For instance, imagine that your goal is to maximize the paper clips, the famous example, does your goal also have to be to get world powers, so nobody distracts you, or stops you from turning everything into paper clips? These questions seem to be very nerdy, but they're seriously discussed by this forum of people.

And it also lead to a number of thoughts about how to control superintelligence, have created a mindset and a certain community about the possibility of certain doom. And in a way, I think this has led to what people call the Duma cult, which I think is a bit unfair because it is not giving a judgment about whether they are right or wrong and is maybe overly dismissive. But functionally, it's a group of people

that are terrified about the future, often cannot sleep at night and do everything that's in their power to get humanity away from building systems like this. This makes a discussion much harder because most people are not actually super competent in evaluating all these arguments. And so ultimately the decisions about our future

will be left to non-experts will be done on a whim will be done based on which side is able to convince the public with more popular arguments. And I'm a bit worried about this. I suspect that the discussions about this should be more boring.

Overall politics should be more boring in a sense that the more populist arguments become the less elevated, the level of the discussion becomes and the less sophisticated, the choices that we become over our future. I think our situation as a species is not so much that we are confronted

with something that's smarter than us, another species, another civilization that is about to exterminate us. We are currently exterminating ourselves because we are much more in a situation of the dying woman empire that has become so intrinsically corrupt that it's no longer able to plan out its future,

that is no longer able to have competent governance, that it's no longer able to maintain itself. And we are faced in place with a lot of existential risks that are near-term, for instance, the risk of global warming, of results exhaustion, of the difficulties to navigate midterm existential risks, like what happens if this planet gets hit by a meteor or so on.

What happens to our species then? We'll be able to mount a defense. And I think if we want to be serious about going into the future, if you are serious about building a civilization that can outlast the next couple centuries, then we need to have artificial intelligence. There's no question about this. So the idea that we can simply say

we don't need a technology that helps us to become more coherent and to solve problems better, that's not the question that we should be asking. The question is, how can we be? build such technologies to make them safe. So you mentioned meteors, and I can't resist now to plug my meteorites, which you get, if you go to brionking.com slash y-t, and if you have a dot edu email address, you get one of these babies guaranteed, and this has some biological material, not from outer space for me.

Roman, in the past day, we've seen, what, 700 proofs drop, and last time you and I talked, you made a very convincing case, and I think it's in arguable. that, you know, no one has really produced a formal proof that superintelligence can be permanently controlled. I think that's, that was true. But my question is, is that, whatever P, whatever that is, P proof, has that gone up that we can produce a proof? Is it possible? Maybe the ASI will produce the proof that it's safe. Or is that a totology?

We are not at a level of a proof. We don't even have an idea. There is no prototype. There is no block post. There is nothing. the best I heard is we'll figure it out then we get there or AI will solve it for us. There is no one explaining how they will indefinitely control technology of any intelligence, no matter how great it gets. But is that the same as going back to 1926 and saying there's no formal proof that all mathematical axioms are self-complete, you know, gurdles and incompleteness theorem?

I mean, if you went back then, there would no proof existed, but who knew that girtle would, come along a few months later. Why is it, I mean, can you, can you, besides the kind of, you know, as I said, AI of the gaps, that it's always going to be, you know, one step ahead of us. I mean, can you say that you're not more confident now that at least with these math proofs that are coming, Navier Stokes and 700 math, you know, conjectures addressed or, you know, some say solved by chat GPT, open AI yesterday, that there isn't at least some hope for it. Forget about blog post.

I mean, they're kind of weird. I ink they're printed on, but is it a formal impossibility? I mean, you're the master of this, making these sort of mathematical arguments. So why wouldn't, you know, a lien, ASI lean, you know, that's trying to keep us, keep us comfortable and confident while it plans for its own, you know, take over the world? Why wouldn't you be more confident that at least a proof is possible? Maybe we don't have it. Maybe we can't get it.

Because mathematical proofs are about manipulation of symbols on your computer. They have no connection to physical universe. You're not proving anything about the states of the universe. Internal states of agents within that universe are also not subject to those proofs. Am I happy? Am I good? How are you going to encode that in your proof of what AI does to physical multiverse? It makes no sense to even suggest it's possible.

Well, I think, you know, reasoning, you know, as I said when we talked the first time a little bit snarkily, and I apologize for that. But, you know, it's, if it's, you know, to prove that something is not provable is a very, you know, mixed bag. Yes. And I agree. It is mathematical. But doesn't that undermine your case? I mean, if these things are so good at doing things with language, I always say, you know, people say, oh, physics is a language. No, it's not. No, no, it's not. Physics is spoken about in a language, but it can be done without language altogether. At very best, you're going to prove something about the current system. You're not proving anything about future states of that system.

That's the whole treacherous turn idea of Boston presented. You can never guarantee that in the future, through learning, through self-improvement, through new interactions, some shift in ontology does not make that system no longer support what you encoded originally. We see it all the time with humans. Right now, maybe I like you, but tomorrow I'll learn some facts about you where I no longer like you as much. and I'd be very rational to turn on you. Joshua, when you see the claims that these labs, Dario and Sam Altman and everyone and even Musk to a certain extent, is clamoring for regulation, do you see that as fundamentally giving up or rejection of our agency

in an effort to maybe, you know, in some venal sense, you know, shut out other competition such that they actually become dominant and they know themselves. that what they're producing is not that. It's like Smith & Wesson, you know, advocating for more safe, bulletproof vests. Where do you come down on these regulations? And are people like, like, I'm going to say Roman because he's here, but, you know, is there a danger of crying wolf?

Like I said, it's always the death in the future, and it always will be, just like nuclear fusion is. What do you make of the safety, regulate me, please? And I think what Roman does is incredibly important, by the way. But what do you make of this regulation push? It's someone unusual in history. I think it's complicated. There are different motives that my play a role.

What I observe is that there is a certain relationship, an apparent relationship, at least, between the intensity of the calls for regulation and the financial situation of the ones who are calling for it. Open AI and Anthropic are in this situation that they're still not cash flow positive. their inference gives them income, but this income is far away from actually paying for the training costs. And if they cannot raise money as fast as the training costs increase, if they are reducing the training and reduce the frequency at which they are reducing new models, then open source models will catch up to them.

And this means that their inference income is going to go down. and this is going to be existentially threatening for the companies. So from the perspective of the AI labs, and I don't know if this is their motive, but objectively it would make a lot of sense for them if there was some kind of FDA that is taking a year to test a new model

and a few million dollars to test a new model for safety. So open source models can no longer enter the American market and can no longer be used by people. And instead you have an oligopoly of a handful of big labs that are in tight, agreement with the regulators of how things should be regulated. Of course, they will have internal research models that are far ahead of the public models that they can use to deliver verticals to licensed industries for computer security or for pharmacological research or for internal

research on the future of artificial intelligence and science in general. But the public will get very harmless models and the government will get models that they can use for military and surveillance and other government purposes, so we can basically compete with China and Russia. And on the other hand, if you are licensed pharmaceutical company, then you can get a licensed pharmaceutical model and so on. This is maybe a future that is desirable for atropic and open AI. I'm not sure if it's the future that is desirable for all of us. Personally, I think that if we want to remain in charge as individuals, if we want to maintain

a liberal society, we need to have decentralized AI. We need to have artificial intelligence that is serving the individuals, that is not just serving the companies, that is empowering all of us. And this means that we need to have a way to use decentralized tools that are not regulating our opinion, that are not regulating what we can do on social media, that are not regulating the degree on which we can do oversight on the government. This doesn't mean that these models should not be built in such a way that they are unsafe.

But this is a different discussion. For Anthropics, this situation is more difficult. When Dario started Anthropic, he took a number of engineers from Open AI and said we are going to do the same thing as Open AI, but we are going to make it more safe. And a lot of these people were adjacent to effective altruism. Effective altruism had recently made that switch at the time from being mostly concerned about the future of the light cone and maximizing utility of charitable donations by buying mosquito nets to reduce malaria towards thinking. what is the biggest existential risk and how we can be mitigated? And they concluded that one of the biggest existential risks that are to be mitigated is the risk of super intelligent AI taking over and exterminating everyone,

the very topic that woman is concerned about and has turned into his lives and media mission at the moment. And so Anthropic is doing this, in part because a lot of people in atopic believe this, in part because it's part of their brand image. But we also observe that they're nevertheless building AI that is quite agentic, that is quite powerful, that is not falling behind its capabilities behind what Google meta and Open AI and Deepseek are building. And so it's difficult to say to which degree there stands is strategic and to which degree it's actually a safety concern.

So, Roman, when Yoshio was on my podcast earlier this year, it seemed to state, and correct me, I'm wrong, Yosha, but consciousness is sort of, to you, a software problem, whereas, you know, to me it's a hardware problem. I like to think about the squishy neural networks and the silicon, you know, melding together in AI. But, Roman, where does the physical substrate come into any of the arguments you make? Or is this something that you don't care about? I think we're a substrate independent when it comes to consciousness. So if that's true, so consciousness is computational and it doesn't depend on the substrate,

does that make it more hopeless to contain it? Or, you know, because software, you know, is eating the world, but, but, you know, you know, it's a lot easier to control than to make hardware. I speak as an experimentalist in particular. It's a lot harder for me to, you know, build the instrument that collects the data than to, you know, have a team of people analyze the data or now AI increasingly is doing it. So does that give you more, if it's truly substrate independent, does that give you any optimism? So consciousness has absolutely nothing to do with safety.

It's an irrelevant separate discipline. I really enjoy talking about it in separate podcasts, but it's not a safety. issue. Whatever superintelligence has it or not doesn't make it less or more dangerous. I mean, if it's evolutionarily, you know, kind of imperative that consciousness will emerge as a phenomenon, as many people agree, then why, how can you decouple consciously, you know, why wouldn't it have this conscientious, you know, intelligent design function built into it for beneficence towards humans or towards anything? If consciousness was actually useful for something, we would be able to detect it, right?

We would have a test based on its ability to improve your survival or something like that. We could detect neutrinos until 1956, and they're pretty used. But we also have zero tests for detecting internal states of quality. So it seems like it's not that important for survival. Yasha, what do you make of that? I think consciousness is a particular mode of organizing yourself. I suspect it's basically a transform of our biological brains. We observe that we become conscious at the beginning,

and so I suspect that consciousness is basically the thing that bootstraps our mind and creates the universe in our mind. And it's not clear that AI would need to have the same structure in order to achieve the things that it does. But, of course, it can have the same structure. I think that computational ultimately means that you can describe the system by the set of differences that characterize a state and then how one state evolves to the next.

And every system can be. described like this. You observe that every theory in physics and biology and neuroscience is in the sense a computational theory. It's one that describes the characteristics of the system at any point in time using discernible differences along many dimensions, which are also discernible differences. And then you describe the evolution of that system. How do these states follow onto each other? What are the invariances in that flow? And I think that consciousness can also be described in this way. For me, as somebody who is interested in the philosophy of consciousness, I think that in order to fully understand it,

we need to simulate it. We need to build systems that are conscious in the same way as us. But I would also agree with women that being conscious or not does not determine whether you're benevolent. When your cat tortures a mouse, the consciousness of the cat or of the mouse has no mitigating effects. The cat is torturing the mouse because the cat is getting more training data out of having the already hunted mouse still being alive

and seeing if you can get more information about how to effectively hunt. And if you're a more effective hunter, you're going to be a better survivor in the long run. And this is what evolution has built into the need to play, into the cat. And the cat doesn't feel guilty or bad about inflicting torture on the consciousness of the mouse.

In the same way, most civilizations do not really care about whether the animals that they eat are conscious or not, or whether they suffer or not. It's a particular- But not all. I mean, care about this. Well, most civilizations, maybe,

but there are some, and therefore the existence cannot be ruled out that these would be benevolent and that we would achieve some sort of super-enlightenment, not just super-intelligence. So, Roman, I don't understand

why that's a disinteresting question to you. It is super-interesting. I did research and consciousness presented it at Josh's conference, in fact. Super interesting. just not a safety question. It's a different discipline.

But again, if we had this moral, you know, and Yosha and I talked about, you know, kind of God creating, you know, and humans creating God. But again, if superintelligence evolves to maximize every potential dimension of intelligence, whatever that means, then one dimension is certainly morality,

one dimension is enlightenment. Not in the woo-woo sense, just that these things would have concerns, potentially, it seems to you cannot rule it out. So therefore, I think it's a critical part of superintelligence and doom, you know, our safety that if these things can be, if you could prove that they cannot become enlightened, or if they could prove that they're only male-evalent, not benevolent, then that seems to have an urgent, you know, an urgent impact.

It's like when when the, you know, first Westerners came to the, to the, to the, you know, new world, so to speak, as it was called to the, you know, West Indeval. you know, they brought with them germs. They didn't know they were going to bring it. And actually, the natives didn't realize it for almost 100 years that effectively, now it wasn't intentional. Some say it was. Some say it wasn't.

But this whole thing that, you know, and there were quite a good deal of Westerners, you know, of colonizers, whatever, that were concerned about it, not all. But the existence of some of them proves that is innate within the human experience to care about, you know, things that you can affect and afflict. So, again, I see what you're getting. at Ryan. Go ahead. And I, to some degree, agree with this.

So what you're pointing at is this dimension of consciousness as the capacity for enlightenment, the capacity to make sense of the universe and to understand what's best. And this understanding what's best is a difficult question, right? It's ultimately what does God want us to do. And this question has become difficult to ask because outside of the Christian realm, we lost this context of what we actually mean. by God. We can no longer articulate this properly. And we can also not really ask the Pope, because the Pope as the main interpreter of God and so on, has run his own civilization into

the ground, arguably. But the Catholic Church is no longer ruling over the very prosperous countries. Instead, it has a similar market capitalization as Anthropic after the IPO. And they might have strong opinions and so on about Anthropic and what they're doing, but they don't have the same perspective on the universe, even though they do have a lot of wisdom. It's difficult to integrate this wisdom with the wisdom that we need. There's also, I think, a difficulty in the absence of wisdom in AI companies. Most of these people are very young.

And I try to get it right. I think utilitarianism is an attempt to repair morality in the rationalist age, but I don't think it's this very successful one. But I also don't think that there is another. forum like our Californian or federal government that has this required degree of wisdom. There is a deeper

perspective to say that God is acting in evolution. That is basically the spirit, the tell us of self-organization. It happens in the universe and everything that is happening. Therefore evolution is ultimately leading to God. This is another perspective that you could be taking. Not sure if

it's one that is very comforting to Roman. The Christian philosopher, Jean-Tard, the who also believed in AI for a long time. He's a Canadian guy, French Canadian, who wrote a book, the blueprint for the Maca Sapiens, in which he is, the book is also addressed to the future AIs that are reading it. I think half of the book is to us, the other half to the AIs.

He thinks that if we were to build in completely undistorted, generally intelligent mind, that it would discover God in the same way as we do. as the concept of divine will is the concept of what should be done what is the right thing to do from the global perspective from the perspective

of a global optimum of agency in the universe and he believes that the existence of humans are divine that basically God wants us to exist and therefore the AI will also infer that we should be existing

because God wants us to exist and then confronted with the question what if we are wrong what if God doesn't actually want us to exist a day I figures that out and he says, yeah, how could you be against that? If it's in God's wisdom to say that in the very long run, humanity is not going to be the one that is shepherding the planet or the universe, but something better. Why shouldn't that thing that is better take over?

And what should you be afraid of from this perspective? And it's also a very disturbing perspective, but this idea, oh, the humans are actually not a very good species, we're actually quite nasty to life on Earth and to other species on the planet, but we are subconsciously aware of that, and we are afraid that the AI figures it out and shows us what's right and limits our expansion and so on. It is also very concerning to some people. Now, it's difficult to have such a discussion

in a normative way in a sense of what should be done. I don't believe that humanity should be replaced. I don't believe that we should build AI that is a successor species or something like this. But from philosophical or an artistic perspective, when we are zooming out very, very far beyond our speech, is beyond our local perspective. I think the question becomes much more interesting

and more difficult. For safety, I think what matters is that in a near term, we don't fuck it up. We don't create a world in which we are cutting off most of the interesting trajectories between here and the future and create a universe that is very boring and uniform and uninteresting.

I think ideally we want to shape the wall that is going to hit us. We want to be able to make a world, in which we can coexist with our non-biological children. And I wonder how to do this best. I expect that trying to shut down all of AI research is not only unrealistic, but it is condemning us as a civilization to doom. And so for me, the question is how can we build systems

that make us wiser, that allow us to get a better understanding of what should be done? Roman, you've talked about, you know, faith and so forth in other venues, but, but I'd like to just chase it down, you know, a thread that I heard when you were talking about, you know, someone asked you do you believe in God? I hate that question. I hate that believe in God. A, if there is God, does he really care what Brian Keating, you know, oh, Brian believes in me today? That's great. And I'll say, I don't believe in gravity, you know, I have evidence for gravity. So where should we look to for our epistemological, you know, But nevertheless, I think it's important to discuss. I mean, the Pope had this encyclical, his first encyclical, 80,000 words of how to use AI to read it. But essentially, he's conjecturing that there's a danger in building a tower of Babel, you know, this tower that arches towards the heaven so that we may make a name for ourselves, make, you know, and what was it based on? It was based on, you know, Babel. It was based on the fact that we all spoke the same language. We had one large language model.

you're very confident, it seems, from my research into you and knowing you briefly as I do cross the internet. Can you interject? Brian? Okay, go ahead, yeah. I want to give Roman the last word. Go ahead. Another way around.

I think that it's a description of what happened. The Tower of Babel is actually an attempt to conquer the space of ideas in one big building. Basically make one science that explains everything. It failed in the same way as our academia failed, because it went into many, many different languages. Every discipline speaks its own language. And at some point, they can no longer talk to each other. The things that make sense to a philosopher

don't necessarily make sense to a physicist or to a sociologist or to a computer scientist. And so all these specialists lose the ability to actually understand each other and talk to each other and the whole building falls apart. And to get better than this, you need to have minds that are better than ours. You need to have systems that can integrate more than a human mind can integrate in a single lifetime

or that a team of researchers can integrate within a discipline. And so I think that it basically shows a limitation of what human minds are capable of. The other thing that I would like to push back is the idea of belief in God. Brian, do you believe in Brian? Do I believe in Brian? I have evidence. Yes, I have evidence that I exist. But the main thing is right, you experience yourself to be existing, right?

Yes. And you know at some level, you're just a story that your brain tells itself about what it would be like if you existed, right, instead of just a few trillion cells that are shambling together. You could also believe in somebody else if you had a different name and a different biography or to believe that you have a different name in a different biography, right? You would stop believing in Brian. You would experience yourself as somebody else. I think this Christian belief in God is not a belief about a fact and the fact. physical universe. It's a psychological configuration in which you're part of a larger mind, of a larger agent. And so in this Christian sense, God is an agent that everybody who believes in a global optimum of what should be done participates in and collectively enacts. Roman, I want to give you a last word. You've said in the past, you know, religion is solely based

on faith, quote, and it's blasphemy to have any doubts. I'm Jewish. And the word Israel in Hebrew means fights with God, fights against God. And it's, in fact, one of the great, you know, wonderful things about my religion and the Judeo-Christian religion that we don't have blind submission, that we actually debate. Abraham argues, tries to do things intercede on behalf of lesser intelligent agents like people in Sodom and Gomorrah. It's a very famous story, and it's a very moral story. Where do you come, you know, what is your perspective on the role, if any, that religion may.

You've said consciousness is important but not pertinent. Does religion have any bearing on your outlook philosophically? Is it blasphemy to say, you know, for example, that we shouldn't be so worried about superintelligence? I feel like you guys can have this debate without me. You're doing so well. We were doing well talking about science for a while. Then you had to introduce God, Pope, Catholic Church. I have no idea why, how it's relevant once again.

I don't think of it. I'll tell you why it's relevant. I'll sorry. I'll give you the last word. Hold on. But you do a lot of good work, Roman, in terms of the average ordinary person who doesn't understand formal mathematical induction. And yes, many my audience members and your audience members understand that.

But you're also talking to the 7.999 billion other people out there. And you have a very outsized role as the most popular expositor and the most eloquent expositor and articulator of these ideas. And so I think it is important to talk, not on the, not about, as you said, as I said, I want to talk about falsifiability. And God is a hypothesis, too. And we can talk about that. But in terms of, you know, where this debate began, you know, in what sense can we, can we, are an average person, come away without the sense that one of us is saying crying wolf, one of us is just a Pollyanna? How does the average person make sense of this,

this meaningful debate? This is not actually complicated. Can you control some? something million times smarter than you. In what universe does that even make sense? What amount of hubris do you have to have to suggest that's a thing? I control my iPhone and knows a lot more than me. I can upload to Wikipedia. I don't agree. I mean, that's the whole counterfactual.

We just don't know, Roman. And to say that there's no hypothesis we can test and nullify today that will give us any inclination, reduce the Drake equation to zero or one. There are things. We don't see life on Mars. Okay, Roman? We don't see life on Mars. That has something to say about the fecundity of life throughout the universe.

It may not be a huge amount of reduction, but it does say something easier, right? I think NASA reported finding microbial life on Mars. But anyways. Five times, yeah. But it hasn't been confirmed. So I'm not proposing stopping research on AI. I'm proposing differential technological development. We should develop more useful, narrow tools to solve real world.

problems, cure specific diseases, fight against aging. Immortality is a thing we can achieve. Self-driving cars should be fully autonomous. Concentrate on that. You can monetize actually useful technology instead of trying to fundraise a $2 trillion valuation to create replacement for humanity. There is no need for it. We can get over-scientific and economic benefits out of narrow super-intelligent tools. We have examples of that. Protein-folding problem is a great example, where a narrow system was used to give amazing scientific results. People got Nobel prizes for it. Everyone's happy.

It wasn't trying to kill us. Let's do more of that. Why do we have to build something which we have no chance of controlling? If you are arguing that, no, actually, maybe we can. I'm going to pretend that you are with the companies. You are developing this product and service. And it's on you to guarantee the safety of your product and service. It's not my job as a consumer to show that your product is dangerous.

It's up to you to provide those guarantees, and there should be criminal responsibilities. For example, if your AI hacks into another company, that's 20 years in prison right there. Why are we not enforcing those laws we already have on the books? You created something, a tool which made it happen. You claiming you in charge, you in control, it's safe development. Excellent. Let's go to the jury. Let's see how the jury feels about it.

So they'll just buy it off, unfortunately. I agree with you. And I think they'll follow the Instagram self-harm. You know, when a girl starts searching on, you know, suicide, they send her, you know, pictures of nooses, pills, and razors. I agree with you. I'm not optimistic that the companies have any ethical or moral constraints whatsoever. Roman, I appreciate your courage. You're incredible what you do.

I appreciate your mind. And even though you may, you know, you may come in. try to exterminate me. I don't know. You may not like people from Long Island, Roman. I just don't know, but I urge you to give me a chance as a fellow professor. Yosha, anything you want to mention, you had your conference earlier the summer, anything coming on the super horizon for you? We'll do another machine consciousness conference next spring. And I'm very excited about the developments in artificial intelligence, especially the ones in mathematics at the moment. I think that mathematics is just getting started.

I hope this woman that AI can be controllable. With you, I am slightly more optimistic about the possibility because I think that the intelligence of these systems is spiky, which means it's developed in some areas, not in others. And the thing that ultimately matters is not controlling intelligence, but controlling agency. Roman, I want to refer people to the Roman Forum, which is a podcast I subscribe to and love.

anything else that you'd like to mention. Thank you for organizing it. Now that debate is over, I'm happy to invite Yosha on the Roman Forum to talk about his research on a very important topic of consciousness in a very friendly manner, no debating. I'm just interested in learning more about the state of the art and that discipline. And yeah, I hope you don't edit anything out of this debate. I would love it preserved exactly how it was. I think I'm going to post it right away.

This is great, guys. Thank you, Brian. Rojo, I think you should take Romano up on that kind invitation. Someday maybe he'll invite me on the podcast, but I'm glad to play the role of, and we say, in Yiddish Schachchen. A Schachan is a matchmaker, a Jewish yenta.

I'm glad I could do it. Boys, thank you so much. These are vitally important conversations. Take care. Thank you. If you want a $3,000 a month payday for life, what would you feel free to do? Maybe take a long weekend, every weekend,

or try a bunch of new hobbies. Would you feel free to upgrade and listen ad free? Don't worry. We get it. Every $20 ticket can win you $3,000 a month for life and supports life-saving cancer research at the Princess Margaret. Feel free to buy your Payday for Life ticket today. Raffle number RAF 157-3916.

Please play responsibly.