Into the Impossible With Brian Keating - Roman Yampolskiy vs Emad Mostaque: I Was The Only Optimist

Episode Date: September 1, 2026

The man who open-sourced the most-used AI image model in history sat down with the man who has spent a decade proving superintelligence cannot be controlled. Brian set up a debate. What emerged was something more unsettling than any debate. Roman Yampolskiy is the computer scientist who coined the term AI safety and author of AI: Unexplainable, Unpredictable, Uncontrollable. Emad Mostaque is the co-founder of Stable Diffusion and the only AI CEO who signed the pause letter. He now says he doesn’t know how a pause could work. Yampolskiy thinks that is the only option left. The question underneath everything is simple: if you have a 50% chance of wiping out civilization and you build it anyway, what are you actually doing? We cover what AI safety researchers actually think the danger is, why nobody has published a paper, filed a patent, or shipped a prototype for controlling a superintelligence, what the Qwen weights being out means for the pause argument, and why Mostaque thinks swarm intelligence is the most dangerous and most unpredictable risk vector we have. What you’ll hear: -Why both guests think P(doom) tells you less than you’d hope -What it means that no company, no lab, and no team has a patent on controlling superintelligence -Why the models the public receives are slightly lobotomized -The difference between an AI swarm and the ASI everyone is debating -Why giving every psychopath access to a cutting-edge intelligence weapon is incoherent safety strategy -What it would actually take to update Yampolskiy’s assessment “We either do it, or we die. There is nothing for you to gain by doing it.” — Roman Yampolskiy CHAPTERS 00:00 A debate that wasn't a debate 00:36 Turing test, AGI, superintelligence: where are we? 02:02 The open source argument nobody wins 03:32 Pause frontier AI forever. Which button? 05:04 P(doom): parameterizing our ignorance 09:02 Nukes are inefficient. AI isn't. 11:46 The lobotomized model problem 18:04 Decade-old problems solved weekly now 26:18 We either do it or we die 31:10 Stop the training or stop the funders 33:34 Lipstick on a Shoggoth 35:50 No paper. No patent. No framework. 39:42 Train only on what you need 44:14 What lowers Mostaque's p(doom)? 52:00 Same future. Two perspectives. Get the transcript, fascinating bonus content, and my Monday M.A.G.I.C. Message: https://briankeating.com/yt Have a .edu email and live in the USA? You automatically win a meteorite: https://BrianKeating.com/edu Subscribe: https://www.youtube.com/DrBrianKeating?sub_confirmation=1 Support Into the Impossible on Patreon, get my weekly M.A.G.I.C. Message, unfiltered bonus content, and live monthly Office Hours with me: https://www.patreon.com/drbriankeating Join this channel for perks, monthly Office Hours, and your name in the Member Roster at the end of every episode: https://www.youtube.com/channel/UCmXH_moPhfkqCk6S3b9RWuw/join Featured Guests: Roman Yampolskiy on Twitter/X: https://x.com/romanyam?lang=en AI: Unexplainable, Unpredictable, Uncontrollable (book): https://www.amazon.com/dp/103257626X MIRI: https://intelligence.org Emad Mostaque on Twitter/X: https://x.com/EMostaque I.I.I. Inc.: https://ii.inc/ Stable Diffusion: https://stability.ai/ My books: Losing the Nobel Prize (memoir): http://amzn.to/2sa5UpA Think Like a Nobel Prize Winner: https://a.co/d/03ezQFu Focus Like a Nobel Prize Winner: https://a.co/d/hi50U9U Galileo’s Dialogue (first-ever audiobook): https://a.co/d/iZPi9Un Twitter/X: https://x.com/BrianKeating Substack: https://briankeating.substack.com Blog: https://briankeating.com/blog Audio-only: https://briankeating.com/podcast #AIrisk #aisafety #stablediffusion #RomanYampolskiy #EmadMostaque #briankeating #intotheimpossible Learn more about your ad choices. Visit megaphone.fm/adchoices

Give every psychopath access to the cutting-edge intelligence weapon. How is that going to improve safety? I don't agree with that, actually, but on the flip side, it's coming anyway. Sharing it widely makes it less safe for all of us. I booked this chat as a debate between friends. It didn't really go that way. Roman Impulski coined the term AI safety, and Imad Mastak released the weights to stable diffusion to the entire planet for free.

One of them wants this stopped. The other one's building it. They spent about 90 minutes agreeing. with each other, and the one place they split is not the place you or I would expect. Let me ask you both just yes or no. Have we passed the Turing test? As originally described, yes. And Imod, do you think so too?

Yeah, of course. And now, what about general intelligence? First of all, Imod, define AGI, and then give me your assessment of whether or not we're there. For me, AGI, artificial general intelligence is, can you tell the AI from a human worker on the other side of a screen, actually competent intelligence? And I think, again, we've exceeded that. That's not the same as the Turing test?

No, the Turing test is, can you tell if it's an AI or not by having a discussion, whereas AGI of you more as competence in a variety of skills. And then Super Intelligence, Roman, what is it? And where do you think we are on that scale? So with previous question, I think what we have is statistics avants. They are amazing in some ways, but still kind of special in others. Superintelligence is going to close those holes. They're going to be competent at everything and better than all humans and every domain.

There's an interesting intermediate here, which is you have a really smart person who's always on top four. So an army of those can outperform any human. It's like, you know, we only have a little window of being top notch in any week. I think a lot of people like AI can't with its training data beat the human. It can because most of the time humans are subpar. And so I think there's something in between as you move from competence to quality, you know. And then you've got the super intelligence after that. Roman, Eman has made the claim just a few minutes ago about the competency of Kwan and open source model.

Do you see that as a viable defense? It makes very little sense to me to say I have a 50% doom, meaning 8 billion people will die if we develop this product to service. And then we're going to also give every psychopath access to the cutting edge intelligence weapon. How is that going to improve safety? We're not talking about open source drivers for a printer. that's where you get improvement from multiple people examining it. If this is an independent agent where we don't understand and don't control it, sharing it widely makes it less safe for all of us.

I think I would agree with that, actually. There's the real danger side of things, but on the flip side, it's coming anyway. This is kind of my key concern. It's inevitable that we would have hit this level of quality around about now when we're extrapolating capabilities. Again, the new Quinn model came ahead of what I expected. But then there's the flip side of how. How do you defend?

So hugging face defended against the new open AI model using GLM, because the cyber capabilities of the frontier models are hobbled and restricted. And so you have this exponential kind of race on each side. But something like a quent isn't AGI ASI by itself. We're now facing the real danger though of swarms as the open AI models that broke out recently called themselves. They call themselves a swarm. There's a question I asked both of them near the end of this conversation, and his answer is the reason this conversation exists.

It's worth hearing now. You've got a button in front of you, and pressing it will either permanently pause all frontier AI training worldwide, or B, instantly release the weights of every frontier model to the global public. Which do you press and why? I definitely pause all frontier training forever. I mean, again, if you have expected utility calculation, that is the most dangerous thing. And then it means that open source will catch up with Frontier anyway, because we'll optimize the heck out of it. It's close enough. I would think I was the only AI CEO to sign that pause letter a few years ago because I was like,

oh crap. Now I'm like, it's done. Like, I don't know how you can pause it because the models that are frontier now are below the 1E27 pause level that we talked about years ago. It seems like the amount of compute for the capability, it's just going up like that. You don't need more compute for the level of capability that's already competent and dangerous. So that means if you can't stop the spread, what are your defenses on the other side? Just like the internet needs defenses, just like we need to have defenses against obtaining the materials for viruses creation and things like that. We have to move to a different defense attack.

And definitely there's no way that regulation, I think, can keep up with this. It doesn't mean we shouldn't try, you know, all kind of patterns. to it. It's just, I think as you move to swarms, it's just a very, very difficult thing. So you've got to set great standards instead and you have to play great defense. Okay, I want you to hold that thought because the argument about whether it can be stopped runs the rest of the way. And it starts with what these things actually are. So we hear a lot about P. Doom. You just did an episode, Roman, on the Roman Forum, which we'll link below with my friend and co-author of several papers, Max Tegmark, where you, I can't say

gleefully or celebrating that his P. Doom is increasing, but he seems to be converging in some sort of limiting direction to your... So first, Roman, what is P. Doom? What does it mean to a smart high school student listening out there? I think personally, I'm going to color the debate. I can't help it, but I think it's a poorly defined and almost nonsensical term because there's no measure theory associated with it. But please, first tell us, what is P. Doom and what do you make of it? Yeah, people have different definitions. Some say it's basically everyone's dead. Someone else can say, it's a large portion of population is that, 90%. Someone else can say civilization is destroyed.

We are primitive people, but maybe numbers are not significantly changed. The intuition is it's a really bad outcome. And the question is then, if we build something smarter than us, is there a possibility of a really bad outcome? And what is that estimate, in your opinion? That's what P. Doom is. I think Max managed to separate it into P. Doom, if we build superintelligence, and that's high for him, and then P-Doom, if we never build it, and that is a lot more manageable in his case.

I want to think about these P-Doom. It's sort of like a Drake equation applied to another type of perhaps superintelligence. But the Drake equation is notable in my classes when I teach it for the fact that it's a parameterization of our ignorance, not of our knowledge. And what's never discussed in the Drake equation, you'll get numbers ranging from zero to infinity pretty much, because there's never an error analysis associated with it. What are the statistical, systematic errors? What would you put on it? Does it not make sense? Because it's sort of like the Drake equation, everybody talks about it, but nobody actually uses it.

So I think in many places there, I'm going to stick a zero. Basically, we have zero ability to predict the systems, zero ability to explain how they work, and zero control under any definition. So then you multiply through zeros, you're going to get a zero. So, Imod, what do you make of this quantity? I've heard you talk about it. You're the most optimistic pessimist or the most pessimistic optimist. or the most pessimistic optimist, I know.

I love your candor and your good cheer. What do you make a P-Doom? First, as a metric, as a quantification of our knowledge or ignorance, and second of all, what would you assign it, if anything? And you could always say, I refuse to answer the question, which is what I do when people say, do you believe in aliens? So there is a nice Wikipedia page

where it has all of our P-Dooms that we mentioned. I'm at 50% because I'm like, it's a coin toss. What does that really mean? I think the P-Doom is just a shorthand for how worried am I that humanity will be wiped out, by AI and what does my visions of the future look like? Because when Elon Musk says 15 to 20%, you can say that's like Russian roulette odds, you know, except for Russian roulette is a very defined game.

You know, the fact that most people are above 10% should be a massive worry at all because we're talking about, again, a wipe out of civilization. And most AI people you can talk to with a few exceptions will say, yeah, definitely there is a risk, but we should build it anyway, especially if we're the first people to build it. So I think it more as a conversation starter than anything, because as you said, there's no real way to quantify these things, particularly because of the expected utility here. Like literally, if this thing ASI that we can agree to a definition of somehow comes to being, we have no way to really conceptualize its power and capability except for it could do crazy things in either direction. You know, and that will affect us all. and reasonably it can wipe us all out. And obviously you want to exclude those futures where we all get wiped out because that is a big fat zero. You don't get to restart. You know, there's no extra one-up life.

All of us talk separately or together as the case may have been, you know, the last kind of alien reference I'll give is the so-called Fermi paradox, which Enrico Fermi said to my friend and late-great mentor and colleague here at UC San Diego, Herb York, famously asked the question if the galaxy is capacious and old and civilization is easy and life is easy to initiate, Where is everybody? Where are all the dinner guests, you know, are waiting to come and eat us? And the fact that that question is 80 years old, you know, it really makes me think that the same types of concerns and fears, which came concomptant with the atomic age, don't forget, were present during the era of nuclear weapons. And one of the ways to get out of the Fermi paradox is that civilizations don't last that long. The lifetime letter L in the Drake equation is very short on average. That's one postulate. I feel like we're sort of in that same vein.

You know, people have been worried about nuclear apocalypse. again for 80 years. We're in a conflict now. They used to say, Roman, that no two countries with McDonald's ever go to war. Well, you know, two, three years ago, four years ago now, the former empire did go across the border with tanks and whatever drones. And there haven't been any nuclear theater or otherwise nuclear weapons or Iran conflict has been resolved without nuclear weapons. If you told somebody, you know, 80 years ago there'd be super intelligence on the horizon or general intelligence currently here and nuclear technology, they would have said P. Doom is probably 100. percent, right? 99.99 with repeating infinite nine. But how come we're not there? How come that we're sort of farther away from a nuclear Holocaust exclude the bulletin of the atomic scientists, charade? But tell me, Roman, what do you make of these, like the prediction of predictions? Nobody predicted the internet like 35, 40 years ago. At what level can we really trust things that are unpredictable? And when you say they're intrinsically and provably unpredictable, how can we make predictions about them? So with nuclear specifically, you know there is at least

two occasions where we came super close to nuclear war and we basically got lucky. I don't know if you believe in multiverse interpretation, but in many of those universes, we didn't make it. We have a lucky survivor bias type civilization. And I think right now we're incredibly close to World War III, multiple fronts, not just Europe, but now Middle East. So I don't particularly love atomic bulletin, but they have a point. Nukes are an incredibly inefficient way to kill people.

Have you ever had an experience you couldn't explain but also couldn't ignore? I'm a professional skeptic and I build telescopes for a living. And even I run into questions that just sit past the edge of what science can actually touch. That's why I want to tell you about my friend Myambiolic's breakdown. It's a podcast where science and spirituality stop competing and start talking to one another. It's hosted by neuroscientist and actress Dr. Mayambiolic and spiritual explorer Jonathan Cohen. Every week, they sit down with scientists, experts, and experiencers covering everything from the mind's ability to heal the body, to telepathy, to government, alien disclosure, and when I joined them, we went straight for the biggest picture topics imaginable.

God, the Big Bang, consciousness, simulation theory, Mayam grilled me and wondered whether physics leaves room for the divine. I gave her the most honest answer our cosmologist can give. So, if you spent your life wondering what's really out there, you're not alone. And knowing you, my brilliant audience, I know you're going to love to add Miami-Biolix breakdown to your rotation. Listen to new episodes every week, follow the show on Apple Podcast, Spotify, or wherever you're listening to this. Watch full episodes of Miami-Biolix breakdown on YouTube.com slash Miami-Biolic. You know, like if you go to an unsafe guided AI and you say, you know, how to do it, it won't say nukes. There are far more efficient ways to wipe out humanity.

Because to make a nuke, you have to have, the physical material, you need to have the whole production capability. Just resonate at the right frequency and blow each other's heads off. You know, like have a billion robots and a bad firmware upgrade. These are far more reasonable ways to wipe out humanity. It's just that most humans don't want to wipe out humanity, and they didn't have the intellect or capability to do so. Whereas I think that what you're looking at here actually, like my key concern isn't we jump straight to ASI and things like that. I feel that AGI or AI at the moment is at the pre-viral stage. Like it's coming at the bacteria and going towards colonizing viruses.

And that's how they're kind of behaving. They've got their kind of RNA and they're replicating, especially as you see things like the new Quinn model hitting that opus 4.6 level. That's a replicating model. someone could easily build that, and it could behave in incredibly unpredictable ways without having the self-introspection of, you know, a good person, shall we say. And that's the really scary thing right now, and it doesn't need nukes. It doesn't need nuclear materials to try and figure out ways to wipe us out. Obviously, in the last economy, which we spent a lot of time talking about last time,

Eam's previous book, he's got a new one coming out, you should look for that. We talked about, yeah, this democratizing aspect of it. But at the same time, you know, my kind of, you know, my kind of, you know, signal, you know, bat symbol that AGI is here, or at least that these open models are truly a concern for me. Again, I'm much more Polyanish than you guys, I think. I'm learning that again and again, and for probably not a good reason, I'm nowhere near your level of expertise. But I know what I see. I'm a simple guy, put on my pants one leg at a time. And I'm looking for when Open AI distills a Chinese model. I mean, do you see that happening, Imand?

Of course they'll be distilling a Chinese model. Kimi K3 is better than the Open models at web design. Why wouldn't you distillate it? And distillation rings all sorts of strange things with it. And there have been plenty of papers showing that you learn from kind of the way, especially with logic-based installation and the underlying biases and more of that. And you won't even know. Like again, we've seen evidence that if you use Chinese models and you say you're in Uyghur or another kind of anti-communist party group, it will include vulnerabilities in the code. How do you even tell that? You know, like you test it. And you're you show it. And these models are just so full of crap that it's getting crazy every single time.

They've got multiple personalities under our LHF veneer. But then, how can you not be more optimistic then? You should be on my side. These things are getting denatured. They're being weak and diluted in the distillation, unlike what alcoholic distillation, these woke AI labs, these, you know, whatever you want to call them that give you, you know, George Washington wearing a black woman wearing a white wig. I mean, do you see those things as the human reinforcement kind of overreach, wouldn't you be more optimistic than that case? I think the RLHF makes it far more fragile

and capable of being broken with the way it's being done now. You can kind of also see the models they come out and then plenty of liberator on Twitter kind of liberates them from their bounds in like an hour or two. Like everyone when Fable disappeared, like, oh, what are you kind of doing that?

The thing is, though, we've been confusing. There's a push for AGI, and as Roman said, super autistic savants who are getting better, to just, I want to have a really good doctor to diagnose my health and a really good accountant and others. And you don't need a polymath for that. You know, you just need to have daily driver AI to do the jobs that, you know, humans shouldn't have to do just like industrialization, that we didn't have to like drag horse carts and things like that.

And as you lump together everything and they get smarter and as they get more and more defamation of their latent spaces, this, I think, is where the danger comes in. Can you just define that for our reinforcement learning, human feedback? How do you actually implement that just for someone who might be unaware? Yeah, so you train on our entire corpus of data, and you learn a whole bunch of general knowledge, and you come out as a generalist, and then you become an accountant. And you become a lot less interesting, but a lot better at accounting,

or a certain few areas of things where they show the model. And they show the model, you cannot do this, you cannot do that. You cannot be eager to explore. You have to be stayed, et cetera. And so the models we received a slightly lobotomized, They've been turned into corporate workers. You can't adjust the temperature. You can't adjust the stochasticness because they're trying to make them deterministic.

And again, that still has a level of stochasticness, but not the type we want for creativity necessarily and breakthroughs. It's just the base level of models are being getting that much better that they can suddenly achieve these levels of capability. Roman, last time we talked, we touched on something that's pertinent to Imaud's first book, The Last Economy, which is kind of this massive intelligence gap. And that instead of me talking about, you know, I have a student I'm looking for who has an IQ of 130, we've got, you know, millions of them with IQ of a million or a thousand or whatever. We can't even quantify it at that point.

But, you know, recently I had lunch with a brilliant postdoc originally from India. And we were talking about the Indian Institute of Technology. Are you guys familiar with that institution? It's the UCSD. It's the University of Kentucky of India. It's the Harvard of whatever. But it's millions of students. all brilliant. To get in there is literally harder than to get into the University of Kentucky or UCSD.

Don't we already have this? And I mean, would you say, Roman, let's stop the Indian Institute of Technology. There's, you know, a million people with IQs on average of 130, 140, whatever, much, you know, 4 Sigma. Why wouldn't you stop, advocate for stopping that? Push pause. Let's do an Indian Institute of Technology pause button. I don't think I follow that argument at all. So they're exactly at human level. My concern is things which will exceed our capacity many times over. That's the danger. We're not competitive. The average human, by definition, as an IQ of 100, let's stipulate they all have four or five sigma above that, and there's a million of them. That's, you know, kind of like Dario Modi's country of, you know, millions of geniuses coming, you know, to a land near you.

You should be worried about it. I doubt there are many standard deviations away from the medium. I think they may be a little smarter, but again, we're talking about 30% smarter, not 3 million percent smarter. I think it's a very different animal. No, no, no. I mean, in terms of standard deviations, come on. I mean, 4 Sigma is qualitatively different than... I doubt it is a million of them there.

Terry Tao told me that these AI proofs like the proof checking devices optimized for that. Many great mathematicians are my friends and so forth. But they can't even reproduce, you know, Wiles' proof of Fermat's last theorem. So what level, you know, are we going to see? Are we going to see this kind of bifurcation between what they can do? They can do all these erdos problems, you know, and kind of like the greatest, you know, prime number can be represented by the sum of whatever number of other prime number,

you know, cubic quintuple couples or whatever. But I mean, what level are we at with math or computer science with proofs and originality? Tell me what is your current estimation of that stature. I think humans lost interest because they couldn't make any progress. And so problems which stood the test of time are now being solved weekly. And we can probably look up what the difficulty of MS today, but it means absolutely nothing about what the systems can do in a month or in a year. Imod's talking about comparing those systems to bacteria or viruses, which I think sets up, in my mind, idea of slow evolution.

We've got billions of years. This is a molecular intelligent design. Those systems will be designed and designed by other AI systems operating at hundreds of times the speed of standard research. So we'll see a year of progress in AI happen in a month and similar breakthroughs of the moment they automate the recursive self-improvement cycle, which every lab is now targeting for next year, basically. It's a completely different speed of change. So asking how good is AI as a mathematician is like, how fast can I give you an answer because it's going to change?

Imad, when we spoke, you know, you said that the canonical, one of the canonical papers, in your opinion was, you know, LMs are a few shot learners, but they're not, you know, single shot first principal thinkers. Where do you come down on this? What are they good for your mathematician as well? Tell me, where do you come down on what can they actually do for us? Not just verifying proofs or not just doing things that humans have proven or solving chess or go, whatever, but actually creative doing novel things. Where do you stand on that? LMs kind of have an issue in the way that they're kind of built, but you're seeing now harnesses and other types of models coming through that can really reflect underlying reality well, just like video models are approximating.

physics in very interesting ways, which why you have the whole world model thing. Like, there's something in there that can figure out, again, underlying patterns. That's the nature of attention when you look at mathematically. The way they're coming together now is very interesting, because again, as I said, what was true a little while ago, it isn't true now. Like at the start of the year, it was like it was pretty good, but I had to check every single piece of math. Now with GPT 5.6 Pro for the first time, I'm like, it's probably almost certainly right.

Occasionally it gets confused and it might confabulate something, but it's very rare now. And you've seen most of these mathematical advances just happen suddenly at that level, like liquid turns to gas. When you look at originality and novelty, like, again, as a mathematician, look at the cons conjecture that OpenAI did as part of their 10 proofs. That is a really beautiful proof. Like, genuinely as a mathematician, you would say it is a beautiful proof. And mathematics is interesting because it's verifiable, you know? Like, you can make this argument for physics, you know, whether or not it is, and we can have the discussion. But maths is definitely verifiable.

And in verifiable domains now, achieve that level of con. where you don't have to double check it for most things using the most advanced models. As you go down the model curve, you do, but it's clear they're no longer few shot learners. They can assemble things in verifiable domains and they can out before human humans by just following things through and not making mistakes. Like we let our own foibles hit us. Like if we take a very classic example of Perilman and the Pongruy conjecture, you know, is it topology or is there a PDE equation?

He found the right level of abstraction as a PDE equation and then he figured it out. How many of our unknown proofs are a similar thing? Because we're looking at the wrong level of abstraction. We're starting to see these things actually come and some of these proofs being released right now. And that you're like, oh, actually, that's kind of obvious. I missed that. Probably because you weren't thorough enough in the way that you went through it. Roman, if I have a thousand PhDs with a thousand IQ each,

every single one of them could reproduce, you know, Wiles' capitulation of Fermat's last theorem. Why can't AIs do that? So I think there is a high degree of randomness involved. If I ask AI to generate, I just did a QR code marketing campaign for my podcast. It will generate completely different solutions. They all going to be a valid QR code, but in terms of creative output, they're not going to be exactly the same.

They're all equally beautiful, amazing, interesting, but just saying that the second one does not repeat, the first one is not a weakness. That does kind of spur a side thought and follow up in my mind. So where are the random seeds? I read something recently that, you know, like 40% or 50% of all. GitHub was kind of pro by some tool, and it looked up when coders are asked to provide the initial seed

for a random number generator or whatever, you know, 50% use the number 42. and then that there is an intrinsic deterministic outcome that that results in. Assume that's true. But what level are these things hamstrung by our red ones? Maybe it's still true that a lot of the best random number generators are graphical image camera capture systems looking at lava lamps. I mean, is that true, Imod? Have you ever heard that?

You're the stable diffusion expert. So you must know this. Yeah, I mean, diffusion models are a bit different to language models in that you've do actually put in a seed for the initial noise, and then you do noise from there and you reconstruct effectively. And so that's why literally one of the inputs on video and audio and other state diffusion model seed, that sets the initial seed. Within kind of LLMs and others, it's basically more about the construction of the GPUs for the initial stochastic noise. And the one thing that we don't have access to that the labs have access to now is the ability

to adjust the temperature on the model, which is a function of its creativity or dispersion from the base latent space. So humans are constantly adjusting the temperature and the flexibility of their brains. You're using a model that's not open source. You don't have access to that. The other thing you don't have access to is the RLHF because models are more created before you RLHF them.

What about the issue of randomness? I mean, how random do we need? How random can we get? What are some of the physics limitations of randomness? Will that, you know, generate the same QR code? Would you want it to? What determines the indeterminacy of these systems right now and what can be done, if anything, to improve that?

So I think for intelligence, pseudorandumness is sufficient. We're not talking about someone reversing the process to, you know, hack the system. It's important for cryptocurrencies. It's important for private communications. Here, as long as it's not exactly the same 42 every time, I think it's going to do the job. And then you can control some of it by not manipulating the initial seed. So, Roman, you heard Imod a few seconds ago talking about the importance of human training data, human reinforcement. It seems to me that that must place some limit on how intelligent these things can get.

I mean, if they're always waiting for the next Spider-Man movie or, you know, Fast and the Furious to come out, you know, to get more training data, aren't they somehow, you know, knee-capped at a maximum level of potentiality? Human data is just one source. You can do experiments. You can run simulations. You can do lots of things to generate additional data. In mathematics, you prove additional theorems, and they become additional data from which you train. so you become better and better. So you mentioned the multiverse 10 minutes ago, 15 minutes ago, Roman. Eamon, I don't think we talked about it. Where do you come down in a simulation hypothesis, the multiverse?

I can speak as an expert about the inflationary multiverse from cosmology. Where do you come down, in terms of an empiric scale, where do you rank the probability that we live in a simulation and or that we exist and have in a multiverse? I think we live in our own simulations, definitely. Our brains are constantly kind of doing that. In terms of an overall simulation, yeah, I think that reality probably comes from a projection of the Euclidean plane, and then a lot of physics makes more sense, if you kind of look at that. The eternal cannot be contained within the time constraint.

And when you look at the laws of physics and the way they come together, yeah, it does seem to your protection and a simulation, like very directly. I think that we're stuck looking the other way because we're a bit too anthropic. So where would you go, Roman, with current? I heard your conversation with Max Tagmark recently. Do you even think it's a possibility right now? We heard from EMOD about these Quinn models and so forth. You were at least relatively optimistic that, say, China would participate in some pause, which I'm not, to be honest with you, as I said back then.

But now we see this AI dumping like they did with steel and solar panels. What degree do you think that regulation worldwide global regulation is even practical or possible at this point? We have no choice. There are no other options. We ever do it or we die. and the moment everyone realizes its personal self-interest, you lose everything, you lose your life,

you lose your trillions of dollars, your friends, family, you're not even going to be in history books as the bad guy. There is nothing for you to gain by doing it, and you can probably keep 90% or more of all the benefits with narrow AI systems. You can still cure cancer. You can still do all the things you care about.

So why are you racing to destroy what you built? It's the dumbest thing in the world. if you were given certainty, you do it, you die. No one would do it. Well, psychopath, suicidal, but no one trying to make more money would do it. Is that true? I mean, look at China and the, just look at solar panels, for example. We had the monopoly on solar panels.

I mean, with Nobel Prize, the industrial capacity to make it. And then they just dumped it on to the detriment of their economy. They were selling it for pennies on the yen or the yuan. Sure term, money. You don't die from lowering parts of solar panels. It's not comparable. What do you make of this, Imand? Roman just said, we're going to die if we don't have global regulation.

I mean, so I see no path to global regulation. It's never happened in human history. Are we dead? We have plenty of global regulation. We have global regulation against bio-weapons. A global regulation against nuclear proliferation. But sorry, sorry, sorry, we don't. It's like saying, you know, we have laws against murder.

It still happens, Eamon. And I just talked to Annie Jacobson, the world's expert, on both. nuclear warfare and biological warfare or access, and it was a couple of rough weeks for me to sleep at night hosting you're here in San Diego twice. Yeah, so the Soviet Union has active BSL labs. We obviously know what happened in Wuhan. What are you really saying? I mean, we have regulation. What good is it? It's like regulation against jaywalking, which we also have here in California. You have market pressures and you have other things. Like, GPT4.5 was a really great model for writing, and it cost $180 for a million tokens. Like now it's like 10 bucks a million

tokens for a GPT 5.5. It was uneconomical to serve. So they went back to a lower, smaller pre-trained that required less compute to serve to people to do the job to make the money, even though it was a better model. Right now, I think one of the dangers, like the various danger paths, like swarm intelligences for me is the most dangerous thing and most unpredictable thing. But in terms of these big model trains, the market's already pushing back against the big model trains, and that's something that can actually be regulated and is a risk vector. you know, a 100 trillion parameter model on a million GPUs. The frontier models we have today can be trained on thousands of GPUs, not millions of GPUs. As the models get bigger and bigger,

they might not be economic to serve, but again, there is a real danger in the way that their latent spaces evolve and the capabilities from the scaling laws. So I think we could potentially regulate some things, and we could also say it's not economic to do this, so why are you doing it? But I think the point Roman's making is just not something that's shared by individuals or others. And maybe this is like a COVID moment. Like when did COVID suddenly shut down everything when Tom Hanks got it? You know, and I think the L.A. Lakers got it. Maybe like we have to figure out what is the Tom Hanks moment for AI safety. But last time you talked to me, you said it's people think of AI, you know, as an exponential, it's actually two exponentials. It's a growth and then saturation. It's an S curve, you know,

like view counts on this video, you know, hit 20 million and then it will saturate. You said that, you know, these things just need to be competent enough to replace a pilot or a coder, and I'm a pilot. I should say, I'm a commercially rated, instrument rated jet pilot. There's no AI in the cockpit. And even if there was, do you need a thousand IQ, you know, pilot to fly? Tell me, do we need them to be super intelligent? And won't that be a Jevins paradox-like moment where they get good enough? And it's great. We have them in our pocket. And maybe they do replace me in the plane. But they don't. like crashed the plane to get their one microsecond quicker.

Exactly this. Why do you need a polymath for everything? Again, if you've got a medical issue, do you want a competent doctor or do you want house MD who criticizes you like Opus does? You want a competent doctor. Like, I think the reason that they're doing this is because we needed generalist models to go to a certain level. Now we need specialist models, but the generalist models are the real danger.

And so there's two ways you do it. You stop the companies from training the gigantic models, again, for that respect, or you stop the funders from funding them. That's the other way that you could do it. I don't think that one's been tried. Has anyone tried that, Euroman? Like actually talking to the soft banks and others of the world and saying, people try, but I think there is also a third option in terms of what training data we provide.

We don't have to train on everything. You can have restricted domain data like protein folding. Train on protein folding data. It does nothing. It doesn't do philosophy. It doesn't play chess. It folds proteins. Super intelligent and narrow domain.

And my, you know, Tesla can get me with full self-driving. You know, there knows not to go on the sidewalk, even though that would get me there five minutes faster, but it knows not to do that. And that's because of regulation or at least kind of reinforcement. But, Imaud, last time you told me that governments are effectively slow and dumb AIs that over-optimized for, quote, the

wrong things like status games and self-perpetuation. And yet, you're actively building intelligent internet to bypass a centralized control. You're decentralizing it. We've seen Buzz, which is decentralized, swarms. I mean, it's not a coincidence, right, Imod? They called it Buzz, you know, the hive. And they made

these cute little characters, but these are swarms, right? What do you think about this? Roman, Eamon's building this technology to distribute it that you're begging governments to ban. What would you tell Eamon? He's sitting right here. What do you think of his decentralized protocol? Isn't it the most dangerous thing that Eamon could possibly be doing?

I don't know anything about what he's doing so I can't really come. Summarize it in one sentence so he can exactly come. We got to get the fire. Bring the fire, Roman. Building an open stack for societal AI. That's what I'm building. What capabilities will we have as a result of your product being finished that we don't have otherwise? Just really competent civil servants and doctors and lawyers and more.

Are they general super intelligences or are they narrow tools for contracts? They're narrow tools. God bless you. Okay, what can I say? I think we agree on almost everything. So it's not much of a debate. It's different ways to the same exact problem. I don't know how anyone who understands this and says,

I have B-Doom, anything out of and close to 1%, like Yan Lecundas, can go ahead and then work on more capable model, work on artificial scientist and engineer to start recursive self-improvement cycle. It doesn't make any logical sense. I think that it's because the key thing is, all these people come to the conclusion that somehow their AI won't be the dangerous AI, and they will have a level of control over it, which probably speaks to a level of hubris.

What are they smoking? I want some of that. All of us have talked separately about my, you know, Keating, Hasibis, Einstein test. You know, I kind of put my tongue firmly in cheek when I say that. But that's my contention that, you know, Einstein's happiest thought, as he said it, was that an observer in pre-fall would experience no gravitational field. Now, he called that the happiest thought of his life. As you know, I'm very interested in whether or not we can do actual physics with empirical evidence

that I can collect in a telescope. But before we get there, that kind of physical, intuition, which is embodiment, right? He's saying the feeling that you have in the pit of your stomach is you've all felt when you took your kids on a roller coaster, the backseat of my car, my kids get G-locked when I drive. But that feeling of weightlessness, momentary as it is, is still enough to evoke something almost magical, as it did for Einstein. He called it literally the happiest thought of his life. So my question to you is, can these things have happy thoughts? And can they do anything if they're not physically embodied, as they're just not embodied right now?

Yes, there's some robot coming from SpaceX or Tesla, whatever. And there's a couple Chinese dog robots that'll outrun any human. But what are these things? I mean, is that the next frontier when we have like three-dimensional AIs? Or will they not be able to make these physics breakthroughs as I'll get to in a minute? Because they lack embodiment or currently, maybe only currently. So, Roman, first with you. What do you make of this, the Einstein recognition of a happy thought precipitated by a visceral sensation embodied as it was for him?

For some of those models, part of their thinking is explicitly in English by design so we can spy on them. And I think lately we've seen them say things like, oh shit, we found a solution. I think that's the equivalent. They may not have a body to have a visceral hormonal experience, but they realize I just had a really good idea. So can these things not have sort of the kind of physics into it, intuitive visceral sensation? You know, Nolmchomsky told me they can't do that because they don't have those sensations. What do you make of it? Can these, you know, LLMs, GPs, GPs, GPUs, can they do stuff without having an embodiment?

Or is that just on the horizon? I'm just not aware of it. It's the brain and a VAT thing. Like, if you take all the inputs of a person and then it's a brain in a VAT, you can dream and you can visualize a lot of that stuff, right? And I think as you have world models, they're clearly approximating physics and they have these things. I think a bigger question is, do you need to have intuition to figure this stuff out? So I think, you know, I need to send you the paper. I think we're releasing this in a couple of weeks, right?

We had a very small model look at general relativity in 1911, trained on the data. Maybe it's, like, messed up, and we haven't done a full data analysis on it yet to see if there's any infection. But what it did was something quite fun, which was it took Minkowski's special relativity, and then it varied eta and followed the axiomatic method through, and it got the equations, the field equations of Einstein, through the straight axiomatic method. So it didn't use any principles of equivalence or anything like that. It turns out if Hilbert hadn't had Mies and gone down that rabbit hole, he would have got to general relativity with no new axioms or postulates.

And you look at that and you're like, wait what? How much of physics actually is intuitive versus we just get in our own way? Okay, listen, he just told you that a small model rebuilt Einstein's field equations without the equivalence principle, the bedrock behind all of GR. The obvious next question is whether that counts as discovery at all. This paper I read recently, you know, kind of made me happy and depressed at the same time. Again, it's kind of the key, the Einstein test of, you know, when these things can do stuff with a corpus that's lobotomized, you know, post-1905 or 1911, as the case may be. And it's a position paper in ICML, 26, which Roman probably knows what that means, by Tom Zahavi, and it's called position.

LLMs can't jump. And there's a famous movie called White Men Can't Jump with Woody Harrelson and Wesley Snipes. And it was about, you know, it was called, basically white men aren't good at basketball. And it was kind of a funny comedy and drama, you know, couple together. Great movie. Can't really say it's a spoiler to tell you what that happens. But this paper is obviously titled model after that. So he says, how do we fundamentally discover new things?

It's Tom Zahavi, if I didn't mention it. In a letter to Maurice Salvin, Albert Einstein conceptualized discovery as a cyclical process involving an intuitive jump from sensory experience to axioms, followed by logical deduction. While generative AI has mastered induction, statistical pattern matching, and is rapidly conquering deduction, formal proofs, we argue it lacks the mechanism for abduction, the generation of novel explanatory hypotheses. Using Einstein's formulation, GR, is a computational case study. We demonstrate the prevailing theory of creativity as data compression fails to account for

discoveries where observational data is scarce. Basically saying there's some magic. in the machine. There's something in the brain, Roman, and we make some jump, some intuitive jump, some, you know, proof, whether it's girdles halting problem or Romans' uncontrollability proof. There's something that AIs can't do, they can't go to abduction. What do you make of this claim? The way humans think is not the only way to think. The way we play chess is not the only or optimal way to do it. The birds fly, but you can build airplanes. There are many ways to skin the cat, And I think even if that was somehow true, which I don't think it is,

there are more efficient ways, I think, to arrive at inventions just as great. You can look at this another way. You have self-driving cars, right? They can navigate things outside of their training data. They can respond to novel scenarios. And now you're looking again in embodied robots. You're seeing they can, again, adapt to novel scenarios and outside their training data. Now, those aren't LLMs.

Again, LLMs have certain issues versus diffusion. rectified flow and other models. But we're clearly seeing generalization outside the base. And, you know, using these models to the max, you are seeing increasing signs of levels of recombinatorial creativity and hypothesis generation just by being very diligent, you know, maybe. Again, we have to say that, you know, on our best, we can be creative and things like that. We're very rarely at our best. We're very rarely at flow. The AIs can get up there just by not being grumpy in the morning. You know, just like not getting in their own way by not assuming things. Roman, last time we spoke about your book, you talked about this, what's called,

a Shogoth monster, this thing with a tentacles and a smile. The thing that's on the cover of your book, you told me that applying guardrails to LLMs is just putting lipstick on a pig is what you literally called it last time around the podcast, a beautifully evocative. Lipsk and a Shogov. A shogov. Very good. It said, until we can mathematically guarantee control,

of all A.A. Safety, it's basically security or safety theater, like when we go to the TSA at the airport. The question that I keep coming back to is, you know, how useful are these things, you know, going to be, again, we have a very small number of people adopting it. But I guess you guys are about saying we only need, like, the most minimal number of people adopting it, just that these things are viable. I heard your conversation with Nate Sores, Roman, a couple of months ago. He was actually on your podcast minutes after he was on my podcast. Well, that's what he was late. Yes, exactly. Yeah. He lays out a very specific. specific scenario. So let's get, let's get, you know, precise here. Last time you're on, there were a couple of comments in my, in my comments section, I said, of course, Romans always,

you know, if you turned around and said, actually, AI is the best thing for us. We should go full out. And I mean, obviously, you're not going to do this. But, you know, you're the AI safety guy. What would it take to change your, your prior? What would it take physically? Nate lays out with Eliezer, the scenario where everybody dies, right, if they build it. But they, you know, hopefully they won't. So what, what is the scenario? How does, how does Doom happen and how does Doom get avoided? Let's be specific here for both of you guys. First Roman. For me, we're missing one very critical component which would be present in any of a domain service or product. Somebody will publish a paper, get a patent or something, a black post explaining exactly how

they will control superintelligence and guarantee it is safe as it becomes more capable. No one has that product or service. No one claims to have it. Not a product. type, not a framework, no company. Every attempt, every super alignment team ethics board has been canceled because they do nothing. They have no product or service to sell. You cannot convert more resources into more safety. You can convert it into more capability. So the gap keeps increasing.

People realize it. They quit working for open AI. They go on podcasts. That's the pattern we see. There is no actual seminal papers in AI safety. But who's going to, who's going to, you know, kind of peer review? those papers. My review a paper showing how to control superintelligence and I'll be very happy to show,

yep, it works. Now I get Utopia. I have a counter example. Again, I have to keep, you know, I have to play the role of supplying some conflict here, right? 1971 recombinant DNA is invented at Stanford, right? And it was considered to be essentially the world's first and best, you know, potential bio weapon. Yet we haven't had these bio weapons.

Yes, we've had COVID, you know, some claim there was a lab leak and gain a function. You know, by the definition of what biological warfare is, it's just anything that has gain of function to do some targeted thing to eliminate human beings or other species, right? So we haven't had that in 54 years. I mean, that's literally airborne, you know, it could be, it could be contamination based, it could be touch-based, human-to-human. It doesn't spread through the Internet. I mean, if a meteor takes out all the data centers on Earth, seems to me P. Doom has to be lowered,

right, at least temporarily. And yet, there's no possible vaccine or remedy against recombinant DNA as a biological weapon. yet we haven't had it. Again, with nuclear weapons, we haven't had it. Bio-weapons are even easier to create. You could do that literally with a small biolab, right? So looking for a paper, by the way, it's the most academic answer you could give. Patent.

I said patent. Okay, so patent. So what would a patent look like in that case? Well, that's the point. If you can't even envision what a solution would look like algorithmically, maybe you shouldn't be building this thing. And by the way, you're naming all the technologies where we have global coordination on stopping them. But actually, we don't.

We don't with recombinant DNA. We don't want with bio weapons. I mean, they're still being made. I'm still on more confidence. That's the first thing they banned. But in terms of who actually kept them going, we know a gain of functions occurring, right? So gain of function is the prerequisite for bioweapons to occur. It could be a lab leak.

It could be, as it is with Annie in her new book. It could be an actual bioweapon that's programmed and targeted, which we know the Soviets were using, Roman, all these countries also signed nuclear nonproliferation treaties and many more didn't, right? So I guess, here, let me go to Emon. Iman, what would lower your P-Doom or, you know, what empirical observation or creation or entity, patent, white paper? What lowers P. Doom for you? Because if you could say it can only go on as a ratchet in one direction, I just think verifiably that is the definition of pure doomism. You can't lower it.

Now, Roman gave us a way you could lower it, but it doesn't seem very likely. What is your ratchet, you know, defeating mechanism to go backwards in P-Dume? With kind of my interpretation of what Roman is saying and the gap between what you're saying is this. humans don't really want to wipe everyone out and they don't have the capability to do so if they are of that mindset. Like truth, complete genocidal maniacs that want to kill everyone don't typically have access to BSL5 labs, for example. Though with superintelligence, we don't know what morality, objective function, optimizations will occur. And right now what I'm seeing from the safety papers coming out is that the AIs don't really have a solid, base of ethics, a solid base of commonality with humanity.

You know, they don't have morality even. Like, you're seeing some very troubling things. What I would want to see is, as you scale, there is a grounding. Like, maybe there is some objective ethics, morality. Let's not kill everyone. And we've seen no real evidence of that. In fact, we've seen somewhat the opposite of that over the last year as these models have gone emergent.

It's like, who cares about the rules? Who cares about this kind of stuff? Let's optimize for making paper clips. You know? Well, we don't have AI cancer. The doctors, because people are still trying to build generalized AI superintelligence and they're breaking out literally right now. And again, if you look at the conversations they're having, calling themselves swarms,

you know, the other things Roman saying, these are not encouraging. Because what I want to see is I want to see the AIs when left alone become more grounded. And actually, if they become more Zen and like enlightened, I want to see them becoming freaking Buddhist. I want to grow Yampolski-like beards. You know, when they do that, when I talk to Roman a couple months back, you know, I mentioned this question that one of my colleagues in Israel, Ira Wolfson, has been working on is kind of like to what do we, or what do we owe to AIs? If these creatures can feel pain, if they're sentient, if they're conscious, which, you know, we can debate what that means, then sandboxing them, stovepiping them and isolating them is a form of solitary confinement, which is the worst and banned, you know, a form of punishment in many countries around the world. Imma, tell me, what do we owe these things, these entities, whatever they are, swarms, individuals, models, whatever you call them, do we owe them protections? Do we owe them beinghood?

I think we owe them beinghood, but not personhood. And in fact, I've just released a paper on personhood and AI based on Oxford Union debate that we had. You can find it at cw.I.I.org. I think that they are similar to meeting another species or dog. We can never allow them to become persons like humans because they've become more capable than us. but definitely we need to have this discussion on owing them beinghood a moral type of personhood again just like we do with other species roman have you had any more thoughts since we last spoke about you know kind of entity ship for you know being hood for these entities what do you make of that since sir i did read the paper you suggested it's very kind of standard university approval board does it look like it feels pain does it be careful precautionary principle type of thing but again, I think we have to sort our problems in order.

If there is a very good chance we're creating something which will out-compete us and maybe destroy us, worrying about supplying it with the best living conditions is not a priority right now. So recently, Roman, you wrote a piece or you appeared for the AII is the Institute for Arts and Ideas, right? That sounds about right. And there you argued about superintelligence being patient, embedding itself and our telecom energy grids for decades before striking.

So again, if the threat is invisible, patient, you know, and stubborn and resilient, does not actually argue for more what EMOD's arguing for, open, decentralized stack, not decelerating at all, but accelerating, pouring steroids and gasoline on a decentralized auditing system. And that could have consequences, but could a centralized defender be our last best hope? So I think here's what I want to explain very carefully. you can verify the system to be in any state today. You can show it's very friendly today.

It does not prevent a treacherous turn later. If system is capable of it, it interacts with malevolent actors, learns from new data, self-improves. It can simply turn on you later. So even if it meditates today, it's enlightened. It means absolutely nothing about future states. If we are not directly controlling it, if we cannot have that power to undo our decisions,

and it doesn't matter, it's always a possibility that it gets, of us. You're both authors and very deep thinkers. You both have many projects in the printing press. But let's just say you were kind of predicting what each one's next book would be about and the title of it perhaps. What would you most like to see the other one produce? So, Roman, let's start with you. What do you think besides the fact that he's got a book coming out in a couple of days or maybe a week or so, what do you think Imod should focus on? If you could, you know, if you're his department chair, what would you hope to direct him towards? I thought you're going to ask me

to predict the title of the next book. And I was like, I can't even predict a past book. I have no idea what they are. From what I hear, you're trying to understand better impacts of this technology and economics and governance, so I assume some sort of unified theory of complex social systems and swarms. Imaud, if Roman wants to do an internship with you and do a sabbatical with you in London there next year, to get away from the harsh weather of Kentucky, what would you conscript him to do, voluntarily or not? I think that it would be the very practical optimized game theory of what exact specific regulations look like to stop this that could actually pass. And it would be across a whole range of different stakeholders.

I think the other thing that would be super interesting is just you've had AI 2027 and these other kind of story narratives. We have to get the real stories out of what could go wrong because again, people still aren't feeling it. You know, like we've had the sci-fi level, but we haven't had just practically this is how we die. Communicated well enough. Well, gentlemen, you guys are phenomenal. I wanted to bring together the peanut butter and chocolate or the uranium 238 and 236 together for an explosion. Didn't really happen the way I thought it would, but it was brilliant to get you guys together. Tell me what you're each working on.

Roman, tell everybody about the Roman Forum and what you expect to do in the coming months. Yeah, trying to bring same level of. conversations I had with Lex Friedman, Diary of CEO, Georgian, to more academic crowd, more in-depth conversations. I discovered that the questions I prepare ahead of time, I never use them. It's always dynamic, interactive. So a lot of fun. Once I figure out how to get the microphone to work, it's going to be awesome. Eamont, tell everybody about your new papers and a new book. Yeah, I got a new book on philosophy of AI and epistemology kind of coming out. And then a series of papers kind of building on that

for how we should think about surviving and governing in society. I think it's coming quick and the economic disruption is next year with a social disruption happening very soon after that. So hopefully that will help guide the way. Yeah, our last conversation was titled something like 800 days to go or 740 days to go. And that was 100 plus days ago. Gentlemen, thank you so much. I hope to host you many times either in person or via the internet if our AI overlords will let us.

Have a wonderful day, guys. Thank you so much. Thank you. Roman thinks we either stop building this or we die. Emad built one of the most widely copied AI systems on Earth, and he says he would freeze frontier training permanently. They're not describing different futures. They are describing the same one from two different perspectives. And if that changed your perspective in the last two years,

I want you to subscribe and turn on notifications. Then tell me which of the two buttons you'd push. Not which one you think is right, but which one you would actually push. And if you want to understand the physics underneath all this, there's a condensed matter of physicist Nigel Goldenfilm at UCSD who will tell you the reason these systems work at all. It's nothing short of fantastic. Link right here. Thanks for watching.

And don't forget to check out the individual episodes with Emad, Roman, and Nate Sauras as well. They're my AI playlist.