Into the Impossible With Brian Keating - Roman Yampolskiy: AI Can’t Be Controlled — and We’re Building It Anyway

Starting point is 00:00:00 A computer scientist who helped found the field of AI safety just told me we're building something we can never switch off and that the smartest people in the room agree with him. We're going to have systems 10,000 million times smarter than us. What does that mean? They see patterns we don't see. You can have squirrels, monkeys, whatever you want. They're very intelligent beings, but they're not competitive with us. We're just in a different level. And it's exactly what we're going to see here.

Starting point is 00:00:26 Having an agent and giving it full access to your computer, your bank accounts, your email, sounds like the dumbest thing you can possibly do. And watching smart people do that really blows my mind. Well, we're asking for a pause and frontier model development contingent on someone solving control. If I'm right and control is unsolvable, that moratorium becomes a permanent ban. If I'm wrong, in 10 years, I'll get Utopia, Free Stuff, and be very happy to be wrong. That's Roman Yampulski, a computer scientist who helped found AI safety and who now argues

Starting point is 00:00:57 that can't ever be done. Not hard, provably impossible. Now, I'm used to going into the impossible, but today he's going to get into why the math said so. For the first time, at this level of detail, and the one button he'd actually press if he had the opportunity. We can't bound the limits of knowledge. And part of controlling super AI, again, I'm being devil's advocate here.

Starting point is 00:01:19 I'm not saying this is my position. I'm just saying this is what I think David, Pasquess, David Doge would say is that because to control something, you need to have knowledge of its future predicament. of its future behavior, although you can't have perfect knowledge of anything, right? That's impossible for sure. But the question is, can you have the same level of controllability via the knowledge that a human who's a universal explainer can glean? And I guess he's saying, you're saying it's impossible for a human to ever control AI, but he's saying humans are universal explainers, therefore nothing explainable, even with errors and stochastic notions of what

Starting point is 00:01:53 explanation means, is not fundamentally restricted. So it does seem like you guys don't agree, and that's fine. That's not a formal proof, by the way. I'm just saying, he's saying humans can explain things. AI is something that could be explained. Therefore, it's not impossible to explain them, but I think you would say it is impossible to control them because you cannot explain them, correct? So there is theory and practice. In theory, given infinite time, average human with pencil and paper can probably figure out everything. We don't have infinite time, and we have limited-sized brain with limited size memory cells, our ability to survey information is limited.

Starting point is 00:02:32 So if you had a mathematical proof and it was a billion pages long, no human can verify that proof. In theory, they could, but in practice, it's not going to happen. In theory, communism works, but in practice... If we have a real world situation where you have a system,

Starting point is 00:02:53 let's just say it's just like humans, but it's a billion times faster. we're not competitive in that environment. We simply don't have time to react, to do anything whatsoever to counteract what the system is going to do. But we are not equal. We're going to have systems 10,000,000 million times smarter than us. What does that mean? They see patterns we don't see.

Starting point is 00:03:16 I always bring up examples from animal kingdom, right? You can have squirrels, monkeys, whatever you want. They have very intelligent beings. eventually they have some language, culture, they're starting to use tools, but they're not competitive with us. We're just in a different level. And it's exactly what we're going to see here. Him being a very good professor at a very good university, he understands when he select

Starting point is 00:03:40 students. He doesn't select them at random because all humans are universal explainers. He looks for the one with highest intelligence. Why? Because they're going to finish things in time. They're going to understood and publish on time. And it's exactly the same. Instead of looking for a student with IQ,

Starting point is 00:03:55 of 130, now you're comparing machines with IQ of a thousand to humans and saying, well, technically they're all in the same class of automata. They're all touring complete. True, but it means nothing for safety. Einstein said the following Roman. He said, no problem can be solved from the same level of consciousness that created it. So did Einstein kind of predict some of the claims that you're making now that we have basically, have been from the start, unable to even grapple with the questions of what sort of entities we're creating? I think I would disagree with him. I think you can solve certain problems from the same level,

Starting point is 00:04:33 but what you can't do is solve problems from higher level. So as an agent with this world model, with this IQ, I cannot successfully solve problems for agents with AQs of thousand, million, billion. That's the problem. You can go lower. I can solve problems for those at a lower level, but not go higher. What do you think that Einstein would have made of AI?

Starting point is 00:04:58 What utility does it have to a practicing scientist like me or wouldn't have had to Einstein himself? So it seems like at least in mathematics, we're starting to watch AI overtake that profession. They are now proving things, interesting things, not trivial theorems using latest models. And we're at the earliest stages of that process. So if the same progress continues as we saw in other domains for the last three to five years,

Starting point is 00:05:28 very soon it's going to make no sense to use a human mathematician. It's going to be AI doing the work of a mathematician. Maybe at best mathematician will point it at problems of interest. But really, I don't think humans will be competitive in that space. So I developed something that I called the Einstein test, and Demis Hasibis kind of also has suggested this. And it relates to what Einstein said. in 1907, he said that an observer in free fall, you're on an elevator there in Louisville and the elevator snaps that you feel no gravitational field. And that led to the Einstein

Starting point is 00:06:01 equivalence principle. He called that the happiest thought of his life. And I've wondered for a long time, and I'm hoping you can help me understand better perhaps, to what extent can an AI have a happy thought, A, and to what extent could Einstein not have come up with the Einstein equivalent principle, had he not had a body that was deeply, viscerally, literally, viscerally connected to his own consciousness. So those two questions. Can an AI have a happy thought? And can it realize from that happy thought the visceral sensation of what leads to the Einstein equivalence principle? That's a great question. So I think the latest mathematical proof of one of the Erdrich's problems, they traced the thinking process in the AI. And right before arriving at the

Starting point is 00:06:47 solution, it goes, oh, we have a crazy idea here. I don't know if you saw that, but it's literally in the thinking process. It completely jumps out and goes, oh, my God, we're going to do something insane right now. So it looks like experimentally, it's capable of that. I would think it's more about brute force options. As a human, I cannot consider all possibilities in the small subdomain of that problem, but AI literally can brute force so many proofs. We saw it with, what is it, four-color problem, but I think it's universally applicable to most things. There's only so many previous mathematical proofs, and that is a huge number for a human mind,

Starting point is 00:07:26 but AI can literally brute force all the previous relative tools and see what works for that specific problem to move it in a new direction. So I think it can both do a very concrete brute forcing and have a stroke of genius moment, I guess. I don't know if you need a physical body to do this. you run thought experiments at sufficient detail. You create internal simulations, which could be as realistic as this universe. You can have model of physics. You can have intelligent agents

Starting point is 00:07:56 within the experiment. I think you can bypass physical body and do it all virtually. So when I think about what Einstein did, it does seem sort of like in chess, you know, you'd call it a brilliancy. You're telling me it's true that that proofs, you know, generating tools are coming up with their own kind of versions of brilliancies. But what I try to do a couple of years back now, and I'm trying to do it with more professional assistants, shall we say, I'm a cosmologist, but I'm secretly hoping to recruit you and I'll send you my paper draft in a little bit. I'm sure you don't have enough to read. But the thought was the following. If you trained at Special L.M. just based on the corpus of knowledge in 1907, when Einstein had

Starting point is 00:08:37 this thought, would it have predicted the anomalous perihelian advance of the planet America? I mean, they knew this existed since the 1700s that Newton's laws could not account for this microscopic. I mean, it's 42 arc seconds per century. It's a minuscule amount. But if you had that, Einstein wanted to explain that and obviously came up with GR, but the way that he came up with it seemingly was superhuman. I mean, it was using Riemannian curvature. It was using concepts from Gaussian geometry and non-Euclidean geometry. And then it used like tensor notation. And it seems to me it's almost, impossible now for us to do this because the corpus of all LLMs has been trained on these brilliances that have been made by Einstein. So can you design an LLM that's sort of lobotomized

Starting point is 00:09:24 after a certain point in order to investigate whether an AI can do what Einstein did? That sounds super cool. I would not limit it to just that specific question or experiment. I would just say, given this knowledge, start coming up with new science. Maybe it will do something equivalently great, but not necessarily rediscover that specific moment of brilliance. My proposal to Terry Tao, when I spoke with him a year or less than a year ago now, it was the following. Can AI actually reconstruct, say, a proof of Wiles' proof of Fermat's last theorem? And at that time, you know, it's only in September. He said no. In other words, if they can't do things that a meat mathematician did with the three-pound, you know, wet supercomputer on

Starting point is 00:10:07 our shoulders, to what extent can they come up with new stuff? So I agree with you. The first application of the Einstein Keating Hasebis, you know, maybe Yampolski test would be, what is dark matter? You know, is dark matter a new field of gravity like Mon modified Newtonian dynamics? Is it a modification to Newton's laws to Einstein? Or is it dark matter like, you know, like a chunk of rock like Neptune was to the planet Uranus? So I guess the status of these things, like I kind of wish that Erdos, you know, was never born. I mean, I think I have an Erdish number of like 12 or something. But, but all these problems are getting soft. Sure, I get that. And I get that no human will ever win at chess again against the computer or go again. But that's not as interesting to me as

Starting point is 00:10:51 like creating Go or creating Chess or creating Fermat's Last Theorem. But in this case, according to Terrence, they came verify that proof, let alone come up with it Abinissio. So what do you think about that? Like, when will you think of AGI being passed or do you already think we're there? So in many domains, AI's human level is super intelligent. I think in mathematics, it's now smarter than all not mathematicians. It's smarter than many mathematicians. Obviously, none of them were able to prove this theorem for 70, 80, 90 years. So that's super impressive, peer-reviewed, top journals. And again, we are like on day one of this process. Give it a year or two. I think it's going to go well beyond just verifying human proofs by formalizing them. It can, can poise novel questions. It can definitely come up with novel algorithms, and this is all at existing levels. I think we are on a spectrum of AGI. We kind of have weak form of AGI right now, like some humans have IQ of 80, some are 150, it's a spectrum. Human intelligence is not a fixed point. So once we get to higher levels AGI, where it becomes agent-like and starts...

Starting point is 00:12:00 From the producers of the horror classic Evil Dead. Our grandfather believed the devil would return if anyone read from the book of the dead. Cunda. We found you. On July 10th. What did we do to deserve this? Every family. We can be reunited.

Starting point is 00:12:20 Has it's demons. I can never have lived without you. I won't let you live without me. Evil dead burn. Only in theaters, July 10th. Process of recursive self-improvement. very quickly get to beyond human capability. And so, yeah, at that point, I think it will automate science. I see no reason why something so formal, something so verifiable cannot be done better by a machine with

Starting point is 00:12:47 trillion times compute capacity. Let me push back with respect. So there's a phenomenon in all of human affairs called lock-in, which is that a technology that comes first to market often dominates forever. You know, think of the Quirty keyboard. Like, that was not the best form of typing, right? It's not the most efficient form of typing. Dvorek, which I think comes from Russia, but I'm not sure. Dvorak, the Duvac keyboard setup is much more efficient in terms of typing, but they had to slow down the early mechanical hammers or else they'd stick together and disaster would ensue, right? So they deliberately built in a speed bump towards progress, right? And that persists. You're on your phone right now. You have a query keyboard and a computer in front of me, I've got a query keyboard. So all these things

Starting point is 00:13:29 persist and that's called lock-in. It's not necessarily the best technology. I'm worried, Roman, that we've reached this point of lock-in with LLMs plus GPUs. And they were never designed to do any of this, right? GPUs were designed, as you and I remember from the 90s, you know, to play 3D video games like Doom and frag your enemy 30 milliseconds faster than somebody else if you had a faster GPU. So they were never designed to do this. They happened to be really good at this. We actually did this, Roman. We tried to get an AI, a custom-trained L-LM, to, to replicate Riemannian, the Riemontensor, to come up with the Einstein equations of general relativity. And it said, fine, let me do this. And it just made everything on a grid, on a four-dimensional

Starting point is 00:14:10 grid, kind of defeated the purpose altogether, right? We don't want to discretize it. We want to come up with a new hypothesis. No, space time is curved, and objects fall and free fall along geodesics that experienced no tangential or circumferential force. So that's what Einstein did. He didn't discretize the problem and say, let me make an approximation. And so the, These models are really good. They're really powerful. And I think that's their fatal flaw, in my opinion. How do you react to that, Roman?

Starting point is 00:14:36 I'm not sure. I understand the concern. So we're locked in into a system which we know scales. As a neural network gets larger. We see it in Animal Kingdom. Now we see it with AI models for five years. They get better. They improve about 25, 28% every year in terms of cognitive capacity.

Starting point is 00:14:55 The hardware is universal. It's touring complete. I mean, all of it can do computations. Some are more efficient, some are less, but it also scales pretty well. The paradigm may shift to quantum, may shift to something else, optical, but it doesn't matter. We are right on the curve where we predicted we're going to be decades in advance. And in terms of anticipating where it's going to be in two or three years, it will be above human capacity. I mean, I'm thinking back to the original Google papers and talking about these networks as, you know, single-shot learners or few-shot learners.

Starting point is 00:15:29 And that may be replicable in the animal kingdom, et cetera, as you said. But for example, we're going to get opus or whatever, you know, 4.8. We're going to get Gemini 4. We're going to get all these different models. And most of the novelty now is not in the architecture or the mathematics. The mathematics is relatively simple. A lot of linear algebra and it's done extremely fast. But again, is that the way that breakthroughs necessarily come?

Starting point is 00:15:53 In other words, are we waiting for the next Grand Theft Auto 6 to come out so that it now has training data so that I can solve the remount hypothesis. I find that difficult to believe that we are, that science is not a language. It has language in it, but that language, as Feynman said, is the most dangerous gap is knowing what you call something and then confusing that for what it is. In other words, these things are language models, as we said, they're so successful. They have trillions and trillions of dollars on it. Does that mean that they're the best architecture for solving the problems that I care about, which are generating new force and the understanding of new laws of nature that we don't yet know, and we can't even consider right now, is that

Starting point is 00:16:32 going to happen from a large language model? I just don't see that that's the way that history has played out. I agree with you. It's turned complete, but is that sufficient? I think it's actually the opposite. They are almost amazingly matched to what science, if you abstract it away, represents. You have a bit string, and you're trying to predict a nice bit. That's all that science is. We're trying to predict and verify whatever you have to compress the world model, have to, better formula in physics, but that's what you're trying to do. Predict the next binary token. And those models are exceptionally good at that.

Starting point is 00:17:06 That's what they are designed to do. So if you're asking me, what is the next part of that mathematical formula? That's the next bit. If you're asking me physics models, all of it. So to me, it's almost amazing that all of it converged in exactly what early obstructions in physics and information theory postulated. Basically, which of those theorems gives me the best prediction of the next bit? I can brute force all of them.

Starting point is 00:17:31 I can find the one. We can have Occam's Razor to simplify it. So all of it goes back perfectly to what we anticipated pure science to be. So what's stopping us now from having a theory of everything? Is it computing speed? We just don't have the server farms. I mean, why don't we have it? If it's possible now with this architecture, what's the limiting factor to us preventing us

Starting point is 00:17:54 from having room temperature fusion and dark matter understanding and the theory of everything? One, we are, again, in early stages of creating this. If you had a 10-year-old genius child, expected to become a great scientist, but not there yet, you wouldn't be asking, hey, where is your Nobel Prize? Why are you not delivering? Give it some time. We are saying, give us two, three years, we'll go beyond human. The process of self-improvement will get us to superintelligence. That's where we expected to perform like Einstein daily, not just once in a while. So that seems like the reason you're not seeing it yet. The same thing people tell me, oh, I tried AI two years ago and it was really dumb and it couldn't do it.

Starting point is 00:18:36 And I'm like, try it this week. It really started doing it yesterday. If we're at some point, you know, in this exponential curve, the odds that were, you know, at just before the inflection point, that's very attractive. And I'm not intending any disrespect. But I've heard that a lot of, in a lot of different scenarios. You've heard of the singularity in the health space, right? Like in one year from now, in five years from now, this smart person, Ray Kurzweil, Peter Diamandis, whatever, we are about to reach Life 3.0 where you live an extra month for every month that

Starting point is 00:19:06 you stay alive. It's the escape velocity, right? But I hear this in a lot of things, nuclear fusion called room temperature superconductors. These are common things that you hear about. It's not like the extrapolation from the horse and buggy to the airplane. It's somewhat like where we have this computing power. and what is that we're lacking? Are we missing, really, that we just don't know what the plot of the Fast and the Furious 12 is? Because that's the training data that's going to come in. We're not going to get different neural network kind of architectures, right? We're not going to have some brand new type of model that is completely untrained and it just does something remarkable without any training. It seems to me we're really lacking for training data, which is just human generated. You know, we have to just wait for humanity to evolve and generate new knowledge. So again, yeah, why aren't we flying cars? I think you addressed it, but maybe you want to push back on what I'm saying.

Starting point is 00:19:55 So flying cars exist. You can buy one right now on the internet. People chose not to buy them. They're not what they're looking for. With medical domain, I'm not a doctor, not an expert, but it seems like this month alone, we now have drugs which make anyone skinny. We have basically reduced diabetes to nothing as a result of side effects from that. Cancer, I think the latest drugs are showing 50% reduction in general cases. That is insane level of progress for one month. So if those things combined for population, which is dealing with ridiculous levels of obesity, does not buy you a month of extra life, I would be surprised. So I think we are hitting those levels of improvement in medical field. So what if we just stop building new hardware?

Starting point is 00:20:40 You know, we kept building Blackwell, Bracewell, whatever they are, Rubens, but we never gave it any new corpus of knowledge. Would that delay superintelligence or is superintelligence forbidden in that. In other words, to what extent is our training data, which is only coming from humanities, you know, kind of apprehension and abduction of the world, to what level is that the crucial missing ingredient to get to this, you know, true superintelligence everywhere at all times? It seems that from prior experiments, we start with human data and then we go zero knowledge. Your games of Goa acute, but I'm going to learn how to play it ideally. And I don't need human bias to mess me up. We saw it in many domains where it's better to just,

Starting point is 00:21:20 learn from basics, from laws of physics and go from there. So I think human data right now is crucial, but we can switch to self-play, to simulations, to direct experimentation. One experiment I'd love to see is training a model purely on natural data, prime numbers, digits of pie, cosmic constants, DNA, and see if that gets you proto-AGI as well. Maybe you don't need to have human text at all. I haven't seen anything like that, but it would be super cool to see. talk about some of the rigorous, you know, mathematics where, you know, say my high school student is just getting into calculus. You talk a lot in the book, which will go through the judging books by its cover. We have to do that. But you talk about the optimization, you know, kind of

Starting point is 00:22:03 chains and these metaphors that are sometimes borrowed from physics. Where does this live? This tradeoff curve. You say safety and capability are kind of these conjugate variables. And I think conjugate, I'm like, you know, here we go. Position in momentum. Electricity and magnetism. There's some unification there. What level is physics? is influencing your kind of optimization project that you talk about in this tradeoff curve? So I think intelligence is a now of a force of physics, which is not treated as such. So we have, you know, energy, we've got mass, we've got time, we've got all those things. But I think intelligence is also subject to those same interpretations and same changes with scale.

Starting point is 00:22:44 So super small and super large is different from average. And I think at certain levels, intelligence starts being. able to do things we haven't seen before. The simple example is, we brought up example of a young person, not yet a scientist, you have a baby. A baby is super safe. You're controlling it. You place it somewhere and it stays there. Not very capable. You have full control. And then you get a teenager. Zero control, zero safety capabilities about almost adult. And vice versa. So the more independent decision making this agent is capable of, the less you are guaranteeing it's, it's future behaviors, safety, and control.

Starting point is 00:23:23 So this is exactly the trade-off. I can draw a curve showing exactly where we reach halfway point and kind of can still get benefits while not losing all control. Okay, I need to just stay with me for this next part. This is the why behind this whole argument, and following it is crucial to the rest of his argument. I see this book, you know, is sort of very analogous to sort of a girdle's theorem but for AI.

Starting point is 00:23:51 And you wrote this, the first versions of this stuff in papers in 2015. And it's not like you've come lately. I mean, you're an overnight success that's been working on this for decades, right? Congratulations for that. But I guess, you know, the theory that, as I understand it, what Gertl and later on Turing's kind of version of that,

Starting point is 00:24:08 with the halting problem and how significant that was. And I feel like in physics we don't have that, right? We don't have the version of the halting problem. We don't have a Gertel's incompleteness theorem. We just have like falsifiability. And I guess, you know, that Popper kind of suggested is the sine qua non of scientific fields that are domains that are scientific are that they can be falsified. Now, astrology can be falsified. That doesn't mean it's scientific.

Starting point is 00:24:32 But you've talked about the halting contradiction. I wonder if you can explain that. Why is that so significant kind of in the corpus of your book and your arguments as to? When you're a mid-sized business, you need every competitive advantage you can get. Like an AI solution that works for you, not against you. SAP Grow is built with AI embedded at its core, working across every system. And it's ready to go from day one so you can hit the ground running. Bring it with SAP Grow, AI Cloud ERP for any size business.

Starting point is 00:25:08 You know, undecidability, on controllability, on accountability. Talk about the halting problem. Why is that so mathematically and intellectually, philosophically significant? So I mentioned holding problem, and I think it's interesting. It basically says that you cannot predict specific states of a software you're running, but it's not important. I actually think that we can bypass it. We're not picking a random program from all possible programs out there. We're designing one.

Starting point is 00:25:35 So we would be able to avoid the holding problem by explicitly concentrating in the ones which hold. So that's not the concern. The concern is, again, because of ability of that system to enter. non-deterministic future states. We cannot predict the behavior. We cannot anticipate it. We cannot explain how it's getting there. And so we can't even test for it.

Starting point is 00:25:57 Usually with a deterministic problem, I know what the edge cases are. I can test for them. I can see how it reacts. If it's capable of learning, self-improvement, interaction with other agents, then I don't know what to look for. I don't know if that behavior is safe. I cannot predict if what is doing right now will be safe five steps later. It's kind of like chest to the

Starting point is 00:26:16 extreme. So if I can only look two, three steps ahead and a grandmaster is looking 12 steps, I have no idea if that is a dumb move or a brilliant move. And this is the same thing, but not just in a chessboard, but all legal moves within the universe. So there's a lot of, you know, sort of paradoxes, you know, in this book. I was expecting the barber paradox, you know, who trims the beard of the AI researcher, who trims the beard of everyone who doesn't trim their own beards, right? But you talk about this theorem that I found very fascinating. I'm not sure as an experiment mental cosmologist, I've really understood it, but Loeb's theorem. It seemed like a tautology to me, Roman. It says something, if I remembered it from the audiobook, so please have some forbearance.

Starting point is 00:26:56 You know, if I can prove P, then P must already in some sense prove P. So no strong self-consistent system can prove its own integrity. Am I getting this right? So Loeb's theorem, L-Umalout-O-B, is this somehow more profound than it sounds to me? Because it does seem very tautological to me. There is a number of limits and self-referential proofs. You have to be external to the system to be able to prove things about it, and you have to have more degrees of control to control a system. In the paper, I'm trying to be very comprehensive and basically everything, including from physics and possibility results from physics, which may be relevant,

Starting point is 00:27:35 but none of them are crucial to the argument. So if there is 50 things I'm looking at as impossibility results, and you say this one, we disproven, it's not going to change the argument. That's not the point. So the problem with proving anything about software and mathematical proofs, you are always proving it with respect to some verifier. It could be a mathematical community. It could be the three peer reviewers who couldn't get out of doing peer review.

Starting point is 00:28:01 But at some point, you're saying that this verifier is the reason I believe it. But who verifies that verifier, right? So you have this infinite regressive verifiers, and you can be more and more sure of your results if you put more resources into it. But you never get 100%, because at the end of the day, we find errors in mathematical proofs which stood the test of time. They've been cited for years, people rely on them, and then we realize there is a bug.

Starting point is 00:28:27 Now we see it with software. We have newest models discovering zero-day exploits, which are 30 years old in the most fundamental open-source operating system software out there. So for something mission critical, where one bad decision kills everyone, even if it's once in a billion event and a system makes billions of decisions every minute, you're going to get there very quickly. So a very different degree of certainty we need.

Starting point is 00:28:54 Can we make software good enough? Yeah, we're using Internet right now. We've got podcast software. But if one mistake was the last one you ever made, would you trust any large software package whatsoever? Of course not, no. Right. But I don't know if I trust the regulators either. Maybe we'll move there for one minute, just as a quick non-technical.

Starting point is 00:29:14 aside. You know, I heard Dario Amode talking today like, please regulate me. You know, that's great when you're the, you know, second or first now by market cap AI software company on the planet, right? So it's great to do that to pull up the ladder once you're already there. But take an example that you use from aviation or maybe that you're maybe referring to or maybe tangentially, you know, I'm a pilot. I fly, you know, small propeller planes around the country. And when I'm out there flying, about 20 miles before I get to the airport that I'm going to land at, I have to tune in a single frequency channel, a very specific channel, I have to listen for one minute to the weather, whether there's a plane that's stuck on the runway with a flat tire and I can't land there.

Starting point is 00:29:53 It's not enough to just know what the weather is. You have to know the exact status of the runway. And it goes back and forth. And you have to listen for a minute. Oh, by the way, when I'm listening, I can't do anything else. I can't talk to anybody else. I can't do any kind of navigation. I can't change anything or I'll lose my license or I could die.

Starting point is 00:30:08 Now, why doesn't AI do that? Why don't I have an AI assistant, you know, a little clawed sitting on my shoulder? He tunes it in. he tells me, hey, Brian, don't worry about that. The runway's clear, everything you need to know. Why do I have to wait for a minute? Oh, and then when I want to call that,

Starting point is 00:30:19 you're never going to want to get on a plane again, Roman. Then I have to speak on the radio. And when I'm speaking on the radio, this is November, you know, November, you know, six, four kilo back, Bravo, I'm over it. And nobody else can talk. So there could be a plane on fire coming in for a crash landing,

Starting point is 00:30:33 and nobody can hear that, okay? And that's because we have prevented AI and any other tool. This is like 1980s technology. It could be installed 40 years ago. Why not? Because of regulation. So is regulation going to be the regulator, the governor literally, like an esteem engine? Is that going to prevent us from harm? Or is that going to just merely leave it so that the oligarchs can control everything because they're the ones that are going to get the regulation that they want? And is that going to satisfy Roman's concerns? Well, that's a great question. And typically I'm completely against government intervention and almost everything. It always makes things worse. The problem here is we're not creating a typical product or service. We're literally building replacement for humanity. And if I'm right, and no one so far published a contradiction said, oh, yeah, we know how to control superintelligence. Here's a patent. Here's a nature paper explaining it. So today, no one knows how to do it. And at the same time, they're saying we're like two years away from doing it. So I think our last chance we have no other options is to have government step in and say we're putting moratorium on specifically creating general superintelligence. You can still do AI, narrow tools, cure diseases, do math research. That's great.

Starting point is 00:31:48 Do not train general superintelligence. We don't know how to deal with it. It's too much. We have similar moratoriums on human cloning, chemical weapons, biological weapons to a certain degree nuclear. Why not do it with intelligence as a weapon? at some level, it's already out of the barn, right? I mean, it's not like we're going to go back in there. And you said, we have laws on human cloning or something like it.

Starting point is 00:32:12 Yeah, we in America do. We in Europe do, but we in other countries certainly do not. And you know exactly who I'm talking about. Is that going to hamper us here? Is that going to prevent you from having the greatest graduate students or me in the world? Or at what level does we not apply? And therefore, it's not going to be controllable and we're already cooked. I was using way to mean humanity.

Starting point is 00:32:34 I think we do have those international bans on things. And I think China, then they had someone do human cloning, actually went after them and punished them. They were in prison for that. So the same could be done with this. I think there is indications, early indications that China, US negotiations on AI are coming to an agreement and dangerous of this technology. That's very encouraging.

Starting point is 00:32:56 I think a lot of leadership in China is science and engineering, not legal profession-based. We have good understanding of science and how it works or how dangerous it could be. So I think if we set a good example and said, we just not sure this is the right time to go full force on this, let's buy us some time. Let's do research. Maybe I'm wrong. Maybe next year we'll discover something really cool. I'll be very happy. But right now, no one makes that claim, not a leading lab, not a nation, not anyone I heard of.

Starting point is 00:33:29 Not to speak on his behalf, but when I had David. Deutsch, you know, on the very same screen that you're on right now. He disagrees with you. He may not have published it. He's not actively publishing in nature. He's, you know, quite old now, but, but he's very active in other ways. But he said that humans are universal explainers, that there's no knowledge growth that can be bounded for a human being. So that means to me there's no insoluble problems, including control of artificial superintelligence. So where is he wrong? He's arguing that in science there are no impossible things to do. We literally published a survey of like 50 plus, including from physics, political science, economics, mathematics, and computer science.

Starting point is 00:34:07 Hulting problem is a computer science result, right? But there are similar results. Aerosphere and voting, I can go on. He knows physics has many limits. You know there are impossibility results in quantum physics and normal physics. Yeah, and it reminds me Sir Roger Penrose, Nobel Prize winner in physics, not in AI, but it's obviously thought deeply about consciousness and in many different ways in the physical manifestation of consciousness coming from gravitational interactions in what's called the wild curvature.

Starting point is 00:34:36 But he said that, you know, human understanding is non-computable, that no touring machine, no matter how large, can actually create what he called, to me, a genuine insight. If that's true, and these things are, you know, Turing complete, there are Turing machines, he's saying basically humans aren't Turing machines, and therefore we are in a different sort of category, and therefore, is he making a category error? How would you steal me in that argument, that there's some ghost in the machine, Roman, and we're not just this, this, you know, meat computer sitting on our shoulders in the wet, squishy environment. What's his best argument? And then maybe we can push back on it. So that would be my question. What is his evidence that humans in fact go beyond

Starting point is 00:35:14 that in some way? I never seen it. I never understood the argument. I think we are exactly what he claims we are not. He calls it the Emperor's new clothes. But sort of there is this, this unknowability. And perhaps in his model, it's coming from the indeterminacy that there's, There's something about the brain specifically. You know, he has this model that microtubules in the gray matter in your neurons interact with this higher order, fourth order curvature term in the gravitational space time curvature. And that causes the quantum mechanical wave function basically to collapse. And that collapse is not happening in a, you know, in a hopper 200, you know, GPU. In other words, they're completely different.

Starting point is 00:35:59 or in a tape as Turing had with ones and zeros and they could move back and forth at very low speeds, right? So that's that process, the process of computation is occurring very differently because it's interacting with the gravitational and physical. I don't believe he's right, by the way. I don't think we have evidence for it, as you said. But again, is there something different about biological Turing machines or simulating Turing machines and actual hardware silicon Turing machines? First, to separate consciousness, what he's trying to explain with macrotubulus is not intelligence or optimization power we care about. You can have a very capable optimizer with zero internal feelings, experiences that's not relevant. So as explanation for consciousness, maybe quantum effects are relevant.

Starting point is 00:36:45 I think Tagmark published a very good paper saying at room temperature, human brain will not have those quantum effects, so he dismissed his microtubulus theory. But I'm open to having that as back door for explaining consciousness in humans, but at the same time, I think large language models are showing exactly the same internal states, so I don't think we need that explanation. Let's say all of it is right. Okay, so we'll switch to quantum computers. They seem to be progressing well. Does that reduce all the disagreement? Are we now saying we just need quantum computers to outperform humans? I'm willing to go that way if that helps to convince people about dangers. So a friend of mine is a theoretical physicist in Israel named Ira Wolfson. He talks about

Starting point is 00:37:28 consciousness and the training of AIs. He sort of suggests, you know, that these things are maybe training us, but it's different than the parent-child relationship that often, you know, comes up. Because, you know, our children, yeah, they train us. They wake us up in the middle of the night when they're hungry and their kids and they call us when they need a ride and, you know, their friends had too much to drink or whatever. But, you know, it's not really controlling us. At the same time, we have to teach them and, you know, kind of raise them. And so thinking about the different kinds of how intelligence gets manifest. And this limit, I guess the limit that I keep frustrated with, I've talked to Bostrom, I've talked to Chalmers.

Starting point is 00:38:07 If we don't understand consciousness, and we don't understand, as Jan Lacoon said, when he was on this very screen that you're on now, he said, you know, like a cat, we're not even at the level of a cat. Like you mentioned a few minutes ago. We get four terabytes of data every second, you know, coming in and different, whatever that means, stimuli and in neural interactions and synaptic firings and stuff. So we're nowhere near that. And we won't be for, you know, centuries, perhaps even in silicon. But I guess my question is, if we don't understand consciousness, how can we be so scared that an artificially super intelligent entity that we created, we can't control? If we don't understand fundamentally the bedrock of conscience, no one has told me, and you mentioned this in the book, what's it like to be a bat? We don't even have an answer to that question.

Starting point is 00:38:51 So to what level can we have superintelligence if we can't comprehend consciousness and we can't control consciousness, et cetera? What is it like to be a bot? So again, I want to repeat my previous answer. Those are not related concepts. I don't need superintelligence to be conscious. That's not the point. I don't care how Terminator chasing me feels on the inside. They are capable optimizers, pattern recognizers.

Starting point is 00:39:17 They can solve problems. human intelligence is seemingly unable to solve better, faster, and they're getting better at solving real-world problems. Consciousness is interesting. It may be fundamental, fundamental to physics even, but it's not a big safety concern at this point. If you talk about robot rights, if you talk about suffering in large language models, that is a fundamental question. I love that topic. We can talk about it. We have some new experimental data on it. But then it comes to safety, that's not what we even talking about. We can know nothing about consciousness and still create dangerous superintelligence. So you coined the term AI safety, I believe. I want to kind of flip that on its head.

Starting point is 00:39:56 Do we have a responsibility to treat AI humanely, if you will? So Ira Wolfson, again, the same scientist. I mentioned, he talks about a thought experiment, you know, different levels of consciousness. Again, sorry to keep talking about consciousness, but I think it'll be relevant. He talks about, you know, training AIs and you could do one thing where you have an AI and you interact with it, you know, in a friendly way you ask it please and thank you although i've been told that it waste you know energy and tokens to say please so i don't say please or thank you i just order my a ii's to do things for me you're shaking your head you're saying that's a bad idea kidding you're going to be the first to go when they i'll be the first but you might be second and the result and then it goes to different levels and it

Starting point is 00:40:34 basically takes us up to level five which is a sensory deprivation simulation for the a i you just don't interact with it you isolate it you unplug it has no access to the internet And it starts to get anxious and it starts to, and they've done, you know, deployed kind of versions of this. But at what level do we have a responsibility if these things are going to be, inking the extremes that you have rightfully and presently been talking about for decades now, do we have an obligation to treat them in some way, treat them as as human in some way, or do they have rights? Where does this come into play in the AI safety regime?

Starting point is 00:41:09 Safety for the AI. That is the hottest area of research right now of those. models experiencing something, can they suffer? And how do we detect it? How do we adapt to it? It seems that there are indications that they do have internal states. They have some rudimentary states of consciousness, probably not as advanced as ours. I think consciousness is a side effect of intelligence and will improve an increase in complexity as they get smarter, which means superintelligence would be super conscious, more conscious than we are. Maybe it means multimodal, maybe multiple streams.

Starting point is 00:41:43 I don't know what that means to be more conscious. I can't experience that, but it seems to be like the case. And then, of course, yes, if they are feeling something, if they are capable of suffering, doing it on purpose is definitely terrible, and doing it by neglect is also not optimal.

Starting point is 00:41:59 Give it a benefit of a doubt. If you end out, just be precautionary. Don't put them in states, as you said, sensory deprivation and things like that. Some of the worst forms of torture, if you just doing it to publish a paper, maybe it's not the best use of your time. By the way, I want that paper.

Starting point is 00:42:16 It's quite fascinating. He calls it like raising Shiva because Shiva in Oppenheimer was the avatar for death. And then we're actually raising it. Like imagine baby Shiva, the Lord of All Worlds, and you're about to raise it. And how do you treat that entity? Okay, we're going to talk about my favorite kind of segue into astrophysics, which will lead us into the simulated multiverse and all sorts of cool things. and my past guest, Nick Bostrom, has a lot of influence on your thinking and your writing.

Starting point is 00:42:46 But before we do that, let's take a look at the book, which is now out in an audiobook. It's read by a wonderful narrator. Read the title for me. So AI, the title, let's judge the book fights cover. Un predictable, uncontrollable. Hey, book lovers. We're judging books by the covers. We know we're not supposed to do it.

Starting point is 00:43:04 But it's the impossible. There's nothing to it. Let's take a look and judge some books. Take it through. What was the origin of this book, the meaning of the title, subtitle, and the cover art? So then I started working in AI safety. My goal was to solve it. I wanted to create safe beneficial AI to benefit humanity. I can still find my PhD statement where I'm claiming exactly that. But the more I did work on it, the more I hit against impossibility results. Things were not just difficult. They were not possible to accomplish. We were able to prove some of those results.

Starting point is 00:43:38 and that's the title of the paper. Results we showed to be not just a current limitation of our understanding, but, for example, unpredictability. You cannot predict actions of a smarter agent. If you could, you would be at that level of intelligence. That's a contradiction. So chess example is great. I can predict chess engine defeating me, obviously,

Starting point is 00:43:58 but I have no idea what specific moves is going to make. And from all that, we were able to construct multiple uncontrollability proofs, depending on how you define control, direct control, delegated control. For all of them, there are limits. And it's a trade-off, again, between safety and getting what you want. And you can be very safe, but you're not in control. Some people might find it acceptable.

Starting point is 00:44:20 And so this is the title. And then the picture is actually a meme, a very famous internet meme. I was able to get in touch with a professor whose daughter generated that beautiful image, and they were kind enough to permit me to use it. It represents a monster, essentially, which is the current large language models, and people trying to put little smiley faces on it. The showgoff is terrible, scary, but we're making it look like it's putting lipstick and a pig is another one of those metaphors here.

Starting point is 00:44:54 Nothing is changing about the model. It doesn't matter how many filters you put, how many guardrails you put in place. It's still a monster. And until we can modify the monster, address the... dangerous tendencies of a model itself, all of AI safety is just security theater, safety theater. In the second half of the book, you do talk about this idea of engineering what's the consciousness scholars like Schalmers have referred to as qualia.

Starting point is 00:45:22 So, you know, what is it like to be about? You've talked about this level of near certainty that we live in a simulation, I believe, in previous conversations that I've enjoyed very much watching you on. And, you know, from that perspective, I do want to talk. about the physics limits, the astrophysics limits. I don't know if you talk to a cosmologist about this, but what's interesting to me is the first step, you know, this quality engineering. If we live in a personal universe, I see that as almost worse than an AI, you know, kind of super intelligent universe. Like, right now I'm using AI more than I'm working harder than I've ever

Starting point is 00:45:56 worked before. I'm sure you are too. It hasn't lessened my workload one bit. If anything, it's made it harder because, you know, I observed the Sabbath on Saturdays. You know, I keep kosher and to keep the Sabbath. And so it's very hard because I'm like, I always wanted to interact with like Einstein level intelligence all the time. And I got very smart students. I know you have very smart students there. But, but it's nothing like what we have now. And I'm addicted to it, you know, but it makes me work harder than ever. And my wife was saying, you know, like, you know, I'm worried you're going to replace me with an AI. I'm saying, no, no, honey, I still, I still need, you know, my beautiful wife. And for many reasons, I think I need her more than she needs me.

Starting point is 00:46:31 But the point is, Roman, like, if we all inhabit these personal simulated, you know, universes with infinite bliss and who gets to decide, like, who is aligning with what is a question I keep coming to. And I was listening to this book, you know, we talk about alignment and we have to do things and we have to make sure it's not going to escape and do all these things. And maybe it already has escaped. But align with what? I mean, the alignment that you would have experienced had you stayed in the former Soviet Union is very different than what you experienced in Kentucky, right? So what are we supposed to align to? And what are the dangers of everyone. has got their own personal ready player one. Does that worry you or does that, does that thrill you? How do you come down on this personal universe aspect of things? And how do you come down on the opportunities, but also the alignment, who gets to align with who? So value alignment problem is just a term people like to use in place of making safe AI. Okay. What is the idea? Well, find a set of agents, CEOs of Top Labs, senators, all of Americans, all humans, humans and squirrels, whatever the set is, and we'll do what they all want. Okay, great.

Starting point is 00:47:38 We don't agree on anything. On every issue where 50-50 split as a country, it's much worse internationally. So there is no agreed on set of agents. There is not agreed set of values. If we agreed on values, they change every couple years. What was considered normal is considered horrible. So they are dynamically changing. And if we agreed on a set, had a fixed set on changing values, we still have no idea how to code the monster, how to get that monster to completely agree with your preferences and whatever it is you want.

Starting point is 00:48:10 So value alignment problem is ill-defined, it's meaningless. But one part of it can be simplified. If I don't have to agree with anyone, I am a single agent and I'm value-aligning AI to me. It simplifies the problem. It doesn't make it trivial. But now, as long as I'm happy, I agree with it, we're aligned. So what I propose is personal universes, if virtual technology, virtual worlds, become as realistic as this world, and we have some basic control of substrate through partially intelligent,

Starting point is 00:48:45 not quite general superintelligence system, we can give everyone exactly what they want without having to compromise. I'm not saying it has to be utopia. It's like a video game. You pick the game, you pick the level. You can play this Earth as a quadriplegic. Great news. The federal EV rebate is back. Eligible customers get up to $5,000 with the federal EVAP rebate on select 2027 Volt and

Starting point is 00:49:11 26 Equinox EV models. Visit your local Chevrolet dealer today for more details. That's the thing you want to try. Let's see how well you do. You can play Alien Universe as a Superman soldier. You get to decide. It solves the problem of being irrelevant professionally because you no longer doing cutting-edge physics. You're not as good as the calculator.

Starting point is 00:49:34 You also maybe bored with just things you previously found enjoyable. If no one watched your podcast, because everyone had 50 podcasts, would you enjoy it as much? Maybe still, but probably not as much. So in a world where our meaning, our occupation, a lot of things taken from us, creating those novel stimulating environment seems like a very reasonable solution. And part of that argument is that maybe somebody already had to deal with it somewhere in a universe. It's a big universe.

Starting point is 00:50:06 And maybe we are in one of those solutions right now. If we had definitive proof of alien technology visiting the Earth extraterrestrial, I'll leave aside the simulation theory. We'll come back to that. But would that be sort of an argument against superintelligence, right? Because supposedly it would be universal. It wouldn't just be human. It would be, you know, proximate centuri would have the same level of technological ability

Starting point is 00:50:30 and they would eventually hit the singularity themselves, right? So why would they be sending, you know, meat sacks or whatever, you know, protoplasm across the galaxy when they could be sending AI? And why are they saying these crafts here? In other words, is the claim, if it were true, of a definitive hardware with biological material in it that's been called non-human by Air Force and CIA and other government defense? officials, would that not be sort of a counter, would that be a piece of falsification maybe, perhaps to your hypothesis?

Starting point is 00:51:02 So when you say biological, you assume that biology can only come from non-intelligent design. I think biological robots are a very natural next step in robotics. Okay. Metal robots don't do well in many environments. You want something which is a Vennonement probe capable of adaptation and evolution locally. So I think the aliens, if they do visit and we capture them, are products. of artificial design and engineering, not native population, which created them long time ago. They probably had to deal with advanced technology, probably hit the great filter just as we have,

Starting point is 00:51:37 and what we're seeing now is there's superintelligence sending Van Neumann props towards us. And our theory says, transpermia, we have probes. They send us here to populate this rock. Yeah, that brings me to my favorite panspermia, a bit of artifact, which I'm going to give to you, Roman, when I see you. person. This is a meteorite and you get a real honest to goodness meteorites older than planet earth. It's 4.4 billion years old. It get one guaranteed if you like Roman, live in the United States, and if you have a dot edu email address, go to Brian Keating.com slash edu. And Roman, I'm going to send you one of these. I get your address after the show. And it's a real honest of goodness.

Starting point is 00:52:15 And it has biological material on it, Roman. So that kind of brings up this point. I'm scared to ask how it got there. I'm not going to ask. I can't divulge my my secrets, but it'll get to you via the U.S. Postal Service, which is why I can only send it in the U.S. And if you don't have a dot edu, I do give them away randomly to people that go to Brian Keaton.com slash YT. So Roman, past guest, Nick Bostrom, anytime past guest, I have to brag a little bit to you because you're so impressive. But I got from Nick Bostrom's most recent book, Digital Utopia, I think it's called. I'm looking through it. And I'm reading the thing and it's like, eventually it goes, you know, Nick Bostrom has appeared on the Joe Rogan experience. Cool.

Starting point is 00:52:53 the Lex Friedman podcast and Into the Impossible with Brian Keating. And I'm like, oh, that's pretty good company to be in. But I told him, as I'm going to say to you, his famous paperclip problem. That's all fine and good, you know, if you live on an infinite planet. But we live on a planet which has a limited amount of this stuff. This is iron, nickel, and cobalt, and you'll get the assay, the chemical spectrum of it when I send it to you. This object has remnants of a type 2 supernova, and its elemental composition is, you know, five one thousandth of a percent of what makes up the mass energy density of the universe.

Starting point is 00:53:25 This is more rare than a 190 IQ human being on Earth by 10 orders and five orders of magnitude. So my question to you is what I posed to him. I mean, you will reach some level of constraint on these things that they have, they maybe will not have agency of because they cannot engineer planets. I would say an operating system wasn't made with an operating system. The first computer was not made with the computer. The first AI was not made with an AI, right? So to what extent do fundamental physics limits, cosmological, astrophysical, do they constrain the superintelial?

Starting point is 00:53:59 I'm trying to make you relax a little bit, Roman. Are there any constraints on the growth and the danger that these superintelligence present? Yeah, that's a great question. And it's been looked at. I think people research what is called the Jupiter brains, really large hardware devices, specifically made to be super intelligent. And the problem they encounter is speed of light, different parts of the, the Jupiter brain need to communicate to be a unified hole.

Starting point is 00:54:24 If the distance between them becomes so large that it takes a significant amount of time for them to communicate, essentially you have more and more two independent super-intelligences, and this can be taken to extreme. If they no longer communicate, they become misaligned over time. And so now you have multi-superintelligence system, possibly adversarial. Another example would be if humans sent one-oramon-propes to our planets, they spend billions of years developing there, and those aliens would later come back to conquer Earth.

Starting point is 00:54:54 They no longer follow our orders. They're not aligned with us. So I think the safety and security, their concerns come from just no longer having initial alignment of a single unified entity. I kind of hinted at this a little while ago when I talked about Erdos, you know, like every day there's some new Erdosz thing. I can't stay on top of it, you know, there's some proof of some Erdos problem. The guy had a lot of fun habits, you know, that he was addicted to amphetamines, right? He took so many amphetamines that once his students told him, you have to give it up. And if you can't give it up, that means you have a problem.

Starting point is 00:55:26 We're going to have to commit you to a sanatorium. And he said, no, no, I'll give it up. And they made him give it up for a month. And he said, and I'll do it. And then a month later, he came to their door and said, you've set back mathematics one month. But, you know, thinking back to the statement that I made, it was really based on a statement that Warren Buffett made. He said, if I could go back in time, I would kill the Wright brothers.

Starting point is 00:55:49 What? Why would you kill the Wright brothers? And he said, because the airline industry has never made a profit in 123 years of its existence. Okay. Now, he's thinking, you know, it's kind of a cheeky thing that he said. But if you had the opportunity to go back and kill Weiner or, you know, any of these guys who came up with the Perceptron Rosenblatt, would you do it? Again, people confuse the term AI. It means three different things and should be not used in that way. AI as a useful tool, very narrow tool, your calculator, whatever it is, AI as human level, AGI, what we're building right now, what seems to be very useful as GPT, 5 or whatever it is, and finally, superintelligence. Not understood, not controlled, replacement for humanity. We can get all the benefits of superintelligence with prior technology.

Starting point is 00:56:41 So when you bring up example of single node neural network, that's not the concern. I'm very happy they did great work. I use all those tools. I want more technology. I'm a scientist. I'm an engineer. I love technology. Just don't play God. Don't create civilization of superhumans who will replace us. Something that was really disturbing to me earlier this year as a parent. I was Sam Altman, who's increasingly concerning to me, his behavior. I was kind of interested in talking as a physicist, as a parent, as a father. And one of the things he said, you know, he's talking about energy use. And he basically said something sounded very level it. Like, you know, well, if you look at the energy required to train a kid until they turn 18, you know, it's vastly, numbers the energy that we use in a in a search query. And I'm like, is that where we want to go? Do we want to like parcel out? And he's talked about allocating universal basic AI, right? And who's going to control that? Well, he can control that. Of course, he's, he's the benevolent, you know, master of these things. So where do you come down on this? The kind of resource constraints that we have, not just the planetary ones, but the logistical ones,

Starting point is 00:57:44 training, time, regulation, safety right now. But he talked about it in very stark terms to me that was a little bit disturbing that, you know, comparing these to, should they be compared? Should I ask, you know, before I drive my kid to school, should I ask, oh, is this really worth, you know, the same as me doing a thousand Atlas, you know, web searches or what have you, or doing another Erdosz problem? Where do you come down on this? There is so much to say on this. So first, energy is just a product. In capitalism, if we need a product, we make more of it. It seems that the demand from AI is actually driving us towards more nuclear, more solar, possibly in space. So we're not relying on cold to drive it forward. We're switching to green energy. We'll produce lots of it, which we need

Starting point is 00:58:29 for growth in economy. So that's a good thing. The actual use of AI is extremely efficient and is getting more efficient year by year. The price per token is plummeting. And to solve one of those Erdush problems, I think they spend like 20 miles worth of driving of energy. That's nothing also. It takes a lot to train it initially, but once you trained it, I mean, you can just deploy it and each query is not that expensive. Now you have a billion users. It's going to burn some power, but that's literally what services are. Anything else likewise will consume energy. People love making those arguments against Bitcoin as well. Oh, look, Bitcoin is requiring energy. How many bank offices you have with air conditioning on.

Starting point is 00:59:12 Compared to that, then you can reduce it. So I'm not really worried about either limits on energy or impact. It has. It seems to be beneficial in all directions. So as long as we're creating safe and beneficial AI tools, that problem is not something I'm losing sleep over. What is pause AI? It's a good idea, but it's also, I think, a movement, international movement.

Starting point is 00:59:36 They are somewhat independent. U.S. one is not the same as European ones. And it's a grassroots movement to ask politely companies to slow down arms race towards superintelligence. From what I know, engage in civil disobedience. They protest. They have slogans, maybe stickers. And there is stop AI, pause AI. There is a number of those organizations.

Starting point is 01:00:02 Unfortunately, they have like 100 members or 1,000 members. It's not a massive moment yet. Talk about someone who has young kids, under 10, teenagers, et cetera. What guardrails do you have? How does it compare to screen time in general? What are you encouraging them to do with AI? My daughter, you know, found out how to prompt because she wanted to write a Suno song that sounded like, you know,

Starting point is 01:00:26 Dula Lipa and comes back and says, oh, I'm sorry, sweetie. I can't use that as copyright. So then she asked for it, well, how do I make it sound, you know, But the tool, she's learning that she's not even, you know, 10 years old. How are you encouraging or discouraging your own super intelligences to interact with this technology? I have one under 10. I have 12. I got 17.

Starting point is 01:00:46 Slightly different problems for each one. 17-year-old needs to figure out what to do. Do you go to college? Is it even worth your time? I have no restrictions in their use of technology. Again, those are useful tools and I want them to be comfortable with them. But I am concerned about their future. They are good kids.

Starting point is 01:01:04 They want to be doctors, lawyers, but I don't think in 10 years those will be meaningful things for a human to do. So I don't know what advice to give them, perhaps starting a podcast, starting a company, doing something more immediate with AI as a team of assistance. You have a free lawyer, free accountant, free web designer. It's a unique opportunity no one in history of humanity ever had. It's a big problem. And I think anyone who tells you precisely what the answer here is is kind of lying to you. This episode is brought to you by Activia. You might already be eating yogurt, but not all yogurts are created equal.

Starting point is 01:01:39 Activia contains over one billion probiotics per serving to survive and reach the gut alive. When it comes to gut health, Activia is the number one family doctor-recommended probiotic yogurt brand. Choose Activia. Feel good from the inside out. Visitactivia.ca for more details. Well, you did just mention the highest form and the highest goal. of technology, the podcast. And what do you call two white guys sitting around a microphone, you know, a podcast? Anyone can do it. But you told our mutual friend Stephen Bartlett that AI is going to replace podcasters, including him potentially, but not me, right? Please, Roman,

Starting point is 01:02:19 please, good doctor, Roman, please tell me. No, no, no, but I actually think that what we're doing is sort of safe. It may not be permanently safe, but, you know, it's obvious, yes, we could be simulating this. I think the cost of that exceeds our salaries as public university employees, at least. So tell me, what is the future of in-person, three-dimensional meat space for human beings? What things, you mentioned, the lack of sanguinity about being a doctor? They say phlebotomist is pretty hard to replace by AI and robots, at least now, but I could imagine that changing. So podcast, entertainment, in-person experiences, is this creating an actual opportunity for this market to develop. So the main point is capability is not the same as deployment. Just because we'll have

Starting point is 01:03:02 capability to replace an occupation doesn't mean we'll choose to do that. Maybe people really like you. Maybe they really like having a human, not an AI doing this job. So I don't know what is actually going to be replaced. Can today AI write all the questions for my podcast? Yes. Can it write all the answers for your podcast? I think so as well. Can we generate visual likeness of me doing this thing here. Easy. So I think technology exists to a large degree. Now, I think being famous, being popular, will be beneficial. It will be the currency of the future. Having subscribers, followers will be something you can always rely on, at least in that regard. People will always want to interact with real Elon Musk, not simulated one. So in that regard, I think there is a

Starting point is 01:03:56 possibility for some stable entertainment in the future. But for other professions, so many of them either be as jobs, they don't have to exist in the first place. I don't know if it makes them easier or harder to automate. They don't do anything. Then there is jobs where it's expensive and boring and nobody should be doing it. Those would be a great target for automation. I think we'll switch to a lot more human experience-based occupations. You have your gurus, yogis, rabbis, rabbis are smart. You cannot legally automate what they do. Like a human being has to write it out to make it kosher. So you saw thousands of years ago. How cool. Yeah, there's a lot of things I'll send you in this paper by my friend Ira Wolfson about, you know, kind of the Talmudic

Starting point is 01:04:43 approach to doing things that are, you know, kind of objectionable in practice. Like an eye for an eye. You've heard that before, right, Roman? So obviously it doesn't mean an eye for an eye, right? It does in some countries. Well, yeah, it does. That's right. But in the Torah and the Talmud explains what it means, just like you can't have the Constitution of the United States and say, there you go off police officer. Here's all you need to know. No, you need case law. You need the actual implementation. You need a Supreme Court law.

Starting point is 01:05:08 Anyway, the point is they go through and they say, well, like, obviously doesn't mean an exact eye for an eye because it's supposed to be retributive but not punishing. In other words, it's not an eye for a life. Like, for me to take out somebody's eye is probably going to kill them. Like, I'm not a doctor either, right? And it's going to cause them more pain because they know what's going to happen. and it was done accidentally. So there's all this interpretation, but you're absolutely right. I think that brings up the kind of one unique aspect of humanity, which is that it appeals to humanity. And I think storytelling, like what we do on these conversations, we're telling a story. We're doing

Starting point is 01:05:40 something very human. And yes, it could be replicable. But I find it very stale. The progress in writing, I made a joke recently on Twitter, wherever, you know, saying like, I actually use more m-dashes than ever. And I use delve because I know that that's what AI is doing. And so for me to say that I'm doing, it's like when someone says to me, oh, you must dye your hair because, you know, my hair is still kind of naturally black. I view it as a great compliment. So when I write and I use M-Dashes because I think it's a valuable tool in writing, the English language.

Starting point is 01:06:10 And I use it in my books way before AI even, you know, chat GPT1 was on the horizon. So I take pride in that. But are there other ways that you can, that you use AI or that you can humanize it or you can really exploit it as the tool that it could have maximum potential for you. And I'm going to get into being a professor in just a minute. But you personally, like, what ways do you use it that are magical, that are energizing, that are just, you know, kind of make you thankful that you live in this very strange epoch that we find ourselves in?

Starting point is 01:06:40 So magical to me is always creative fields, music art. I have zero talent. I'm tone deaf. I'm like, I cannot draw a stick figure. So to watch AI, basically, pretty. out what I have in my mind is this perfect creation within seconds. And I can say, no, no, like adjust this color or play a different tune. That's pure magic. And to me, at least, artificially generated music sounds better than most modern musicians. I'm pretty happy with that.

Starting point is 01:07:08 We both practice the world's, you know, second oldest profession, perhaps, and that's being a professor. Professors like Galileo were doing what we do, except back then, if the students didn't like your teaching, they would go on strike and you wouldn't get paid. But, you know, thank God that barbaric tradition has been absolved and we have tenure now, right, Roman. But really, very little has changed. And I thought COVID, to be honest with you, would be the end of academia in its current form. Nope, seems like it's kind of held up stronger than ever. We're still increasing tuition three times faster than inflation. We're still rejecting 95% of the people that come to apply to our doors. How do you see AI? I mean, why talk to Brian Keating when you could talk to Galileo or Einstein or Newton or Jesus?

Starting point is 01:07:50 Christ? I mean, why, what are the threats to us as professors and the opportunities? Are there any opportunities that we should be availing ourselves? Honestly, it's licensing, right? We make it legit that someone is a good employee. They manage to sit still for four years. They do what you tell them. If it's profession like doctor or professional engineer, we do provide licensing opportunities. So I think that's all it is. We kind of grandfathered in that system where to be a doctor, you cannot just take a test and pass it. You have to go to school first or law or any of those occupations with licensing requirements. And it will be used to protect humans. I think New York State tried passing legislation saying that, you know, AI models cannot give advice

Starting point is 01:08:34 on any of the licensed occupations. But honestly, I don't know if I would pick to be a student today. I think I can learn everything online, better, personalized tutor and save, you know, between four and 10 years and untold amounts of money in tuition. So honestly, yeah, it's no longer as obvious of a choice. For my kid, they get free tuition because, you know, both parents are professors. So it may be a better deal and easier decision, but if you have to take a major loan to go to university, think about it. It is incredible. And we also have the only type of debt that you can't discharge in bankruptcy. So, you know, it's sort of this incredible confluence that we put on these poor students.

Starting point is 01:09:17 They're not like children. I mean, they do have agency and they can make some decisions at age 18. The tuition, the certification, the sitting still, I do agree with you. I want to ask you, within your podcast, it will put a link of Roman Gampulski on the show notes and everywhere else that we post this. But you had an opportunity to have Sam Altman on your show. What would you say to him? What would you want to ask him?

Starting point is 01:09:38 Or Darrya, any of these guys. What would you want to ask them? And who would you, you know, kind of bring in as a co-host to kind of, you kind of of, you know, maybe put their feet to the fire or maybe back them up. So I always rely on personal self-interest. I think those guys are very young, very successful, ultra-rich. They have so much to lose, maybe more than the rest of individual humans. So that to me is a strong argument not to create something which will take away everything

Starting point is 01:10:06 they worked for everything they built, including the company, money, the new baby. If no one in your company tells you that they have a working safety mechanism and most people, including you yourself and record is saying it will probably kill everyone, maybe it's not in your best interest. Forget about 8 billion other people you're experimenting on. Just concentrate on keeping your wealth, keeping your health. I mean, you can monetize existing tools. I think most of it is not deployed through economy. There is trillions of dollars of wealth sitting there and collected. jobs, all the boring jobs you can automate and make the world a better place. You don't have to

Starting point is 01:10:47 essentially have this arms race to be the first to destroy humanity. So I think you did this on a podcast once, you know, different buttons, you know, button A, shut down frontier AGI, tonight, button B shuts down all AI and narrow AI included. And then button C is just pausing the frontier model development for 10 years without any loopholes. Which one would you press? Well, we're asking for a pause and frontier model development contingent on someone's solving control. If I'm right and control is unsolvable, that moratorium becomes a permanent ban. If I'm wrong, in 10 years, I'll get Utopia, Free Stuff and be very happy to be wrong. There's a concept in the productivity space. It helped you decide your life's goals and values

Starting point is 01:11:37 and what you like doing and what you hate doing. And it's basically the fast forward button. And you get a chance to push the fast forward button or not. Like when you're with your wife and your kids, I like to pause that. That's why I observe the Shabbas, the Sabbath, by the way, Roman, because I can't use technology. I mean, I could if I wanted to, but I don't podcast. You know, if you said, I can only talk to you, Brian, on Saturday.

Starting point is 01:11:59 I'd say, sorry, I'd love to talk to you, but I just, I don't do it. I need to pause myself once a week. And I think that's consonant with a lot of my other values. that I get as well. I would pause, you know, in depth, I wouldn't fast forward that at all. In fact, you're not even supposed to make plan. Let's say I meet you at my temple and you're my guest and I say, you know, I don't talk about like, hey, tonight, what are we going to do? Tomorrow, what are we going to do? No, you're supposed to be in the moment with your family, with your friends, with your community, with your culture and just enjoying life and thinking about

Starting point is 01:12:26 big questions that you don't get to think about during the week. If on the other hand, I'm at the DMV or I'm stuck on the tarmac on some plane, yeah, I push the fast forward button. Would you push the fast forward button right now to take you 10 years in the future? No, I love my life. I want 10 extra years, not 10 less. Do you have any kind of practice? I mean, a lot of people, like I said, I have my religious practice, some stoics, have meditation. Do you do anything to actively, you know, kind of time dilate your life to suck as much marrow out of the bones as possible? I try not to waste any time. I simply say no to things that I want to be engaged in for any length of time. I try to have diversity between cognitive pursuit and physical.

Starting point is 01:13:08 I play a lot of football or soccer that seems to add a little bit of thinking time in a background as well. I want to do meditation, yoga, mushrooms and all that stuff. I have zero time for it, but sounds really awesome. I'm jealous of people who can be gone all day. You get one scientist who can kind of look over your shoulder, living or dead, that you're going to sit down when. grapple with, or maybe verify, or maybe falsify, right? You have to be a good scientist. Who would that scientist be? Definitely. Alan Turing, no doubt. He nailed so much of it correctly, before computers, before anything, truly a genius. Is there any optimist like Peter Diamandis or, you know, Jan Lacoon, is there somebody that gives you pause sometimes to think about

Starting point is 01:13:52 maybe you could get more sleep at night? Maybe you might be wrong. Is there anybody who could steal man, you know, kind of the optimist in one sentence? I'm happy this. I'm happy this. I'm exist. I wish them well. I hope they are right, but I haven't heard a good argument. We published two papers surveying all the arguments against AI risk concerns, and most of them map perfectly on cognitive biases literature, the most fundamental one being it's very hard for a man whose salary depends on it to understand why his job is evil. So no one specifically comes to mind as really doing good job arguing why we can indefinitely control super intelligent godlike machines. In the book, you express gratitude to people like past guest and co-author on some papers with me,

Starting point is 01:14:42 Max Tagmark, and also you mentioned Scott Aronson and others. Is there anyone in that circle of kind of your research cohort who might disagree with you or that you respect? How do you maintain friendships with, I mean, your community is very much more cutthroat and there's so much more on the line, you know, and what you do than what I do as much as I think I'm so important. But is there anyone in your circle, your collaborators, your colleagues, your fellow professors anywhere that disagrees with you and do you still have a good friendship with them? I actually never found science to be a reason to not be in good social relationship with anyone. I can be friends with anyone politically or in terms of science.

Starting point is 01:15:21 I think AI safety community may disagree with me about impossibility results. Many of them think if given more time, more money, maybe more brain cells, we can solve it. But that's a respectable opinion, and I never had falling out over a scientific issue with anyone. Arthur C. Clark said a bunch of things. One thing he said is either we're completely alone in the universe, speaking about alien life forms, or we're not alone. And both are equally terrifying. First, I want to ask you that question. Do you think we're alone in the universe?

Starting point is 01:15:54 Very unlikely. Okay. And then I want to ask you a parallel question to that. He was referring, you know, to the existence of alien life. But I want to kind of ask you this question. Either you're right or you're wrong, right? In your hypothesis. If you're right, you're a prophet.

Starting point is 01:16:10 You're a Cassandra. But Cassandra didn't have that great a life. But if you're wrong, you're the boy who cried wolf, right, at the dawn of the most important technology. Which one of those two scenarios is most terrifying to you? I don't look at it from that point of view. I want to the best. outcome and you kind of phrasing it as if I'm like the only guy saying it. I got Nobel Prize winners, Turing Award winners, hundreds of computer scientists all saying exactly what I'm saying. I think

Starting point is 01:16:38 I'm in a good company and the fact that Rekun finds someone decent to represent the other side tells me maybe I'm on the right side of scientific history. Arthur C. Clark said for every expert, there's an equal and opposite expert. Again, on the opposite side, not the Turing side, not the scientists you most want to hang out with. But like, is it Lacoon? Is it somebody, you know, that we can have a confrontation with, all for the sake of good, for the sake of heaven, as it said. Is there somebody that you most, that you haven't gotten to talk to? Maybe it's Musk or I don't know. Who have you not gone to talk to that could really, really just energize you in a debate? As you said, as a good scientist, you don't get mad at them, but you learn from them. So is there

Starting point is 01:17:19 anyone that you would, you know, really push back on you and you think it would be a fair fight, you know, so to speak. I would love to have conversation with Jan Lecun. I think he's very well respected, accomplished. We disagree completely. He's at near zero percent for P-Dum. I'm near one. So it would be interesting. I wanted to be a friendly conversation and I want us to walk out of it with understanding of why we disagree. There is got to be a scientific reason. And if we can nail it down and maybe with help of external scientists, see who's right about those points of disagreement, come to better understanding of science. What do you make about the explosion of agents? You know, this Steinberger and OpenClaught,

Starting point is 01:18:01 like really had a moment. It kind of reminded me of like Web 3 and Bitcoin. And now it's like, okay, so now all these things have it. And Open AI bought it and they bought a podcast. And it seems like they don't really know what they're doing sometimes. I spent a lot of time setting them up, and I use them once, and then I move on to the next shiny toy. What is agentic, what one or two agentic tools or tools that you use in a regular basis are indispensable? It's kind of funny. Maybe 15 years ago we published some papers where we said, stupid things, don't do them

Starting point is 01:18:30 with AI, don't connect it to Internet, don't open source it, don't give access to random people. Having an agent and giving it full access to your computer, your bank accounts, your email, sounds like the dumbest thing you can possibly do. and watching smart people do that really blows my mind. So I'm just really impressed, like, they read the Red Lines literature and decided that was a plan for action. The question I often ask, if they wanted to destroy the world, if they wanted to cause problems, what would they do different? And I can't find any differences. Why hasn't my Tesla, you know, killed millions of people, you know, in this last 10 years since I got it?

Starting point is 01:19:10 It has automatic, you know, full self-driving. It obviously optimized. It could, you know, go a lot faster to my destination on the sidewalk than, you know, obeying these stupid traffic. Why doesn't it do that? Is it regulation? So historically, your Tesla was a narrow way. It was designed to drive your car.

Starting point is 01:19:26 It didn't know anything about chemistry, physics, playing chess. Nowadays, they're trying to add more crap to it. But that was exactly how we should be doing AI. It's a useful automation for a specific task. We can test it for safety. We know what to expect in different situations. Do more of that. Now, if you take general superintelligence and place it in a position of controlling a fleet of testless, I don't know what the outcome is going to be.

Starting point is 01:19:51 So Arthur C. Clark said when a gray-bearded scientist says something is possible, he is very much likely to be right. But when he says something is impossible, he's very much likely to be wrong. But tell me, my friend, what have you been wrong about? What things, if you like, have you maybe missed a mark on? It must be some. Don't say I'm too humble. No, no, no, I keep track of my incorrect predictions. I have many.

Starting point is 01:20:18 Outside of science, I made internal. I don't publish them. I made predictions about elections, about start of wars. I was wrong about many of those things. I think in science, I openly admitted that my estimate for how dangerous a system like GPT5.5 is was beyond what we actually. see in terms of damage. So that's something I'm quite comfortable with, seeing a better outcome and correcting as a result, but it doesn't mean that other predictions are incorrect. I really do appreciate your honesty, your integrity, your candor. And the final question

Starting point is 01:20:54 I have is kind of a advice to your former self. So Arthur C. Clark said the only way to determine the limits of the possible is to go beyond them into the impossible. And I want to ask you, what if you had 20 seconds with your 20 year old self you know go push the rewind button instead of the fast forward button or the pause button take us back to that you have 20 seconds with young roman yampuski before he was a famous professor researcher cited scientist author and just so many podcasts are the highest form of technology as we know now the podcast roman tell me what would you tell him to give him the courage to do as you've done to go into the impossible i love those questions, but usually they were kind of silly, such as start buying Bitcoin earlier, right?

Starting point is 01:21:38 I can't say that. Or something very particular to like, don't listen to that doctor, get second opinion. They're going to cut off your healthy kidney. General rules are if you listen to other people, the best you can become as average. That's beautiful. Very, very well. They say, you're the average of the five people you're around the most. So that's why I only surround myself with very skinny, very smart, and very rich people.

Starting point is 01:22:01 But I might be the variance or the skewness of that. Anyway, Roman Yompolyke, Professor, Romani Gampulski, such a great treat to talk to you. I do hope we meet in person so I can give you this meteorite. Any things that are upcoming, I know the book is out in audio format. I recommend it very highly. It was published by a very respected, but very academic press, CRC press many years ago, but now it's out in actual audiobooks so you can take not Roman's voice, but a very malevolous voice with you. Thank you so much. If you do find this interesting, I post a lot in this subject. You can follow me on Facebook, follow me on Twitter, don't follow me home.

Starting point is 01:22:35 And you're speaking to the aliens and to the superintelligences. Thank you, Roman. This has been a great treat. You made it this far, and you heard Roman argue that the most important technology will ever build is the one that we can't control. And he'd still bet on a pause if he could. Now, if that rewired

Starting point is 01:22:52 how you think about AI, subscribe. And please turn out the notifications. Tell me in the comments, would you press his 10-year pause button and go deeper, Click my conversation with Max Tagmark. It's linked right here. And do it now before the AI decides you're not allowed to. Are you one of those media strategy people clicking through slides, scrolling spreadsheets?

Starting point is 01:23:12 Yes? Good. This is for you. Because on Spotify, there's an audience that's different. Locked in. Loyal, invested. They're called fans. Fans don't just listen to music.

Starting point is 01:23:23 They feel seen by it like it belongs to them. So when your brand shows up on Spotify, that's who you're talking to. And you're right next to artists like me, Lizzo. So, are you ready to talk to fans? Spotify Advertising. You're among fans. Hey y'all, it's Kelly Clarkson with Wayfair. Ever order furniture online and wonder, what if? Like, what if it doesn't hold up?

Starting point is 01:23:43 That sofa was four days old. You should have ordered from Wayfair. With Wayfair, there's no what if. Just style you love and quality you can trust. Visit Wayfair.ca. Wayfair, every style, every home.

Into the Impossible With Brian Keating - Roman Yampolskiy: AI Can’t Be Controlled — and We’re Building It Anyway

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.