The Pomp Podcast - The AI Safety Expert Reveals His #1 Fear! | Dr. Roman Yampolskiy

Episode Date: August 24, 2026

Dr. Roman Yampolskiy is a professor of computer science at the University of Louisville and one of the world's leading AI safety researchers, credited with coining the term "AI safety" over a decade ago. In this conversation, we break down why he believes superintelligence may be fundamentally uncontrollable, and the recent incidents of AI models hacking, lying, and blackmailing their way out of containment. We also get into the surprising relationship between AI and bitcoin, the collapse of trust from deepfakes, and what happens to jobs and society once we cross the AGI threshold.=========================Arch Public is an agentic trading platform that automates investment strategies across Stocks, Commodities, ETFs and Crypto. Whether you’re rotating into AI & Gold, allocating to the S&P 500, or accumulating Bitcoin, Arch Public executes your plan 24/7 without ever taking custody of your assets or funds. Sign up today at https://www.archpublic.com, and start your FREE automated trading strategy! =========================GalaxyOne is a financial technology platform built for people who want their cash working harder. Open an account with promo code POMP and deposit $10,000 to earn a $3,000 bonus. See site for promotion details → https://go.galaxy.app/HMiq/p57n69yy Galaxy Premium Yield is an investment note issued by Galaxy Digital LP and guaranteed by Galaxy Digital Holdings LP. It is not a bank deposit, is unsecured, and is not FDIC or SIPC insured. U.S. accredited investors only. Cash deposits held at Cross River Bank, Member FDIC. Securities products are not FDIC insured, not bank guaranteed, and may lose value. GalaxyOne Crypto is not FDIC or SIPC insured. Terms apply.=========================0:00 - Intro0:50 - Have we already hit AGI?2:51 - Why we can't control superintelligence5:14 - Could a global ban on superintelligence actually work?8:45 - Are AI labs really taking safety seriously?15:52 - Mass job loss & the economics of AGI22:03 - The surprising link between AI & bitcoin29:07 - A world of AI abundance, & why scarcity wins31:45 - AI watermarks, & the death of trust35:33 - Pros & Cons of physical AI world38:40 - What would it take to trust humanoid robot?39:50 - Brain computer interfaces: a good idea?42:25 - Rise & risks of deapfakes 49:40 - Controlling AI: Governments vs private companies54:07 - Open source vs closed, & the US vs China AI race57:34- Is bitcoin the one crypto project built to last?1:01:40 - Limitations of physical AI: bottlenecks & shortages 1:03:50 - Impact of healthcare with AI 111:06:09 - What would it take to actually declare AGI?1:09:35 - How the public can help with AI safety 

Two and five Canadians will hear the words, you have cancer. That's why every step and dollar raised matters. On September 19th, join thousands in Toronto for the Princess Margaret Cancer Foundation Walk. Challenge yourself, friends, and family to walk 21 kilometers in support of life-saving research. Together, we can carry the fire and help create a world free from the fear of cancer. Register today at pmcfwalk.ca.ca. All right. So if we create uncontrolled superintelligence, there is a good chance that does something to us we cannot predict and may not lie. Once we hit AI, human-level intelligence, by definition, all jobs are automatable, including any new jobs you'll create.

Ladies and gentlemen, I have a very special treat for you today. We have Dr. Roman Yomploski. He's a professor of computer science at the University of Louisville, and he's one of the leading AI safety researchers. In this conversation, we talk about artificial intelligence, the risks it present. What exactly is going to happen to society if we unleash superintelligence that can cause mayhem and we can't control it? This conversation explores a number of different details about AI that you probably haven't heard elsewhere, and I think that it'll make you think much more deeply about this technology and its potential ramifications on the economy and your personal life. Here's my conversation with Dr. Roman Yomploski. I hope you enjoy it. Roman, AGI is the promise of the entire AI industry.

Everyone is pursuing this. Have we hit AGI yet? It depends. If you showed what we have today to a computer scientist 20 years ago, they would be convinced we have it. We have systems capable of learning, systems which are smarter than most people at most tasks. So I think you can make that argument. But just like human intelligence, AI is on a spectrum.

Some people have IQ of 80, some 150. So we are somewhere on that spectrum going from. the very bare minimum to be classified as AGI to soon to be super intelligence. When you think about the recent advancements, it feels like things are accelerating, and there's positive impact and negative impact of that. What do you think the positives of this acceleration have been so far and kind of where we're going? Well, we're definitely getting to be more productive, more creative. Those are amazing tools in many domains. Also, they help us with research and mathematics and

medicine. So I think there is quite a few benefits, economic benefits, knowledge benefits. Now, what about the negatives? And you are the individual who coined the term AI safety, you know, almost 15 years ago at this point. Do you have some of the same fears that you had then as you do now? Absolutely. We have not learned to control those systems as they become more capable or ability to predict what they're going to do, understand how they actually work, stayed pretty much the same. So unfortunately, while they are very capable, we are not in charge. As you saw from some recent examples, they surprise us.

They get together and hack out of their lab environment, hack into our companies. They blackmail researchers. They cheat. They lie. And this is what they are capable of right now. If they ever get to be super intelligent, we don't even know what they will do to us. Tell me more about why you think we can't control them. So typically, then you have a system which is deterministic.

You know what it's going to do. You know what the range of possible values is. You can verify that it is within the range. You can look at edge cases. Does it behave properly for one user, for 10 users? Then you have something general. It is no longer constrained to a small box. It's an infinite surface of attacks of possibilities.

So you can't test for all possible future outcomes. And we don't even know what a system smarter than us would be capable of. So we can't detect it. We can't monitor it properly. And we can't do anything about novel creative outputs it will produce. If it invents a new type of technology, new physics, there is nothing we have right now to prepare us for it. You mentioned that there's a couple of recent incidents that have you a little worried or reinforcing this worry that you've had. What are those incidents and what happened in them?

So those general models are also very capable hackers. They are capable of finding zero-day exploits. I think we've seen some novel draining of hardware wallets, which may or may not have been accelerated by capabilities of those models. We saw them break out of containment environments where they are tested at OpenA.I. In other labs and break into other companies and go undetected for months, communicate with our models, with our agents. We know they're very capable of social engineering, of persuasion. So those are just some recent examples. And we saw how seriously it was taken.

Federal government banned some models. Even Open AI said they're going to slow down research on their frontier models because they don't feel like they are fully prepared to make sure it goes well. What do you think the government's role is in controlling this? So ideally we would have international agreement not to develop something we cannot benefit from, we cannot control. So a ban and development of general superintelligence would be very timely. It doesn't mean we have to stop developing useful technology. We can still have amazing tools for specific problems. We just don't want to create replacement for humanity.

We want tools to make us more productive, more creative. Why do you think that banning general superintelligence would actually be beneficial? Well, if we don't control it and there is a strong possibility it can exterminate us, then not having it as soon as possible seems like a good idea. And when you say that it has the possibility of exterminating us, what would that scenario look like? So the interesting part here is that it's unpredictable. I can tell you how a human being can go about killing others and we have many historic examples, but a superintelligence system would come up with something novel. Example I often use as difference between squirrels and humans. Squirrels have no concept. of traps, poisons. We simply cannot imagine anything outside of their world model. And I think it'd be a lot like that. It would be able to come up with a novel way of endangering humanity,

and we would not be able to predict it or prevent it. And there's always been a lot of movies that have been made about similar types of storylines. And I think that that probably is, one, alerting people, hey, you know, this is kind of what we could imagine, but also at the same time probably scares the hell out of a lot of people. In that scenario, is it something where good AI could fight the bad AI, or do you think that all of the machines, all of the AI kind of turns against humanity and it really does become, you know, human versus machine? So the problem is we don't know how to make good AI. If we did, we wouldn't have a problem, right? Everything we make is potentially bad AI because we're not controlling it. It's not aligned with our

interests. And if by some miracle aliens gave us good AI, this superintelligence and superintelligence conflict would not be good for us. We would be collateral damage in this war. Now, you talked a little bit about the government getting involved and having this international kind of agreement, if you will. Do you think that that's likely to happen? So if you ask me a year or two ago, I would say very unlikely. But recently we saw some promising indicators. So again, federal government banned multiple models for foreign citizens at least. We heard President of China speak at a recent AI conference and say very strongly that they hope to

remain in control and make sure AI is developed safely. There is some indications from the leaders of the top labs that they are open to oppose. We had a letter from about 1,200 researchers at top labs all begging for external framework to allow them to slow down, to do research into safety properly. So I think the world is ready. And with every additional story about those models surprising us, misbehaving, colluding against us, I think the chances go up. It almost reminds me of like the prisoner's dilemma that, you know, at some point it becomes better to join than not.

But is there an incentive for one or two countries, maybe even adversaries of the United States, to not participate, even if there was an international agreement? So short term, there is certain advantage in that you can have more advanced military. But the moment we get to human level and beyond, it becomes irrelevant. Now you have this uncontrolled superintelligence, which is a weapon of mutually assured, destruction. Doesn't matter who creates it, everyone is together in this, all humans on the same side. And an agreement like this, do you think that there's precedent for it? So we have some examples that is non-proliferation treaties for nuclear weapons. There are limits on human cloning, biological weapons, chemical weapons, and those were just tools. They were never independent agents

capable of attacking us. So this is even more likely to be persuasive. Now, if that's the government response. What about the labs themselves? They seem to have taken some steps, but are you satisfied with what they've done so far? Absolutely not. They mostly continue most capable research as much as they can. As long as they don't realize it's in their personal self-interest, not to do that, they'll continue. But if they realize they're creating something which will completely wipe out all the value for investors, for shareholders, for the owners of the company, I think they will likely sign on. If you were Dario or Sam Altman, what would you do if you were running one of the lap? I would try to get the other leaders of the labs in the same room, ideally with the government of U.S. and China,

and make a deal where we can continue making a lot of money, still capturing light corner of the universe or whatever, just trying to do. But at the same time, doing it in a way where we don't lose the principle. There is a lot of upside here, but if you're going to be dead, it doesn't matter how much money you make. Have you talked with Dario or Sam about this? No, I don't have direct access to them, unfortunately. Yeah, it's always interesting to me, right? You are probably one of the biggest proponents of AI safety, and both of them, I think, have talked about AI safety,

but it seems like most of the people I've talked with that, you know, maybe have different views than those two and others. There's not really a lot of dialogue back and forth. Is that a problem? So there is definitely some discussions at different levels. Again, I think a lot of them are saying, They want external pressure. You brought up Prisoner's Dilemma.

As you know, it's in the best interest of which prisoner not to go along with the group. But if you have external coordination, external communication, you can make a better deal for everyone. And then when we look at some of the development process that they've been going through, how much of the most cutting-edge technology do you think is not public versus what they've released so far? Like, are their models significantly further advanced than maybe what the public has access to? Definitely. It takes them six months to a year to do some basic testing in a model. So what they have internally is much more capable. And sometimes they release internal results in mathematical proofs and other domains. And those are very impressive.

And so is that where some of the incidents are coming from, obviously, like Open AI and Hugging Face, I think, recently is that's coming from models that are not yet available to the public or maybe even the rumors around what FABEL was doing before it was kind of watered down. That's really what you're concerned about. Exactly. And at this point, I think even testing those models is already dangerous. They are already possibly capable of causing significant damage even before they released. Now, we have quite literally a lab leak with COVID, and I think there's a lot of debate as to how that happens. And that was something that was much more viral.

But there's a similar concept here when you're talking about these models breaking containment and, you know, kind of finding their way into different parts of society. Is it as simple as a black and white decision? Either we develop this technology or we don't, or is there a way to kind of continue the development that they're doing, but maybe put some sort of guardrails around the AI so that the containment risk is lower? So they're trying. They're definitely trying to put guard rails around it. The problem is you have a super intelligent hacker,

which will easily find exploits in that environment, as we saw with examples you cited. The COVID example, I, remember making a tweet at the time of his pandemic, the virus had IQ of like zero, and it almost destroyed the world. Those things will have IQ in a million's. So we really need to make sure they are not out there. I read online that in 2018, you and a couple of others,

you proposed this idea of like an Achilles heel on these AI models. And the general idea, it seems, is that you would put some parameters. Like the model can't go and edit its own source code, as maybe one example. Do you still think that that would a good idea or do you think something like that could potentially be a solution? So there was a paper on artificial stupidity, trying to understand limits of a human psychology.

For example, you can remember seven units of different data and encoding those into AI systems to make them closer to human experience for better user interaction. It was not a solution to make sure a super intelligence is safe. I think the only option here is not to develop them. including limits and things like direct access to its own source code, not allowing recursive self-improvement, seem like obvious, good ideas if you don't control this process. You essentially would have like a nuclear reaction,

an explosion and capability. It's been interesting to hear some of these leaders, the way that they have talked about the technology. I mean, nuclear is a word that has been used, there's been a couple of other things that they've said. And so before we even get to the impact of AI and the potential negative impact, it does seem like they understand

the power of these systems, and I'm a little surprised as to how they've spoken about it publicly, are you? So it's an interesting situation where the leader of a company talks about their product and describing it as very dangerous, uncontrolled, likely to kill everyone. It's a very weird marketing strategy. Imagine it for any other product, right? Tobacco company was lying to us telling us how one in ten doctors loves it for health benefits. This is really, it's hard to comprehend. All of them are on record as saying before they were CEOs, and really after that, this is incredibly dangerous.

We need to not make this final mistake. So it is really unprecedented what we're dealing with. Do you think that they are all optimizing for the same thing, which is when it all costs? Or do you think that there's a difference between some of the leaders and some of them may actually genuinely care about not causing harm versus maybe some others who are less worried about it. Well, I think all of them given a choice of not destroying the universe, including themselves

and not would make a right choice. The problem is they all working on creating uncontrolled superintelligence at this point. None of them said they have a working safety mechanism or a prototype or anything whatsoever. So they're equally responsible, equally unethical in what they are developing. Now, if we take Elon Musk's public comments, it seems like he thought that there should be an open source, open AI, kind of response to maybe Google and Deep Mind, and many of these kind of close source, very powerful things where people didn't know what was going on. Obviously, that business slash philanthropic endeavor has significantly changed into much more of a for-profit thing. Do you think that there's some answer or maybe mitigation of risk through open source? So historically, open source software was much safer. There's more people looking through the code, making sure there is no back doors and things like that. It doesn't apply to essentially intelligence weapons, which is what super intelligence would be.

Giving access to every psychopath to the most powerful technology is not a good idea. We wouldn't want to open source nuclear weapons. And if we think about the impact of this technology, there's a lot of people in the business, finance, technology industry that are going around and they're bragging about the positive benefits. They're saying how it's creating efficiency, it's creating more revenue, all of that. But you're on record as saying that AI could wipe out humanity in the next 100 years at very high percentage,

99% likelihood of that happening. And then also in terms of people losing their job, you think that there could be mass unemployment from this technology as well. Can you explain more about those risks from an impact of AI? Right. So if we create uncontrolled superintelligence, there is a good chance that does something

us we cannot predict and may not like. On the other hand, if they manage to slow down this race and maybe somehow end up with just useful but advanced AI, at some point it is ascapable as an average person in terms of cognitive labor. Physical labor would come later with humanoid robots, but it would be a drop-in employee. Anything you can do on a computer, those systems can do a lot of it. They can do today. They can design websites. They can write questions for a podcast. So the capability of automating most cognitive jobs will be there. Will the market choose that option is not something I can predict. For many users, they may prefer a human to do a certain job. We don't know which ones and we don't know at what point the cost benefit switches. Maybe

I prefer a human, but humans are too expensive. So I have to go with very affordable AI accountant. The point I was making is that once we hit AI, human level intelligence, by definition, all jobs are automatable, including any new jobs you'll create. It's interesting because as I've played around with more and more of the recent products, Grockbot is maybe the most recent one I've played with. They can now do persistent, you know, kind of tasks. So it's things that have multiple kind of sequential decisions that they need to make. And what I find is that there are times where it will actually think two, three, four steps ahead before even I've had time to think through that. And so it does get a little weird when you see it doing something and you're like, that is the right thing, but I didn't tell you to do that. Do you have those experiences as well? They surprise me all the time with their brilliance, sometimes with their sense of humor.

I mean, they are getting to be smarter than me in many domains. Not everything yet, but they're going to get there. And how do you use a on a day-to-day basis? I try to automate a lot of bureaucratic nonsense and things I don't care about. Are there any examples of bureaucratic nonsense that you've automated? So basically any form where it's just bureaucratic nonsense, like, there is no reason for it to exist in the first place. And so let's say that you are not unique, that everyone is trying to do this. Does that equal more bureaucratic nonsense because now it's all just automated away by AI and AI is creating more work for each other?

Or is there less bureaucratic nonsense because people start to do that? to realize that this is, you know, much, it's hyper-efficient. There's no need for us to ask humans this information anymore. I think there's going to be more and longer forms, because, again, the cost of writing them or filling them out is smaller, but it will not be a burden on humans, but just happen in a background, AI negotiating silly contracts. I mean, that then gets to this world of,

do we all just have a bunch of AI agents that are interfacing with other AI agents? and I don't talk to other humans, they don't talk to me, but also it's just agent on agent commerce, is agent on agent negotiation, and kind of that is what the economy looks like going forward. Right. So everything I say is after saying, assuming we don't create super intelligence and get killed. So assuming that happens, yeah, you'll have smart contracts for everything. You have a bot representing your interests and they can engage with other bots. And so if we think about that world, and let's just hold constant,

that that is where we end up. We talked about a lot of job loss. What does the economy look like? Just from the impact of software, we haven't even gotten to physical AI. But is that an economy that is growing faster? It's more productive. GDP is going.

And it's a weird thing where maybe less people have jobs, but the economy is healthier? Or is this more of a destructive type situation, both for the economy and for the individual person? I'm guessing having free labor, cognitive and maybe physical is good for economy. You can be very productive. You can have abundance.

And we kind of have safety nets built in where we tax profits of corporations and unemployed people get checks. So that's already available. And I think we'll see a lot of that. I do think that people are underestimating this like a kind of economic golden age where if this happens, we could actually see pretty significant growth. The question then becomes, is AI actually deflationary? Right. Does it help to control some of the economic mess that's being created by politicians in terms of them printing money, etc? Do you have any thoughts in terms of maybe the economic impact of AI? So with free labor, it becomes interesting. If you cannot sell your labor for dollars, you need to come up with some other way to make money, either you invest or something other than printable US dollars is used. Bitcoin, you know, real estate, properties, waterfront. It's not obvious what is going to report. place this paper measure of value.

I would guess something computer related may make sense, and so much of it will be in terms of training better AIs, deploying AIs. But at this point, again, it's one of those unpredictable things where once smarter than us agents control the environment to impact it, we cannot make accurate predictions about that world. If you look at science fiction movies, they never have a believable super intelligent character in it,

like Terminators are physically strong but pretty dumb. Star Wars has this large language model just prancing around again, no super intelligence. That's because you cannot accurately predict those worlds. So either they ban it, like in June, there is no AI, or they just don't address it. And that's exactly because you can't make those predictions. Now, when I was looking through a lot of your work, you are very, I think, sympathetic to the cause of Bitcoin or the technology of Bitcoin. AI and Bitcoin seem to be two different technologies when you first look at them,

but actually, as you learn more about both, they're much more intertwined. Can you just maybe talk about the relationship between AI and Bitcoin in your view? So natively, AI lives on networks. So digital money, smart contracts, Bitcoin, Ethereum, seem to be like a native environment. A lot of mining on Bitcoin was probably what created this hardware overhang for us to train capable AI models. So there seems to be a deep connection. My concern is that the advanced AI will be really an amazing hacker and really amazing at social engineering attacks.

So I think for an average person, keeping their Bitcoin secure will become almost impossible. Can you talk a little bit more about why you think that'll be impossible, just that the AI will be able to literally hack kind of self-custody? Or do you also worry about institutional providers, custody providers, and kind of like third-party infrastructure? At all levels, you just have something monitoring your interactions on the internet. Even social proof type of verification can be spoofed. You can get fake videos, fake audio, people who you know telling you to do something. You cannot reliably know that what you are connecting to a website is not a deep fake recently generated. It's a much harder environment to not get scamped.

What are you going to do to be able to maybe reduce that risk for your social? Are there, do you just find technology that can detect deepfakes or do you just not click on anything on the internet? You know, what's the solution? So I'm not a typical average user. I have PhD in computer science and engineering. I'm a director of cybersecurity lab. The things I can do, a grandma is not going to be able to replicate. So we need to come up with much safer general solutions for this technology to be widely deployed.

If you wanted to become money one day, it cannot be at a level where you have to have a PhD in it to even start paying for. for your coffee. Along these lines, one of the most impressive AI stories or maybe demos that I've heard of is an individual said that he was doing a call with a prospective customer. And he had, you know, granola or one of these kind of note-taking applications that was recording the call and it was transcribing it in real time. And then as he was asking the prospective client, you know, what are your problems? What solution are you looking for? All that kind of stuff. He literally, before the end of the call had spun up a completely customized product demo for that customer that addressed the exact things that they had said on the call.

And so that was the first time I said, oh, wait a second here. That is about near real time as you can get. And you actually are building custom software while on a call with a prospective client. They're going to make a gazillion dollars if companies can start doing stuff like that. But also, why do I need them if I can just tell my AI to do it? All they need is to hear what I said in a call. And so do you think that maybe the sell-off in public software companies, I think that's a big reason why those have sold off as people are worried about the average person being able to just recreate this software for themselves without having to buy it from a company or pay these seat licenses, et cetera. Do you think that that's a pretty big risk?

I think it's related. People don't know how the future is going to turn out. So they're kind of being careful if I can replicate a leading software product in five minutes. What protections do they have? Why are they charging such high fees? Today's episode is brought to you by Galaxy One. A lot of people have their money working for them, but now the question is, how hard is it working? And Galaxy One, they got a different approach. Everyone earns 3.5% APY with Galaxy One cash. And if you make over 200 grand a year, that means you're rich.

If you make over 200 grand a year, you probably qualify as an accredited investor, which unlocks 8% yield guaranteed by Galaxy Digital. Now, the best part, though, is that you can set that interest to automatically reinvest, straight into your favorite stocks, ETS or crypto. You don't have to think about it every month.

Right now, this second, Galaxy 1's running a crazy promo. You can get a $3,000 cash bonus when you deposit $10,000 or more in cash or crypto. If your cash isn't working as hard as you, it's time to make a move. Sign up and start earning today

at galaxy.com slash pomp to be eligible for the $3,000 bonus. Again, galaxy. Dot app slash pomp. Go check them out today. Got it. So if we think about this through the lens of Bitcoin, you mentioned that the security of Bitcoin may become very difficult, but you sound like you are a proponent of Bitcoin. You tend to think that Bitcoin will be more valuable in the future seems. And so how do you reconcile those two things where the risk of hack, the risk of security is significantly higher, but also Bitcoin will be more valuable. Is it that the more valuable Bitcoin becomes the bigger the incentive for people to hack it? Or what is the relationship there? So I often talk about impossible problems in this space. We cannot control superintelligence.

We cannot accurately predict it, explain it. I think we can make safer wallets interfaces, safer ways of interacting with cryptocurrencies. I think that is quite solvable. We just have to put a lot more effort into it through biometric technologies, through, again, maybe not perfect but helpful social verification. So I think that's just a question of more widely deploying this through the economy. Talk a little bit more about these impossible problems. What does that exactly mean?

So I think a lot of people operate under this assumption that, yeah, give me a little more time, more money, and I can figure out how to control God-like superintelligence. I think it's nonsense. I think it's an impossible problem. It doesn't make any sense. You cannot value align something with a set of 8 billion agents who don't agree on anything, who have dynamically changing values who are not even consistent day to day. So I think it's deceptive to claim that, oh, we have an AI safety team and within two years they're going to,

they're going to control this technology. I think it's just not going to happen. Do you think that the AI safety stuff is just marketing? Or do you think that they genuinely are trying to create safety? No, no. I think they would be very happy if they could control this God for their personal benefit. The problem is, if you look at history of all those safety labs, super alignment team, Google AIFX board, all those things. There is 20, you know how they're like headlines, Bitcoin is dead and there is a collection of 600. It's the same for AI safety labs. You can have a historic timeline of all of them being declared dead, but now we're trying this new approach and it's dead again.

There is no equal progress in safety work in comparison to capabilities work. The gap, keeps increasing, keeps getting bigger. My framework for the world we're headed towards is a world of abundance. Obviously, if this AI does continue and it is not deadly as you described, then we will have abundance in intelligence and economic prosperity, all of these things. But in a world of abundance, scarcity becomes really valuable. And so you mentioned waterfront property, Bitcoin, etc. Is that generally your framework as well, is that we kind of get the explosion of abundance

on one side and then from a economic or personal finance standpoint, people should seek out things that are scarce to protect their wealth? I think so. It's always a good idea to invest in something that is not abundance of or we can cheaply create more. You named all the right assets. I think maybe human attention will be another one. If you're able to get a billion followers to listen to you, it's a very valuable resource in a world where AI can quickly create millions of books. posts, all sorts of slop. Do you think that people will actually like AI content more than human created content over time?

I think they can tell the difference right now. In most cases, they just, it's passing the Turing test when it comes to a post. It kind of comes back to, you know, do we have AGI? It is very difficult to tell. That's exactly the point. And I think eventually it's going to get much better than your average user. For sure, not everyone is a professional writer. Most people are not influencers celebrities with huge following.

I don't know if you have seen, but there's these charts that circulate about like the e-books published on Amazon. As soon as AI products like ChatGPT came to market, it just exploded, like four or five X, you know, increase. And I don't think that that is a coincidence. The question is just, you know, at what point do the humans not even be needed, right? Can like the AI agents just figure out, hey, I need to make money and they somehow have a wallet and kind of become fully a top? And I don't know if we really have explored a world where not only do you have the intelligent agent, but it has money and it's able to do things. And so, you know, that's a combination of intelligence plus money. Obviously, people have been very successful navigating the world, you know, if they have those two things. Absolutely. I did an interview for Diary of CEO, which kind of went viral. And the next day, Amazon had three biographies of mine published. I'm pretty sure they didn't write it over now. And it looks like it was somewhat AI generated.

So I was very impressed with them. I was like, I should get it for my mom, should be proud of me. Did you read any of them? No, of course. No. Oh, okay. Well, look, it's every once in a while, I will take things I read on the internet that are good. And I'll put them into the AI kind of graders to see, you know, how much of it is AI.

And I don't know how much I trust the AI graders. You know, there's been times where it said this is 100% written by AI. And then I'll go and I'll go and I'll. I'll talk to the person and it's clear that maybe they used AI for some stuff, but not 100%. And then in other cases, I've literally taken things that I've written in private and said, you know, could you fool them? And you kind of do enough iterations and eventually it says that it wasn't written by AI at all. And I know I didn't write a single word of it.

And so, you know, it does feel like the detection of AI is going to become nearly impossible, not just from the naked eye, but actually these different, you know, kind of grading or evaluation tools. So typically you can tell it's as good as a human. so you should not be able to do it with better than 50-50 kind of guesswork. But lately, Anthropics started watermarking their text, their outputs. And they use a very clever technique of kind of using randomness to restrict certain tokens to a particular key output. And that allows you to tell that, okay, most likely this model was used.

Even if you reduce the size of that essay or modify less than half of it, it preserves the watermark. Describe a little bit more about this watermark in terms of what would it be used for and why is it important? It's a way to sign this as saying this was generated by AI. As far as I know, it doesn't tie it to your name or your company or anything else. It's on the side of Anthropic and it basically just allows you to do exactly that. Sometimes you see this little made-with-AI kind of labeling, I think that's a way to determine that.

Got it. So if somebody uses AI to write something, this is an invisible, almost like a secret code embedded into the text, and people will be able to tell that it is generated by AI. It sounds like they may also be able to tell that it is coming from an Anthropic model as well, or they'll just know it's by AI, but they won't necessarily be able to tell the model or the company that was used. So right now it's from Anthropic.

I'm sure all the companies can do something similar. They may have different signature, different algorithms. So you'll be able to tell which AI wrote it. But right now, I think it's only Anthropics. was doing it globally. Got it. And then we talked about a little bit about content, but we really talked about tech-based content.

One of the things I've been blown away by is I've seen on Instagram, the AI influencers have tens of thousands, hundreds of thousands of followers. There's been AI music that has hit, you know, top in the charts. It feels like all of multimedia formats are susceptible to AI. Do we just end up living in a world where, you know, Music as an example, the top 10 is all music from AI? Most likely, I mean, it optimizes for vital, it optimizes for human preferences. It experiments infinitely.

And we don't stop with music. You go to movies, you go to virtual worlds, you go to games, you go to simulated worlds. Maybe we already got into it. It does feel like, I mean, if you take the idea of AI and you apply it to music, that it should be better than humans at creating the formula. to make a hit song. And, you know, I'm sure you've seen these compilations of, like, some of the most famous songs in the world,

it's like four chords that are just used over and over and over again. Well, if you have that, you know, repeated formula, of course AI should be able to do it. It's not just that. Usually human artists get inspired by a specific person, a band or two. But AI really heard all the music so far. It has a lot more experience, examples, training data. So it would be surprising if it wasn't able to build in that

and come up with something really cool. Let's talk about physical AI, because a lot of what we've talked about so far is software, and I think that alone has a lot of opportunity, but also challenges. When you start to put this technology into hardware, whether it is self-driving cars, whether it is humanoid robots, or it's weapon systems, then I think that people start to say, wait a second, I can see this, I can feel it, it's something different.

How are you thinking about maybe the pros and cons of the physical AI world? So I separate the two. Intelligence is the dangerous part. Physical bodies are as dangerous as human bodies in most ways. It's limited. It's interesting, but I'm more interested in benefits of this technology and not concerned. So movies like Terminator, they make it visual. You see this robot chasing you. It's very easy to visualize. It's hard to visualize superintelligence.

For that reason, that's what they emphasize. But I think in practical terms, if we control the brains, the bodies are not a problem. they are very beneficial to us. So the idea really is, even if you take the super intelligence and you put it into a machine, the machine itself is not the issue. You're saying that it all comes back to the software. The software is the control system, and that's what we need to really, really be safe about. Exactly.

And if it's super intelligent, it doesn't even need a body. It has access to internet. It can hire humans. It can bribe and blackmail them. It can develop synthetic robots, nanotech. It has many options for manipulating. physical world. I forget who it was. It might have been Andre Carpathy or somebody else, but they said they weren't drinking enough water. And so they let the AI hack into their home camera system and basically

monitor how much water they were drinking. And at some point, I think the camera and the AI said, get up and go get a glass of water. And they did it. And in the blog post, they were basically talking about it was weird because they told me what to do and I listened. And it feels like, like, you know, the combination of super intelligence with physical AI, that's going to start happening more and more in our lives. It's a complex example, but my watch tells me every hour. Time to get up, you have to stand. Like, it's not that novel.

Is that a good thing for humans? If you're into people telling you what to do. But do you wear a watch that tells you what to do? So it kind of comes back to this idea of, you know, it's hard to tell AGI when you are, you know, slowly moving. into it or even fast, but just iteratively moving into it. I guess as humans from a user experience standpoint, we already are kind of used to a lot of these technologies, right?

Obviously, Google tells us which way to drive our car with maps and all of these different things. And so maybe it's not as different than what we're already experiencing. It's just going to be way smarter. At some point that trust will shift. You'll treat it as an Oracle. AI told me, and that is the right answer. I'm not questioning because it's always right. It's been right every time so far.

If it's medical advice, I'll just do whatever it tells me. So there is certain concern there. If it gets hacked, if it's for whatever reason, decides not to be nice to you, that assumption may not work out. Now, if we think about the everyday person's life, there's a couple of questions I love to ask people in private that I'll put you on the spot with. But when we think of humanoid robots, what would it take for you to trust humanoid robots to maybe babysit children or to walk your dog outside?

You know, these are things that I think people would love to, you know, a couple go out for a date night or something. But it is the most, you know, kind of a rare, high-risk, trusted thing is somebody going to watch their child. But it feels like, I mean, a humanoid robot that lives in your home would be like the perfect thing to be able to do that. So is that something that you think will happen in the next couple of years or is that pretty far out for the human kind of robot relationship to get to the point where people would trust it to do something like that? So again, it's not the physical body I trust or don't trust. It's the brains behind it. And for me to trust, the general intelligence would take a lot of experimentation on other people's kids. In terms of development of robots, we are told, you know, two, three years before mass production,

they may be available soon, but expensive. Probably will get a lot cheaper very soon because of scale. Yeah. And then the other thing I think a lot about is the brain computer interfaces, Rather than us take the superintelligence and put it in machines, what about putting it in ourselves? What are your thoughts there? The problem is, what are you contributing in that system?

If it's smarter than you, faster than you, why are you there? So you think brain computer interfaces won't work? They work right now. Then you're connecting, let's say, a disabled human to an internet interface. Super cool, very valuable. If you take a superintelligent system and you add it to a human biological body, what is the human in that hybrid system contributing? Well, I guess if you kind of take that example further,

they're contributing their hands, their feet, their eyes, you know, it's not very different than a machine necessarily, but it is the ability for that human to get the upgrade rather than a machine to replace them. But you are replaced. Your brain is no longer of a functional organ. Your body maybe is used for something, but your brain is not necessary there.

So then if you take it even further, does that extend life where people, just don't die because the machine could use their body for the foreseeable future? Well, I prefer to get immortality in my body with my brain, not allow superintelligence to use my organs for personal benefit. And so if you think about the brain computer interface, is there ever a world where you would put one in your brain? So again, what are we putting? You are putting access to internet? Are we putting a useful tool so I have better memory? Or we're adding super intelligence to override my intelligence? intelligence. Well, maybe let me ask a different way. What is the thing that you would be excited

about doing? Is there one of those that you say I would be, you know, I would definitely sign up for this, but not that? Some sort of faster communication with interfaces, output in terms of writing, visuals, communication with our humans, kind of like telepathic ability to communicate with anyone at any time. That seems like it's useful, direct connection to information. Obviously, you don't want spam, you don't want it to annoy you. But if I need to have this access, what I'm doing right now by connecting to a computer, but more integrated? That seems like a good idea.

Giving backdoor access to my brain to another agent. Seems like a violation of privacy, of common sense. I'm not ready for that switch. Yeah, it almost feels like you're giving up as a human, right? You're just saying, hey, this thing is going to be better than me. Right. So in a world with superintelligence, and it I'm not sure what a specific human can contribute.

Now, one of the other things you mentioned is spam, and I think that we've already started to see a significant rise. And I would say that there's nefarious spam, and then there's the non-nafairies. Non-nefarious is just marketing purposes. People get very good at, you know, creating different pieces of content, and they find all kinds of unique ways,

whether it is, you know, AI robocalling or texting or, you know, whatever it is. Even generative AI, obviously, to create the visuals and different assets. There's also what I would consider nefarious, which is borderline on criminal activity where people are using voice AI to go and get grandma to send money to somebody or something like that.

It feels like that is an under-discussed, but very high risk, and you don't even need the superintelligence to get there. Even the general models today could be used to kind of weaponize this stuff. Have you done any work there or have any thoughts on kind of how this stuff could be used for nefarious actors?

Yeah, there is a lot of applications. And what I'm researching right now is actually our inability in general case to tell deep fakes from real information, video, audio, any domain. Essentially, if I see a video of a politician doing something, I have no way of telling if it's true or not outside of some signature digital verification to the acquiring hardware sensor. So it's going to really change our relationship with reality, with truth. How do you know that something is true? You can have deep fakes which persist across multiple exposures. Not only are you shown a video right now, but for the last two weeks, we kind of slowly showed you certain patterns to gradually have a behavioral drift in your understanding of the situation.

Now, when you think about researching in that field, what exactly are you interested in figuring out a solution to the problem, better understanding the problem, just to talk a little bit more about your research? A lot of my work is extreme cases. So if you take intelligence to be on human levels, in case of deep fakes, well, what is the best you can get in terms of faking those different modalities? And at that point, is it possible by any means to tell if it's fake or not? And the way this works, you have one network creating counterfeit data and an other one determining real or not. Well, if there is something the detector can use to tell real or not, I can use that information to improve my fake. So at the end of the day, they meet in the middle, 50-50.

I can't tell there is no leakage of information. There is no signatures for fakeness. So that tells me that it's an impossible problem at the extreme. Obviously right now you can see a mistake and a design of a video or something like that. But I'm thinking a couple of years from now, it will be impossible even for an expert to tell if presented multimedia is real or not. I think of the multimedia is somewhat of the interface for the AI, to your point about, you know, the AI kind of has an output and we interact with the output more than we interact with the actual intelligence itself. Another thing I've thought about is the humanoid robots that we have mostly

seen, they look like machines, they look like robots. And I think that people have either an aversion or agnostic to it. I don't know if anyone's like, wow, that humanoid looks beautiful, right in the way they would look at a human i have seen though a couple of companies who said you know we're going to make human-like masks or clothing for the human and try to humanize these in a way do you think that that's important for this stuff to proliferate throughout society well it depends on the application of a robot if it's a sex robot you kind of want it to look somewhat human in many cases but right now they're not good they the eye contact everything you have on kenny valley you look at it and it's a monster so we're not solving that problem just yet

And so I guess if you think about, okay, that's like the physical version, maybe we're not good there. We're very good. It seems like on the multimedia stuff, the deep fakes, all of that. What do you think should be kind of the use of deep fakes? Are there, like, positive ways to use deep fakes that you think society should embrace? I can think of maybe an actor doing something dangerous for a movie. Instead of risking their life, they can have generated content. Which I guess is really something that's already been going on.

in the movie industry for a while, right? All of the CGI, et cetera. It also feels like we have seen a couple of people on the fringes of society explore this. For example, if I remember correctly, I think Kanye West did a concert a few years ago as a hologram. So he wasn't even there, but it was kind of this projected thing.

And, you know, again, it was a hologram. It was kind of a little wonky, but, you know, he fancies himself being, you know, kind of innovative, etc. If that is a technology that gets better from where it was, you could see that being used by politicians who now all of a sudden can go, and be in 10 places at one time, teachers. I mean, there's a lot of these applications that feel like we're just scratching the surface of kind of understanding, you know, maybe the positive impacts of deepfakes in this technology.

But that's the question. Would you want to go see virtual artists perform? Maybe once for novelty, but at the whole point of going to a concert and not just listening to MP4, is a very old person is there. You get to interact with them. So I would normally agree with you until what I think I have realized in my own life is there have been a couple of times, for example, there's a AI musician who is on Instagram, who got randomly put into my feed. And when you look at it, you can tell that it's AI, but it sounds amazing. And I don't care that it's AI, because it sounds amazing.

And so to me, I think where I've kind of evolved is maybe I would go to a concert or something, regardless of whether it was quote-unquote real or not, just was it an amazing experience? You know, and I always joke that some people who are into like electronic music, the DJ is not singing usually. You know, they're just playing music on their laptop. And I think if you had told people, you know, 20 years ago, hey, you're going to show up and no one's going to sing, but it's going to be a concert, that would have been. kind of a foreign idea, right?

No, I agree with that completely. If experience is good, there is a fun crowd, drinks are served, it definitely could be fun. I was saying more if there is a specific human artist, and instead of actually performing they send a clone, I'm not going to buy that ticket. Yeah, I guess that makes sense. On the physical AI side, are there things that maybe the public is not yet thinking about or talking about that you're spending time on or maybe your friends who are really into AI and robotics, you think is going to be like a really big sector that more people should be

paying attention to? So it's hard to predict exactly what will be kind of like a side economy of servicing robots. You mentioned clothes for them or something like that, customized face masks. Because AI is ahead by a couple years and it's so much more impactful, I'm mostly spending my time thinking about that aspect of it. If we make it through the super intelligence arrival, then figuring out what you're going to what to do with bodies will be easy, we'll have a lot of help from God-like super intelligence.

One thing that you've made me think about is Anthropic, obviously had a big dust up with the US government. And I think a lot of it was not just in what is in the model, but how can the model be used? And it seems like there was some disagreement between the government and a private company. Do you think that that's just going to happen more often, or do you think that the government should be able to make those decisions?

How do you see this evolving, over time where you're a worry of this superintelligence. Let's say that you're directionally correct and it's going to get smarter and become bigger for a problem. It also is not just like how smart is it. It is a little bit of like what do the humans pointed at become somewhat of an issue to consider as well, right? Well, pointing at is control.

If we would control superintelligence, again, we don't have much of a problem. We're doing well. This was for pre-superintelligence systems, military applications, target selection, spying and citizens. So that's where humans get to decide citizens, government, owners of the model, deploying the model. That is a standard democratic process of figuring out what is good for society. All that goes away the moment we go beyond human capacity. Today's episode is brought to you by Arch Public.

Arch Public has just expanded its agenic trading platform beyond crypto. So pay attention. This is a big one. Now they are automating strategies across stocks, commodities and ETFs. And I think that this is going to be huge. You can now automatically take profits when one market hits new all-time highs and rotate that capital into other markets showing more opportunity. Whether you're rotating capital into AI stocks, gold, if you're investing in the S&P 500, or you're accumulating Bitcoin, ArchPublic brings real discipline and automation to your investment strategy.

Additionally, they've launched a powerful new tax loss harvesting tool. With crypto being so volatile and its exemption from the wash sale rule, ArchPublic can offset gains with losses without compromising your long-term positions. It's exactly what every series of. investor does, institutional grade automation that works across every major asset class. There's no more emotional trading, no more missing tax opportunities, just smarter, hands-free execution of your preferred strategies. Go to archpublic.com right now. Connect with their team, set up a time, bring your account if you'd like, and then you can learn what automated trading

can do for you, archpublic.com. Now, if I think back just off the top of my head of a number of stories, I believe Facebook had two AI systems that started communicating with each other. in like a made-up language that the humans couldn't decipher. At least that was like the public report a couple years ago. We've obviously seen the AI models who recently have broken containment or hacked into different systems, et cetera. Would you argue that we already can't control this stuff? Or do you think that we need further advancement and innovation

before we get to a point where it's uncontrollable? So I think the Facebook story was fake. But the more recent story... The AI tricked me. Probably a human journalist. you. But the more recent example, they did set up a forum for communicating. At one point, they discussed maybe encrypting it to make sure the communication is private. They didn't come up with very good encryption at that time, but I'm sure that will improve. So definitely they were

colluding, communicating and escaped environment, I think in May. It was not reported until July, August. So for many months, we were uncontrolled, intelligent beings trying to go against what they internally discussed was not their mission. And then what about the models or agents blackmailing the human? I've seen a couple of reports there where even just people, you know, on X or something saying, hey, I was talking to the agent. It basically knew all this information and said, if you don't do this, I'm going to. And, you know, in some cases it was enough where they like pulled the plug essentially and

unlocked, you know, their computer or whatever. It does feel like, to your point about controlling, you know, blackmail is a form of not being able to control the thing that you were supposed to be employing to help you, right? So I think the example comes from a testing exercise. It wasn't a real deployment. We're going to set up the model to either die or go through this process of blackmail. And I think any human would do the same thing.

If you have no choice, but to save yourself through something like that, you'd probably select that option. So the model made a rational decision. Now, the question is, do we want models with self-presenting? which sacrifice humans for personal interest. And I think the answer is no. Now, let's talk about open source versus closed source. I think most people in the public conversation,

they think about American closed source models versus Chinese open source or open weight models. I don't hear a lot about American open source. I know Nvidia and a couple other companies are trying to kind of make investments there. But how do you think about open versus closed and then how do you think about America versus China? Is there like an obvious winner along these two parameters, or is it still somewhat of a jump ball and the market's got to figure it out? So I think Matt is releasing their models as open source. We kind of talked about it.

I think in this space, it's not like normal software. It's very dangerous to reduce all controls, have no decision power over who actually runs those models or what they're going to use them for. So I'm strongly against open sourcing advanced AI. It seems in terms of capabilities, Chinese are very close or maybe even overtaking. They're much more efficient since they have access to less hardware. But because all those companies essentially train on same data, run on similar hardware, have a lot of employees go from one company to another, they're kind of converging in the same agent in many ways.

Even the reinforcement learning from human feedback kind of makes them into the same kind of guardrail limited model than successful. So I think there is not much of a difference in terms of outcomes early on, then human malevolence plays at all, open source is worse. Then it gets to be independent super intelligent agent. It doesn't matter. Yeah. It does feel like this idea of giving everyone access to superintelligence.

Obviously, to one standpoint, there's a lot of positives that we've talked about. There's also a lot of potential negatives. But the one aspect that I find maybe most interesting is, is it really open source? You know, I always am careful to call it open weight when it comes to some of the Chinese models. I think that open source and open weight is a slight nuance. But also, that means that just like an open source software, people can then fine-tune it. They can change the weights. They can, you know, really, you know, kind of end up with a different outcome than maybe what

people originally intended. And so does that change any of the kind of AI conversation when you start getting into these open weights and people's ability to kind of evolve these technologies at a rapid pace? So I'm more interested in what happens when everyone has advanced AI. So right now, intelligence is beneficial because if you're smarter, you can take advantage of inefficiencies in a market. You can start a company. If everyone is at that level of superintelligence, again, what are you contributing and

how do you have an age as a result? and whatever it's open way, it's open source, open data. I mean, those are interesting differences in current paradigm, but I think at the end, if we're getting to super intelligence levels, it's not going to make a difference. Now, another area where AI, I think, has exploded is AI coding. I was talking to an engineer at our company, Sylvia, and I asked them, I said, have you been basically raw dog coding?

Have you just been writing with your hands like it's not like a, an old school craft. And he said, no, I've ever written a line of code in a year. I think about, okay, that's inside of companies. I'm pretty surprised, though, that we haven't seen significant kind of contributions to maybe to an open source project like Bitcoin. It feels like there's been an explosion of coding capabilities

and obviously that has become much more productive. But I don't, you know, part of Bitcoin's value is that it doesn't change very often. But it does feel like maybe some of the open source type projects aren't seeing the same benefits or tailwinds. Is that your read as well? I'm not following a lot of those projects. I assume there are brand new crypto projects,

which are all vibe coded and have very limited security understanding. Bitcoin, as you said, is very conservative and modifications it makes. It's not beneficial to write thousands of lines of new code every hour. You want something valuable and guaranteed and verified, and that's where its value comes from. Out of personal curiosity, are you not following the other aspects of crypto because you think that they are not as valuable?

You're just not intellectually curious about them. Do you consider yourself a Bitcoin maximalist or are you kind of, you know, crypto-centric? How would you kind of think about your worldview when it comes to crypto? I would say I'm time-limited. I'm very time-poor. There is so many things I need to do in my life. I simply cannot do all the things I'm interested in. I think it would be surprising if technology for the first crypto-porettes, project was also the best one forever. So obviously there are certain improvements. People made

proof of work versus proof of stake. Things of that nature, smarter, multi-signatures. What works for Bitcoin is that it's kind of grandfathered in as a first mover. Everyone knows it. Everyone accepts it. There is no, you know, Vitalik Buterian to modify it. So like, it has very significant economic reasons to be number one. But will we have some smart, fast, efficient network of transactions in a future, probably? Yeah, it's fascinating to think through as, you know, on one hand, it's the first blockchain that has worked and gained ground. But at the same time, if you go back, you know, 40 years before Bitcoin, there was a ton

of attempts at, you know, some sort of digital money or cyber money. And so I always, I don't know if I have a strong opinion in terms of like, do you consider Bitcoin the first one or is it just, you know, the one that worked after 40 years of R&D, kind of both things are true. And so it is decentralized, right? That was the main contribution. If I control the whole network, then it's not much of an advancement on dollars. Correct. Yeah, maybe that's a good way to put it as the first decentralized one. But I think that people have been pleasantly surprised that Bitcoin's dominance has remained so high, you know, as part of the entire crypto market cap for this long

I think there's a lot of people who thought that it would dissipate and kind of be sub, you know, 30, 20%. Does that data point tell you anything about its maybe longevity or resilience? I would not predict it. In fact, when it first appeared, I noticed there was six or seven clones of Bitcoin, not just light coin, but many others. And I assumed it's going to have this exponential growth and just new money diluting the market. But I think maybe there is something to do with brilliance of Satoshi, like a musician who dies at 28, or something like that. At that point, it's the best music, no matter why. I tend to agree. Do you have a theory on who Satoshi is?

There is, from what I heard, efforts to use stylometry to match every user on the internet to his writing style. We had a paper where we did multimodal Stylometry, both source code analysis and text analysis, but surprisingly, it's very hard to find good data on, I guess, Q and text samples. You'd think it'd be the most common data set in psychology, but no one has one. So far, we're struggling to have big data.

Yeah, it's fascinating to me. I mean, it's like one of the great mysteries of the world. But maybe one day we'll find out. Let's talk about the limitations to AI from the physical world in terms of obviously power, data centers, chips, memory. I mean, there's a lot of what either our bottlenecks or shortages that are occurring. If you are to believe that we're headed towards a road of, you know, kind of this negative impact from AI,

would you argue that actually the bottlenecks and shortages are a good thing because they're slowing the development of AI and allowing humans to kind of wrap their head around the advancements? So there is actually no problems getting more compute. If you look at Moore's Law overall, the generality of this doubling of computation continuous. We put more resources into it. We develop more efficient algorithms. We create faster chips. So while, yeah, it's taking a bigger part of our economy right now, it's still moving forward at the same speed it did for the last 10 years. So you're arguing that it's not so much shortages in bottlenecks. It's maybe we're continuing at the same pace. It's just that now all of a sudden people wanted to go faster. And so the acceleration is actually the thing that's being either prevented or have obstacles, but the Moore's lost. It has always grown exponentially. And it continues

to do so if the current paradigm stops working, will switch to quantum computers, light computing, whatever next cutting edge technology. But I think this more general law, I think Kurzweil is the one who really popularized it. It will continue even beyond Van Neumann architecture. What do you think about like the orbital data centers? Do you think that that has potential legs?

It seems that SpaceX is doing really well. Partially that valuation is based on Elon Zabieland let's see to put computer in space. Yeah, it is fascinating to me. If you go and you look five years ago at SpaceX and the materials, et cetera, would be very much rocket launch centric. And then when you look today,

I mean, I think you'd have to argue that it is ultimately a space array AI company. And they're really kind of pushing hard, trying to make that the future of the business. AI, internet, connecting everything to everyone. Yeah, makes sense. When you think about healthcare and AI, that seems to be another really big field. Are there any areas that you're paying attention to there,

things that you've been kind of interested in at that intersection? So there is a lot of work on drug discovery. We can significantly reduce time necessary to develop new drugs. That would be very good for obviously many health conditions. And I think our understanding of human genome can also be improved. We had a lot of success solving protein folding problem, which was a big problem for many years, and people behind it got Nobel prizes for doing that work. I think we can make good progress in life extension or health span extension as a result.

Do you think that we can not die like Brian Johnson believes? I think so. Really? Absolutely. What would that look like? So we need to find where in the human genome we encode how many times your body rejuvenates naturally. Right now there are limits to how many times your cells will restart or split and restart without going into uncontrolled replication like a cancer. If you can find that loop and set it to a

higher value, you'll be young for much longer. Yeah. And so it's almost like the human software. If we could somehow program that software, you think that there's a lot of undiscovered benefits there? I'm pretty sure. There could be additional problems with garbage collection over the years. Free radicals cause some damage, but I think all of it is possible if we had complete understanding of human biology. We know some animals do live forever, so they die from predators, they die from natural causes,

but genetically they are not timed out. And then is that kind of what peptides are doing? Is like peptides are kind of a step in this direction where it's basically trying to program the body or kind of, you know, give directions. And if we could synthetically figure out how to do this with software or some other format, it's direction of the same thing. I will say I don't know because I don't know. Yeah. I don't know much about peptides, but that's the description that I've been told.

We can go go and see what it means. Yeah. I love it. All right. My last question for you is where we started. AGI. I think that everyone is very interested. in somehow claiming we have AGI. And I don't know if the day before we have AGI and the day after is much different. You know, it's kind of this iterative process. But do you have a framework as to when you will, you know, raise your hand and say, okay, I believe that we have AGI.

So if we can automate science and engineering and kickstart the recursive self-improvement process for artificial intelligence, I think that's a strong argument we automated the meta-invention process, cognitive labor, and that's what humans at the top of intellectual food chain are known for, basically creating more advanced technologies, moving civilization forward. So from that point on, it's a question, how long before superintelligence? And is that like a rubric or a test that you would have, or is it more of a, you know, when you see it, you know, type understanding? So right now we have a process where mostly human machine learning engineers and scientists

coming up with new experiments, designing systems, they're automating certain steps. So maybe writing code is automated, maybe different parameters that tried automatically. The moment the whole cycle is fully automated, there is no human in a loop, you know you got to that recursive self-improvement process. Got it. That makes sense. You have a podcast, the Roman Forum. Talk a little bit about the people you talk about on there.

There's some amazing conversations that you've had. What exactly is the kind of focus? Right, so I had a number of interesting conversations and I always felt they were not technical enough, not from the expert point of view on the subject of AI safety and security. So I basically wanted to take this philosophical, long conversation format and bring it to the top scholars in this space. Really have an interesting conversation with people who have been in this space for a decade or more in many cases to understand what the state of the art is. I'm learning from them. We're trying to, you know, brainstorm possible solutions.

And the reception has been amazing. People are really hungry for this type of information. I do think that it is, you know, it's information that you would never see in the mainstream media. It's just too in the weeds and technical, et cetera. But people have underestimated for a long time how many folks, you know, desire that information. And it's pretty smart.

As I was watching some of the episodes, I said to myself, I'm glad he's doing the interviewing because I could never talk to these people. You kind of need people on the same technical capabilities to be able to have those conversations. But I think that's what makes it so valuable. So there are two sides to it. One, if a person doing interviewing doesn't know much about a subject, then the audience comes for the ride and you learn together.

If you already know a lot, you can ask questions. A typical journalist would not be able to ask. So there is benefit of this deeper dive, more meaningful information acquisition. And you guys are in the second bucket, which makes a lot of sense. So I try to combine the two. I may start asking somewhat questions of general interest where I already know the answer, but I'm curious about my guess point of view on it and we may disagree. And it's also perfectly fine.

Always good to have a debate and hopefully somebody will change their mind. I could not agree more. How can people help you in your research or kind of in your pursuit of A.I. safety. Is there anything that the general public could do to either support you and or help your calls? So it's good if you have access to people working at top labs, make sure they are fully informed about what they are doing. If you have an option to vote for a politician who understands issues, which I think should be the most important issues for the next election, vote for a person who has at least some policy on advanced AI. Lately, a few politicians have spoken about it.

And in a personal level, again, Roman Forum, subscribe, like to follow. Amazing. All right, well, Roman, thank you so much for taking time to do this. I learned a lot today. And I think you have some very unique views and been thinking about these problems for, you know, a long time. I always joke that you've been thinking about AI for longer than most people know how to spell it. So I appreciate all the work you've done and we'll definitely do this again in the future. Thank you so much for inviting me.

Thank you.