Bankless - 177 - AI is a Ticking Time Bomb with Connor Leahy

Episode Date: June 26, 2023

AI is here to stay, but at what cost? Connor Leahy is the CEO of Conjecture, a mission-driven organization that’s trying to make the future of AI go as well as it possibly can. He is also a Co-Founder of EleutherAI, an open-source AI research non-profit lab. In today’s episode, Connor and David cover:  1) The intuitive arguments behind the AI Safety debate 2) The two defining categories of ways AI could end all of humanity 3) The major players in the race towards AGI, and why they all seem to be ideologically motivated, rather than financially motivated  4) Why the progress of AI power is based on TWO exponential curves 5) Why Connor thinks government regulation is the easiest and most effective way of buying us time  ------ 🚀 Unlock $3,000+ in Perks with Bankless Citizenship 🚀 https://bankless.cc/GetThePerks  ------ 📣 CYFRIN | Smart Contract Audits & Solidity Course https://bankless.cc/cyfrin  ------ BANKLESS SPONSOR TOOLS:  🐙KRAKEN | MOST-TRUSTED CRYPTO EXCHANGE https://k.xyz/bankless-pod-q2  🦊METAMASK LEARN | HELPFUL WEB3 RESOURCE https://bankless.cc/MetaMask   ⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum   🧠 AMBIRE | SMART CONTRACT WALLET https://bankless.cc/Ambire  🦄UNISWAP | ON-CHAIN MARKETPLACE https://bankless.cc/uniswap  🛞MANTLE | MODULAR LAYER 2 NETWORK https://bankless.cc/Mantle  ----------- TIMESTAMPS 0:00 Intro 3:12 AI Alignment Importance 9:40 Finding Neutrality 14:16 AI Doom Scenarios 21:06 How AI Misalignment Evolves 25:56 The State of AI Alignment 32:07 The AI Race Trap 41:49 Motivations of the AI Race 56:18 AI Regulation Efforts 1:14:28 How AI Regulation & Crypto Compare 1:21:44 AI Teachings of Human Coordination 1:36:53 Closing & Disclaimers ----------- RESOURCES Connor Leahy https://twitter.com/NPCollapse   Conjecture Research https://www.conjecture.dev/research/   EleutherAI Discord https://discord.com/invite/zBGx3azzUn   Stop AGI https://www.stop.ai/   ----------- Related Episodes: We’re All Gonna Die with Eliezer Yudkowsky https://www.youtube.com/watch?v=gA1sNLL6yg4  How We Prevent the AI’s from Killing us with Paul Christiano https://www.youtube.com/watch?v=GyFkWb903aU  ----------- Not financial or tax advice. This channel is strictly educational and is not investment advice or a solicitation to buy or sell any assets or to make any financial decisions. This video is not tax advice. Talk to your accountant. Do your own research. Disclosure. From time-to-time I may add links in this newsletter to products I use. I may receive commission if you make a purchase through one of these links. Additionally, the Bankless writers hold crypto assets. See our investment disclosures here: https://www.bankless.com/disclosures 

Even if you, your listener, don't totally buy the existential risk thing. Maybe you don't buy it. It's fine. But it is the case that the leaders of all the top labs, anthropic, deep mind, have been on the record saying clearly that they do think that there is a realistic possibility that these technologies will kill literally everybody, and they're doing it anyways. Welcome to Bankless, where we explore the frontier of internet money and internet finance. This is how to get started, how to get better, and how to front run the opportunity. This is David Hoffman here without my co-host, Ryan Sean Adams.

But regardless, we are here to help you become more bankless. Today on the episode, we're talking AI alignment and AI safety once again. We're talking to Connor Leahy, the CEO at Conjecture, a mission-driven org, trying to make AI go well. We wanted to do one last episode on the AI alignment and AI safety conversation because we think Connor can really deliver a very compelling and easy articulation as to one. why AI safety is real and why it needs to be treated as such. Some main benefits and takeaways that you're going to get from this episode. First, the intuitive arguments behind the AI safety debate.

The things that you can take to your friends to convince them that AI safety is a real issue. Second, the two defining categories of ways AI could end humanity. And third, the major players that are playing in the race towards AGI and why they all seem to be ideologically motivated rather than financially motivated. Fourth, why the progress of AI power is based on two exponential curves. And lastly, fifth, why Connor thinks government regulation is the easiest and most effective way of buying us time. Here on Bankless, we've had the AI alignment, AI safety conversation a handful of times with different players from the industry. Ever since we had that Eliezer episode, which we were hoping would have been an AI crypto conversation, but turns out it was an AI is going to kill us all conversation. We're bringing on Connor Leahy in June of 20.

A number of months after we first went down this rabbit hole because the world of AI kind of feels like the world of crypto in 2021. It is moving so fast. And so Connor gives us the lay of the land, a snapshot in time of the AI alignment conversation as it stands here in late June of 2023. As well as also articulates in the easiest and most simple terms possible why AI alignment is such a big deal. If you, bankless listener, are not convinced of AI alignment going into this episode. and you remain unconvinced after the end of this episode, I have nothing left for you. And so let's go ahead and get right into that conversation with Connor Leahy.

But first, a moment to talk about these fantastic sponsors that make the show possible, especially Cracken, our preferred crypto exchange for 2023, assuming that we get past the AI alignment issue, we will still need to buy crypto assets using our fiat dollars. So perhaps use Cracken to get that done. There's a link in the show notes to get started. Cracken has been a leader in the crypto industry for the last 12 years. Dedicated to accelerating the global adoption of crypto,

Cracken puts an emphasis on security, transparency, and client support, which is why over 9 million clients have come to love Cracken's products. Whether you're a beginner or a pro, the Cracken U.S. is simple, intuitive, and frictionless, making the Cracken app a great place for all to get involved and learn about crypto. For those with experience,

the redesigned Cracken Pro app and web experience is completely customizable to your trading needs, integrating key trading features into one seamless interface. Cracken has a 24-7-365 client support team that is globally recognized. Cracken support is available wherever, whenever you need them by phone, chat, or email. And for all of you NFTers out there, the brand-new Cracken NFT beta platform gives you the best NFT trading experience possible. Rarity rankings, no gas fees, and the ability to buy an NFT straight with cash.

Does your crypto exchange prioritize its customers the way that Cracken does? And if not, sign up with Cracken at crackin.com slash bankless. Learning about crypto is hard. Until now, introducing Metamask Learn, an open educational platform about crypto, Web3, self-custity, wallet management, and all the other topics needed to onboard people into this crazy world of crypto. Metamask Learn is an interactive platform with each lesson offering a simulation for the task at hand, giving you actual practical experience for navigating Web3.

The purpose of Metamask Learn is to teach people the basics of self-custity and wallet security in a safe environment. And while Metamask Learn always takes the time to define Web3 specific vocabulary, it is still a jargon-free experience for the Crypto-Curious user. Friendly, not scary. Metamask Learn is available in 10 languages with more to be added soon, and it's meant to cater to a global Web3 audience. So, are you tired of having to explain crypto concepts to your friends?

Go to learn.menomask.io and add Metamask Learn to your guides to get onboarded into the world of Web3. Arbitrum 1 is pioneering the world of secure Ethereum. and is continuing to accelerate the Web 3 landscape. Hundreds of projects have already deployed on Arbitrum 1, producing flourishing defy and NFT ecosystems. With the recent addition of Arbitrum Nova, gaming and social daps like Reddit are also now calling Arbitrum home. Both Arbitrum 1 and Nova leverage the security and decentralization of Ethereum and provide

a builder experience that's intuitive, familiar, and fully EVM compatible. On Arbitrum, both builders and users will experience faster transaction speeds and with significantly lower gas fees. With Arbitrum's recent migration to Arbitram Nitro, it's also now 10 times faster than before. Visit Arbitrum.io, where you can join the community, dive into the developer docs,

bridge your assets, and start building your first app. With Arbitrum, experience Web3 development the way it was meant to be. Secure, fast, cheap, and friction-free. Bankless Nation, I'm excited to introduce you to Connor Lehi,

the CEO at Conjecture, a mission-driven organization trying to make AI go well. He's also the co-founder of Illuther AI and open source AI research nonprofit laboratory, which interestingly operates mostly inside of a Discord server, much like our company and so many of us in the Bankless Nation. Connor, welcome to Bankless. Thank you so much for having me. Connor, the crypto world has more or less collided with AI. And Bankless, we had our introduction with that, surprisingly, when we had our Eliezer-Yudkowski episode, which we had to pivot mid-episode from a crypto-AI intersection episode into an AI is going to kill us episode. since then we've continued to go down that AI alignment rabbit hole. And I think a decent number of people in the bankless nation and the broader crypto landscape accept the AI alignment problem, but others completely reject it.

It's interesting to see some people just have a spinal reflex rejection of the AI alignment problem. In this conversation, I hope we can kind of just talk about the conversation of AI alignment in the outside world, the companies that are playing here, the game board that is laid out in front of us, But first, I really just want to dive into the very basics of the AI alignment problem and see if we can once again articulate clearly why the AI alignment problem exists, what it is, and why it's important. And you've gotten a lot of practice at articulating this. So I'm wondering if you kind of handhold us through some of the very basic premises behind the AI alignment problem. Yeah, absolutely. So to start things off, the AI alignment problem or existential risk is really what I care about.

This is that something could be so dangerous that an accident or a misuse could occur of such magnitude that it could threaten the continued existence of all of humanity or, you know, curb or potential forever in some sense. This is, of course, be extremely terrible. I think this is pretty uncontroversial that if such a thing were possible, and if it did happen, that would be pretty terrible. So the way I like to think about this is first kind of from like an outside perspective is if we look at the history of technology, you know, we see that technology has gotten better and better all throughout our history, has given us way more power, way more fordance. A lot of this is great, right? You know, we have medicine and, you know, I have air conditioning in my office, thank God, you know, all these, you know, great things that are

wonderful and I love. But also as technology increases, as your power increases, your ability to control the environment, you have more capacity for things to go wrong or for destruction to occur. So, you know, back on the Stone Ages, you know, the worst possible thing that I could do with Stone Age technology would kill, you know, I don't know, maybe like 10 people or something. You know, if I'm like a pretty big guy, maybe if I'm like super smart about it, I could kill a few hundred. You know, if I have my whole war band with me, I could kill maybe more. But like an individual person just using Stone Age level tools, not going to really be an existential risk or anything of that sort. As our technology gets better, you know, we develop more sophisticated weaponry, redeveloped gunpowder. We develop

stuff like this, the number, the damage that can be caused both on purpose and accidentally increases. Sometimes from benign reasons, you know, if you have bigger ships, more people can drown, you know, that's pretty benign. We don't think that, you know, that's a risk that we are willing to take. But in other greetings, you know, there was no such thing as T&T factories blowing up before we had explosives. And now that we had T&T, when an accident occurred, the collateral damage was suddenly of a type and a degree that didn't exist previously. And this is only continuing, you know, so we went from, you know, okay, you can kill like five guys to you can kill like 50 guys, to kill 500. And now, you know, you press the button and drop a nuke out of an airplane and you can kill 50,000 people or even more than that. So if you would like would graph, you know, over time, the like increase in like blast radius of technology, of like, you know, a misuse of a technology or thing, you would see an exponential. You would see that our technology is growing extremely fast, extremely quickly towards larger and larger blast radius. Not all technologies.

You know, many technologies are very safe, and they're very good, and I think we should invest in them, and we should build them. But there are technologies that have larger and larger blast radiuses. And eventually this blast radius will encompass Earth. If our technology keeps improving, at some point we will have technology that is so powerful that it can destroy all humans. And even accidentally, it would, of course, be easier if you do it on purpose. But at some point, if we have powerful enough technology, we should expect things where even accents are a problem. And so I would make the claim that AGI is in this category. In the category of a trend in technology towards more and more powerful systems,

where even an accident during the development of the system has larger and larger brass radius. And sometimes this is what we accept as a society. We accept that sometimes, you know, a clinical trial might go wrong. And this is something that sometimes we accept to some degree. Not to arbitrary degrees. We don't let anyone do any clinical trial without any oversight. Of course not. generally as a society we think very highly of human welfare and life and that it shouldn't be

endangered recklessly. But as this goes on, as we're dealing with these more and more powerful technologies, like what do you do when you have a technology where the blast radius is everybody, including you? Like, how do you develop a technology where getting it wrong ends everything, ends you, ends your experiment? This is not something humanity has experience dealing with. this is not something that we are generally set up to do. The way we usually do technology is we build something and we fail a bunch of times. We mess it up, you know, if you lab assistants lose their hands, you know, like a bunch of stupid things happen. Government gets angry at you and 20 years later, you know, you maybe have something.

This is fine, but predictably, at some point this stops working. Maybe people could argue that time is not now. And I would agree. I don't expect GPT3 or 4 to like, you know, kill all people. I don't. But GV.D. 5, 6, 7, 8, combined with some modern RL, agentic systems, that's much less clear to me. And happy to go into a bit more about why I think this technology has this kind of black radius. But first, you know, just pause there for a second. Yeah. One thing I really want to emphasize in this argument is the neutrality of it.

It's not saying that AI is good or bad. It's just saying that AI is. and this is a continuation of the arc of technology. Technology is not good nor bad. You're just really putting it into very neutral terms that technology has an arc of goodness but has turbulence with associated blast radiuses along the way. And we've all accepted these

because the blast radiuses has been sufficiently small that it doesn't end all humans. But what you're saying is that continue that arc, get it to AI. AI, it doesn't want to kill us. It doesn't want to make our lives better. Humans will make choices with this technology, but it just so happens that in the advent of a bad outcome,

that bad outcomes blast radius contains all of us. So it's not even like a political stance or an opinionated stance about the goodness or badness of AI. It's merely just a statement about the magnitude of what could go wrong if something does go wrong. Yeah, exactly. I think a lot of discourse, around this kind of stuff is very bad. I think a lot of it is completely politicized and

like, psychologically is about, is open source good or bad? Is this guy a good person or a bad person? And I think this is just a terrible way to think about this. We should think about this very neutrally. The question is not, is AI good or bad? It's not, is open source good or bad? It's not, is Sam Altman a good person or a bad person. The question is just what are the outcomes of the various choices we make. How can we do our best to predict what the consequences of taking various actions are? And how do we feel about those consequences? How do we feel about taking these risks? How risk tolerant are we? Who should be consulted? You know, I was never consulted to have, you know, GPT4 released onto the public internet, which I use as well, for it to be released on my

family and my friends. I was never asked if this kind of experimental new technology should be unleashed into the commons that I also inhabit. And maybe that's fine. You know, we don't generally seek consent when someone, you know, wants to write a book. You know, someone wants to write a book, you know, I don't have to read it. It's not my problem. But if someone is releasing, I don't know, a new substance into the water supply, well, I sure hope that I would be consulted about whether I think that's a good idea or not, because I'm drinking that water. I feel like I have a say in this. And so a lot of the problem with the current state of discourse around the stuff is that there is some good technical discourse.

But then there's also a lot of discourse, which focuses way too much on. intentions. It focus way too much on ideologies, like, just like, you know, that we can't derive from an ideology what is true. You know, if you believe open source is good. You know, maybe you believe that, right? But that's not a statement about reality. It's a mood affiliation, you know. Sometimes open source is great. I worked a lot in open source. I think in many cases, open source is fantastic. I'm so glad that Linux exists. I'm so glad that we have so much great open source software. I think this was really good. But should the genetic sequence of smallpox be public? Like, I don't know, man.

You know, I love science. I love academia. I think it's great that people are getting funded to do all kinds of cutting-edge research. A group of researchers in, I think, Canada, used, you know, government funds to reconstruct an extinct form of horsepox or smallpox virus and published how to do it. I think they should have done that. I think this is bad. I think people shouldn't do that.

I think that as much as I love open access science, as much of I love science, we have to be practical here. I don't expect the upsides of this being public to be worth it. So this is kind of like how I like to think about these kind of things. I like to think about these things way more. It's just like, look, what do we want? Like what are the scenarios? What are the outcomes? And then let's work from there. Let's let's work at this completely neutrally. Let's be realistic here. I like to talk about strategy. I don't like to talk about good or bad, if that makes sense. There's a bunch of conversations to be had here about the players in the world of AI. We have chat GPT and Open AI and Sam Altman. We have stable diffusion with

Imad, there's a bunch of people in this game. The regulators are also in this game. I still want to continue a little bit down the defining the alignment problem conversation because there's something I want to parse apart. You're talking about neutral AI and its blast radius, right? What could go wrong if something were to go wrong? When we had our conversation with Eliezer, Eliezer gave us a very strict prescription of how this will go wrong that your explanation of the AI alignment problem contains

but his is more narrow. And I want to parse apart that because he calls this, or we call this, the AI alignment problem, as in how do we align the goals of AI with the human goals? And that's a more narrow conversation

than I think what you're presenting. You're just saying, hey, AI is powerful. It could go wrong. And it could go wrong for any subset of ways, some of which may be the AI alignment problem, but there are also other ways that it could go wrong.

How do you parse apart the ways that are there categories of AI, like Doom, that are worth parsing apart. One of them's AI alignment. Others is like maybe humans go rogue, and then we've seen humans do this, right? Like, what would happen if Ted Kaczyzzi got their hands on a very powerful AI?

Like, that's another conversation. How do you parse apart, like, the actual sources of destruction here? Yeah, I like to use like two, maybe three or four categories, is that the first one is misalignment. It's just you don't have control. Just lack of control. A thing does something random. That's it.

And like it doesn't even need a human to be involved. Right. And that random things blast radius contains us is what you're saying, yeah. Yes. So the claim is that a accidents can happen and that accidents can have blast radius of this side. So it's like accidents. The second one is misuse. So misuse is you have a system which does what you say and someone tells it to do something bad or for there to be conflict of some kind. You know, maybe there's multiple actors who go to war using this kind of technology or like fight. I can barely imagine something more horrific than multiple, you know, you know, super-intelligent systems fighting. Like, could you imagine? Like, it's unimaginable what horrors

such systems could unleash in the terms of war. That scenario is, like, kind of interesting because it implies that the AI alignment problems actually solved. Yep. As in, we've been actually able to align humans and AIs together to achieve the same goals. But human alignment is not solved. And so using our AI superpowers,

we now commit war, and our war blast radius once again contains everyone. Exactly. So it's even worse than that. There's so many ways in which this problem is very, very hard. This is not a super narrow, specific little problem. There are specific narrow aspects of the problem, specific narrow technical problems,

which are very, very important, and we can talk about those. But importantly, this is a generalized problem of society. This is a general problem of the human condition. It's a general problem of how do we responsibly deal with powerful technology? This is the meta problem that we have to actually have to solve. And this is a problem that people have been talking about, at least since like World War I.

You know, there is a Polish novelman named Balfourke Krasypsey, who after the horrors of World War 1 noticed in this like 1920s, he was like, wait, if technology keeps increasing, but our wisdom and our control as a society increase much slower, well, then all humanity will end. He figured out X-risk in the 20s. And yeah, and so his solution was he had to figure out how to improve the art of human rationality. Sound familiar? Yeah. So there was an L. Yezer in the 19th.

1920s, actually, named Alfred Gersipsky. So this is not a new thought. This is not something that, you know, I'm sure people before Alfred has also come up with variations on this thought. And this is continuing this problem. And we have not yet solved this problem. We are already in the problem, you know, the last time we had a new level of powerful technology, you know, we did drop two nukes on purpose.

And after that, there were several really, really close calls where, you know, nukes almost did stop flying, you know, at least two, where it was just one single person each time who stopped the nukes from actually getting fired. So our track record here is like decent, but it's not good. It's going to be much worse when we're dealing with AGIs and Aeon, especially like imagine if these things are open source. Imagine if every person in the world had a nuke during the Cold War. I expect we would have gotten nuked.

You know, I expect it wouldn't have gotten well. If we didn't have chains of command, you know, several people signing off on something, if we didn't have sensible people that took their responsibility, extremely seriously, I think it just would have not gone well. And the way things currently are, there is currently more regulation on selling a sandwich to the public than there is to building unprecedented AGI level technology and releasing it to the general public. There is no oversight. There's no general processes here as are controls or like, you know, stakeholders or something.

It's nothing. It's just these private companies, and it's a small number of private companies, to be clear about this. It's basically currently mostly just deep mind, anthropic, and open AI, and a few others who are, like, trying to catch up that are really pushing forward to these high-end AGI-level, dangerous technologies and are racing completely out of control.

So how would you expect this to go well at this current pace? It's like we're in the worst possible scenario we could be in, kind of. I just want to pin down the category that we were talking about, I interrupted you and so want to make sure that we clearly identify them, the categories of how AI progress would go wrong. One of them is misalignment. This is the LEASER conversation, the paperclip maximizer conversation. We create this super intelligent AI, and then we can't figure out how to harness its goals and align them with humans, and so it accidentally turns us into paper clips. That's one category. Another category that you defined was

misuse. We somehow do figure out the AI alignment problem, but we just abuse AIs to kill us all. So one superpower fights with another superpower. One of them has AI. Maybe both of them have AI and then we all die because we're using AI to have misaligned human goals. So that's misuse. Another one is accidents. We have super powerful AI. It accidentally does something that we don't like and we're inside of that blast radius. That's a third category. Are those all of them or are there others that are worth unpacking? Yeah. I mean, I think you can even fold accidents into misalignment in a sense is that an align, a truly aligned system. If you tell it to shoot you in the foot,

it will say, no, that's not what you intend it and, you know, bring you flowers or something. If you have a truly, it's like, there's like, maybe like, maybe four is like, you know, like a good hierarchy, maybe four categories. Technology is so dangerous that no one should do it, you know? Just like there is nothing, like you just turn it on, it blows up everything.

Then there's like technology, which is controllable if you're really careful. If you're very sensible, if you're very sensible, if you're very careful, it's fine. Then there is technology that is safe for general use, except if you misuse it, except if you're specifically trying to do something bad. And then there is technology that is good,

no matter who uses it. So you could give it to the most evil sociopath in the world, and it's fine. Okay. I can see AI fitting into all of those categories, including the most dangerous ones. Exactly. I think we will start at one,

and then we can develop technology to move to two, and then we can develop technology to move to three, and then we can develop technology to move to three, And then we can develop technology to move to four. But by default, we get the category one AI. By default, we elevate AI as it progresses, finds itself inside of the category of technology you almost never, ever want to even open up at all.

Yes. This is what I expect. Like, the shortest path to an AGI gets you this type of AGI. A system, which is misaligned, you know, papercliff maximizer has some random values. It's very intelligent, very capable. It's very deceptive. And if you just turn it on,

it doesn't matter who turns it on. It doesn't matter if this is, you know, the USA or China. It doesn't matter if it's open AI or inthropic. It doesn't matter who does it. It just blows up everything. There is no, it doesn't matter. You said the shortest path to get to a super intelligent AI. Can you unpack why you emphasize the word short there?

What is implied under a longer path to an AI? What does that mean? So importantly, this is not a rule of nature. There is no law of physics which states it is impossible to have a safe AI that does good things for you. This is completely allowed by the law. of physics and computer science, we just don't know how to do it. And this is a very narrow target. Each of these, you know, categories one to four are like a narrower and narrower and narrower target. You need to know more and more about control, about intelligence, about, you know, safe

practices, about human psychology, about, you know, values and game theory and whatever, to narrow down on these more and more complex systems. So by default, if you just want to think, which is just smart, it's just powerful, it just succeeds at goals. Well, larger models. You know, just throw compute at it, man. You know, just, you know, continue doing what we're doing right now. Basically, all the research currently being done at AI companies is of the kind which gets you to type 1. There is a very little research that goes into getting you to type 2, 3, or 4.

There's a very great quote, actually, that I read about the other day. It's from Wilbur Wright. So this is one of the brothers who built the first airplane. And what he said was, is that when once the machine is on, proper control under all conditions, the motor problem will be quickly solved. A failure of a motor will then mean simply a slow descent in safe landing instead of a disastrous fall. So this is the man himself built an airplane who realized the first step to building a good airplane was to solve the safety problem. It was to build a safe glider that if something goes wrong,

it carefully slowly descends instead of killing the pilot. Because before this, there was a lot of attempts at flying machines, and they always killed the pilot so they couldn't develop them. So he recognized that we had to first build a safe glider, and then we can worry about the motor part. And he, even though it's almost dismissive of the motor, he was like, eh, you know, we'll figure our motors. It's not that big of a deal. And so basically all of AI currently works on engines. They work on motors. There's very few people working on gliders.

Basically, everyone is working on building bigger and bigger turbojet engines. And, you know, just like as fast as possible, quick as possible. And if you get a bigger and bigger turbojet engine, without improving your glider design, without improving your, wing control design without doing the necessary experience for this. By default, you get a type 1 engine. You get an engine that just explodes. It just like, you know, zooms off into space and, you know, just like does something stupid. And, you know, if it's like an aircraft engine, the risk, the blast radius is contained as before. But AI is not just an airplane engine. Right. Yeah. The AI engine is focused on elevating humanity to as high of an elevation as possible. And then all of the

AI safety people are like, hey, we also need to make sure that we have a spaceship that is containing us that can take us back down in the inevitable case that eventually that engine runs out. Connor, I'm hoping I can just get your lay of the land of the state of the AI alignment conversation because as we've been saying, as you've been talking about, AI progress seems to be moving really fast. It is now June, 23, and it kind of seems that we have to timestamp these podcasts because if we were doing this podcast in just May or April of the same year, it would be a slightly different conversation about what is the state of the AI alignment conversation? Last we checked, there was this letter signed by many world leaders and AI experts asking for a pause on AI progress beyond Chad 2B4.

I'm wondering if you can just update us on the last few months in the mainstreaming of the alignment problem. Like, is there more reasons to be optimistic? Are people bearing their heads deeper in the sand? Where is the world with regards to AI alignment and AI safety? Yeah. Man, has the world changed. The world's changed insanely over the last six to 12 months. to say it lightly. AI alignment has gone to a large degree mainstream. I just earlier today was talking

to a member of parliament who didn't even have a smartphone, didn't know what an AI was, but he heard about this AI thing and he wanted me to tell him about it. And he got quite furious when I explained to him some of those risks and he's like, no one taking this seriously? What? This is outrageous. Like, oh, of course we have to do something about this. So it's quite fun. For me, it's also been a very enlightening experience. So when I first got into this field, you know, I came in from like a pretty classic, you know, kind of like less wrong, you know, Eliezer adjacent kind of viewpoint. Very technical, very nerdy, very philosophical,

perspective on things. And there's a lot of weird social memes in like that sphere around politics that like politics is bad. You should never do it. Don't talk to the public. Don't talk to the government. They're all crazy. Can't talk to them.

And I feel it's completely gaslit because that's just not true. They're going to be wrong. Politics is hard. You know, politicians have their incentives, blah, blah, blah. All these things are true. but this is things you can do. You can talk to them. A thing that's just been incredibly positively shocking to me

is when I talk to normal people who don't work in tech, they really get it. I'm so used to talking to people in tech, and they just totally dismiss these things. You're like, no, yeah, I can never do anything bad, you know, blah, blah, blah, I can't hear you. And I talk to normal people and I explained to them, hey, you know, these things are becoming more and more intelligent

and they can do more and more things, and we don't know how to control them. And they're like, holy shit, well, that's, what? That's terrible. Of course this is going to go wrong. What do you mean? And then I'll repeat an argument that, like, some ultimat makes or something. I'm like, that's not convincing at all.

Like, what do you mean? He doesn't understand how his system works. It's all black boxes. This is madness. And this is the correct reaction. The correct reaction is, this is madness. This is complete and utter ludiccy. Like, let me be blunt here.

Even if you, dear listener, don't totally buy the existential risk thing. You know, maybe you don't buy it. It's fine. But it is the case that the leaders of all the top labs, Anthropic, Deep Mind, Open, Eric, have been on the record. saying clearly that they do think that there is a realistic possibility that these technologies will kill literally everybody and they're doing it anyways. Even if you disagree about the risk being real, I'm quite shocked that someone would like admit that they believe this, you know, state that

they do think this, and then also that they are willing to do it without the necessary safety and precautions. And there are arguments and man are there arguments and we can get into those arguments, the counter arguments. Why actually this is fine? I don't find them convincing, obviously. And we live in this weird twilight world. We're on the one sand. I don't think these people are malicious, to be clear, or like, they're lying per se. I think they're being inconsistent. You know, Sam Altman will often go on the record and say, oh, you know, he thinks AI is the biggest risk to humanity. Cool, great. Thank you, Sam. That's really great. That's like, I'm not being facetious. This is actually fantastic that he has the honor and the bravery to say this publicly. As someone,

you know, it was ultimately a businessman, but still willing to go onto the record with this. That is respectable and deserves credit. But then he keeps racing. There is this incredibly funny interaction on Twitter, where Jan Laika, the head of safety and alignment in OpenAi, tweeted something like, you know, maybe we should be careful and, like, you know, slow down a bit before we integrate all this AI technology into, you know, all facets of our life. And then six days later, Sam Altman tweets about chat GPT plugins, you know, plug in chat GPT plugins, you know, plug in chat GPT. into whatever you want. And I'm like, man, wow. Like, if this was in a movie,

like, I could imagine just like the cut and then, like, you know, the laugh track playing. I'm like, this is truly shocking. And this is a consistent feature, is that this is something I've been pushing on a lot in the current discourse. Is that a lot of the discourse right now is people are starting to wake up to it

and they're being confused. They're confused about like, well, is this risk real? Which is a good thing to be confused about. This is a fair thing to be confused about. And there's other things about, like, this is a very funny thing that I see a lot, for example, on Twitter. People see, like, say, Sam Altman call for regulation. And they'll be like, wait, this is suss. If he wants to be regulated so bad and he thinks this risk is so big, why is he doing this?

Just stop. Just don't race. If you think this is an extra interest, just stop. And my honest opinion, yeah, that's pretty suss. Like, some people criticize me by proxy. They'll be like, Connor, you're one of these dumer people. but you do more people. If you take you so seriously, why don't you start?

And like, well, first of all, I don't race. And yeah, that's a really good question. This is a question that I have for the head of all these labs. Why? Like, stop. I'm happy to go into the arguments that I expect their straw man version to, you know, in case you have any comments. Yeah, maybe we can actually just define race, the whole like race condition side of things.

A decent part of the Bancois audience will be familiar with Moloch and Moloch traps. But it's been a while since we've had a Moloch episode. So maybe you can kind of talk about, just like what this term race means and why we are in a race trap. Yeah. So race conditions or, you know, it's kind of like in the sense of like a race to the bottom is that the idea is you and other people don't want to go somewhere. You don't want a certain technology to exist, but other people are heading towards it.

You think you're better than these people. You're more responsible, nicer, whatever. And so you think, well, you have to get there first before they get to it. And so you start, you know, trying to get there as fast as possible. They notice that now you're trying to get there as fast as possible. And then they're like, well, shit, I don't want that guy to get it because I'm the good guy. And then they start going as fast as possible. So you get this game theoretic problem where now it's a race to the bottom.

This is not unique to AI. This happens, for example, with like, safety regulation. The reason we have regulations for safety standards on the government side, instead of letting the market regulate itself is by default, if you have a market and no regulation, there is an incentive to cut as much safety measures as you can possibly get away with. You want to do as little safety as possible. If you don't get in trouble for your employees getting killed, well, then just let them die.

You know, it's like not that big of a deal. And this is exactly what happened during the Industrial Revolution, is that the pricing of risk and of dangers and these kind of things can be, unlike an economic scale, it's very easy to be mispriced from what we as a culture might want, as a society might want. And this is not supposed to be a statement that regulation is good or bad. Again, it's not about good and bad. This is how we started

this podcast. Not saying regulation is good or regulation is bad. It's never that simple. It's the question of what do we want? What do we as a society want? And how can we get that? And if we don't like where we are currently, what can we do to move to something that we like? So with the race, what we're seeing here is companies undercutting themselves on safety and speed and timelines. Sometimes you'd call it burning our runway. So humanity has a runway until AGI arrives. It will arrive sooner or later. If before that time we don't have a safe glider, we die, and they're burning the runway, they're making it shorter by pushing forward this technology, by investing more into it, by building bigger engines. There's some engine size that when you get to that size, it blows up

everybody unless you have a safe spaceship built around it. And currently, we're not building a spaceship. We're just building bigger engines because, well, I don't want the other guy to build engines. He's really unsafe. He doesn't take it seriously. Or what if China gets it? You know, what if some other country against. No, America, number one, we have to get it first. And I understand these arguments. They're wrong. And I'm happy to go into some of the details about why they're misleading. But it doesn't matter. There's nothing to win. The simple counter argument is there's nothing to win. You just lose. This would work if we were talking about type 2 AIs. So AIs that are safe if you're careful. If we get to this, and I think this is one of the things that Sam Alpin would claim, he would claim,

oh, no, no, we're going to build a type 2 AI. Don't worry. We're not going to do those type 1 ones. Don't worry about that. and I would strongly disagree. I don't think that that is supported by our level of scientific progress on the alignment problem whatsoever. But if we were super on track to get like a type 2, 3, or 4 AI, then I'm like, okay, fine, you know, fair enough. That's a reasonable thing to believe. I would still be more careful than that because I would never be that certain. I wouldn't want to risk it. Even if I believe we're probably going to get a type 2 or 3, I wouldn't want to risk all of humanity on it.

I would like take some time. But this is kind of the situation we're in now is that, of course, course, the people who are the most optimistic will be the ones who end up in the position to push this kind of race forward. It's not a coincidence. And they're going to be the ones who get billions of dollars of investment. This is not a coincidence. It's not a coincidence that, you know, Sam Altman is the head of opening high and not Eliezer. Eliezer is not the kind of guy to lead in opening eye. Of course not. That's not what he would do. And so that's exactly the kind of scenario where we're in right now. Who would you say are the

main players in this race. Sam Altman, certainly of Open AI, China as a techno country as a whole, is maybe in the race? Totally disagree. Who's in the race? Just to say a word on China, I don't want to go too much into this, but like there is a meme that exists, which I would like to dispel, of like, China is this massive rival here. They're going to, you know, overtake us. They're going to build AGI or whatever. I think this is really ludicrous for anyone who actually knows about China. What the Chinese Communist Party wants is stability. And to stay in power more of anything. Do you think they want a crazy uncontrollable technology that could topple governments? No. And China has been very clear multiple times that they're willing

to take massive economic burdens to censor and stamp out their own tech industry. They've done this multiple times. Of all the countries, that I think is like most likely to regulate AGI away, it's China. This is completely in their interest. It is not in the Chinese Communist Party's interest whatsoever for AGI to exist. It is completely counterintuitive. It is completely to their incentives. And also they're very far behind technologically. But this is just like a comment on I'd just say. I think a lot of people are perhaps perversely benefiting from holding up China as a boogeyman. But the truth is basically 100% of the risk comes exclusively from the United States of America. There is no, in my opinion, appreciable risk from non-Western

countries whatsoever. Maybe this will change in 20 years or 50 years or something. But at the current point in time, all these models, all these things are being built on U.S. soil. basically exclusively. Even model, like there's a recent model called Falcon, which was created by the UAE, but it was trained in U.S. data centers. This was done on U.S. hardware on U.S. data centers, as far as I'm aware, U.S. programmers. So, like, this is very much an American and partially U.K. European issue. But that being said, so the main player is geopolitically, it's the U.S.A. I mean, to some degree, the U.K. and the EU a little bit as well, but not deeply so, and not trivially so. And within the U.S.A, it's generally a small number of private companies.

This is Google slash DeepLind, OpenAI slash Microsoft, an Anthropic. These are the main ones. There are other people who are trying to catch up. There's a bunch of other startups trying to, you know, race forward to that kind of technology, but they're all very, very far behind. And most of the other groups in these field

are not nearly as ideologically devoted to superintelligence as these three companies are. The founders of all of these companies have been very clear and publicly so that their interest is godlike superintelligence. They want to build systems that can reshape all of humanity, that can upload us all into the cloud, that can turn the whole world into nanobots.

This is not me saying this. This is like what actually these people have been quite publicly about. This is what they're trying to build. They're not trying to build a better chapup. They're not trying to make the most shareholder return. What they're trying to do is to build godlike superintelligence and then unleash it upon the world, which they believe will bring utopia.

So this is, you know, people like Sam Altman, people like Demis Osabas, Dario Amadei from Anthropic. And again, I would like to state, I'm not saying these are bad people. I really don't want to say this. I've talked to all of these people and they're, for the most part, really pretty great. Like, they're really pretty nice and very, very smart and hardworking and mostly trying to do what they think is right, some more than others. But it should be clear what they are and what they are as transhumanists. They have an ideological interest in doing this. they have strong incentives to be very optimistic and not think too hard about the dangers,

or to find excuses for why, well, it's a race. You know, my hands are tied. Nothing I can do about it. And I understand. I don't want to criticize and, like, say, like, these are, like, evil people. There are evil people involved in the system, maybe. But for the most part, these are people of certain beliefs, who have certain incentives, who are trying to make the world a better place.

but I, from my perspective, find what they're doing unimaginably reckless and cannot continue. Yeah, I'd like to unpack a little bit more around that just so I can understand where the disposition of all of these founders are. You said that the tech industry is more or less in denial about this, at least in comparison to the outside world. And you've also said that people like Sam Altman and maybe the other founders as well have said that, yes, AI is perhaps the most, the largest existential threat that we have. and yet they continue. And so I'm wondering if you can just like diagnose that. Like is Sam Altman and all these other founders, they're just, oh, we're in a race. My hands are tied.

I guess I'll just keep on racing until some sort of external force stops me. Like, how would you actually define why the disposition of these people are the way that they are? So truth is, of course, I do not know. And this is basically psychoanalysis. I could do psychoanalysis. I know some of these people, you know, at least a little bit. I have studied pretty extensively what they do and why they do it. I've read everything they've written and so on.

I have, you know, guesses about their internal lives. The truth is, I don't know their internal lives. You know, maybe some of them are total evil, you know, scheming bond villains. I don't think so, but like maybe, I don't know. Maybe some of them have really good reasons to believe that AI is super duper, super super super safe. I haven't gotten those reasons out of them, even after talking to all of them many times and pressuring them on this. So I don't know what their emotional motivation is,

but what I can describe is their stated opinions, their stated beliefs, and their stated actions. My usual thing, I would say, and I recommend this to everyone. I think this is a really, really, really important skill that a lot of people neglect is watch the hands, not the mouth. These people all say very, very nice things that make you feel very, very safe,

and then their hands do something very different. This is a very consistent feature I found across dealing with, I mean, not just these, like generally powerful, people, you know, politicians, CEOs, you know, billionaires, just watch the hands, not the mouth. If you're very smart, it's very easy to come up with explanations, why it is actually good to do the thing that you want to do anyways. And truth of the matter is you can go back to archives sometimes from like the 90s, from some of these people talking openly on, like, email lists or on their blogs about how they want to build AI, how they want to bring in the

transhumanist future. They want to, you know, create this beautiful, immortal world of transhumanist cyborgs or uploads or whatever, and they want to do it as fast as possible. They want to save all this. Even Eleazar was, at least, in this camp and was, you know, in the 90s and the 2000s was accelerationalist. He thought that building AGII was the best thing we could do and we should do it as fast as possible. He changed his mind, which is fantastic and speaks to his character and his ability to think about things rationally and reasonably. That Eliezer did change his mind when he was quite young. I think he was like 21 or something. but for other people

I feel like I think a lot of them have absorbed part of the arguments but like watch the hands, not the mouth I think a truly damning example of this is the Effective Altruist movement which is deeply ingrained with basically all three of these organizations

to various degrees and look again, effective altruists for the most part are really well-meaning good-hearted, smart people trying to do the right thing are there bad apples? Of course there are. No question about it. Are there weird culty dynamics? Of course there are. They have

exist in any large movement of this kind. Are there, you know, weird untowls? Absolutely. But most individuals are good people. My truth of the matter is that the effective altruists and people like Eliezer are speaking out against these corporations for racing. Eliezer was one of the people who helped Dario found DeepMind. He was one of the people who introduced Dario and Shane Legg to Peter Thiel for initial funding. I don't know how much it helped them, but it was something I was thinking about at the time, how to, you know, push that on. Open AI was founded by people, many of which were Effective Altruist or Effective Altruist adjacent. And Open Philanthropy, the largest funder of Effective Altruist goals, gave Open AI a very large early grant.

It is, I think, the second largest grant they've ever made to an AI organization, I think it was $30 million roughly, was given to Open AI in the early days to we get a safe AGI lab. In fact, from what I hear, it was Dario Amode, later CEO of Anthropic, who suggested to Elon Musk to create Open AI as a counterlab to deep mind to do safety. And as you probably know, like early opening eye was also very open source focused. They changed their tune about that. And then later, after Sam got involved a lot of drama there about Elon and Sam and such that I'm not privy to, I don't even know exactly what happened, but some drama occurred. Later on, Dario led the project for GPT2 and then the project for GPT3.

He was the one pushing scaling laws. He was the one making these systems stronger and more powerful and pushing forward on this axis. And then after GP3, citing security concerns and disagreements with Sam and the direction of opening eye, which seems completely plausible to me and have a reason to doubt this per se. He and a bunch of other people taught people are the GP3 authors and people left to found Anthropic. And what did they do at Anthropic? They built larger models and other racing. And the recent pitch deck was leaked from Anthropic.

They are talking about how they are going to build a 10x more powerful model than GPT4. And to this day, you'll have people in the AI safety community vigorously defend Anthropic or even opening eye or deep mind as the safety oriented. No, this is good actually. You know, there are friends. But, like, you know, there's all these weird incentives going on there. Like the president of Anthropic, Daniela, is Dario's sister, is married to Holden Karnowski, who was the CEO of Open Philanthropy. And this is a great example of Watch the Hands. what has happened is that these people stated very clearly that they think AI risk is real,

this is a huge problem, et cetera, et cetera. But what has happened, what has he actually occurred is that they have accelerated the race like no one else. And I'm talking about this publicly for one of the first times, I think, because I think this is so important to make this clear what is going on here. I've gotten quite shunned by the EA community for a lot of these angry comments. But this is a common feature. This is the same thing with politicians. I've talked to many politicians.

they come to me and I'd say them, oh, this is dangerous. And I said, well, Open AI told me that they tested their model and it was safe. And I'm like, man, okay, where do we start with this? So, like, none of this is surprising. It's corporations doing corporation things. It's Microsoft lobbyists doing their Microsoft lobbying thing.

I'm not mad. I feel like this is what any, I think, reasonable political analyst would have predicted about how the discourse on AI safety would go before I say it. If I talked to, like, an old school, you know, like, I don't know, like, environmental activist who was there for like the oil lobbying stuff. And I asked them,

how do you predict the current state of AI discourse is? He would probably make the same predictions for where it currently is where like, yeah, people are not on board with like oil pollution. They think it's bad. But of course, companies have lawyers. They have lobbyists and they have a lot of great excuses and they have a lot of great comments about like, oh, we should do self-regulation. Oh, let's just know the oil companies, you know, regulate themselves. That's that the oil company to set the safety standards for their oil, you know, extraction, which is unironically what people are suggesting. We have people at Open AI pushing for e-vows, you know, for safety evils, which is a great incentive. But this obviously has to be nonpartisan. This obviously has to be done by people

that are, you know, neutral parties. It's currently not the case. You know, a lot of the people doing the evaluations are people who either, you know, work at one of these organizations or used to work at one of these organizations. And this is just kind of, it's kind of silly. If you haven't experienced the superpowers that a smart contract wallet gives you, check out Ambire. Ambire works with all the EVM chains that are out there. The layer two is like Arbitrum, optimism and polygon, but also the non-etherium chains like Avalanche and Phantom. Because of the power of smart contract wallets, Ambire lets you pay for gas and stable coins, meaning you'll never have to spend your precious ETH again. The web app has numerous Fiat onrams to make it easy to dump your

fiat for crypto. And if you like self-custody, but you use, still want training wheels, you can recover a lost Ambire wallet using an email and password, but without giving the Ambire team any control over your funds. Check it out atambire.com for the web app experience. But also, the Ambire mobile wallet is coming soon for both iOS and Android. And if you want to be a beta tester, you could sign up at ambire.com slash app. And since you stayed to the very end of this ad read, you should know that Ambire is air dropping its wallet token to early users for simply just using the wallet. So if you want to get started with Ambire, all of the the links that you need are in the show notes. Enter Mantle, an entire ecosystem dedicated to the

adoption of decentralized token-governed technologies. Mantle formerly known as BitDAO is the first Dow-led Web3 ecosystem, all built on top of Mantle's first core product, the Mantle Network, a brand new high-performance Ethereum Layer 2 built differently from the other layer 2 that you may be familiar with. Mantle asked the question, how would you build a layer 2 if you had a technology of 2023? Mantle Network is a modular layer 2, built using the OP stack, but uses eigenlayers data availability solution instead of the expensive Ethereum layer 1. Not only does this reduce Mantle network's gas fees by 80% compared to other layer 2s, but it also reduces gas fee volatility, providing a more stable foundation for Mantle's applications.

The Mantle treasury is one of the biggest Dow-owned treasuries in Defy, which is seeding an ecosystem of projects from all around the Web3 space for Mantle. Mantle already has sub-communities from around Web3 onboarded to help the growth of Mantle, like Game 7 for Web3 gaming or Edu Dow for in the world of D-Sai and buy-bit for TBL and liquidity and on-rats. So if you want to build on the Mantle network, Mantle is offering a grants program that provides milestone-based funding to promising projects that help expand, secure, and decentralized mantle. If you want to get started working with the first Dow-led layer-2 ecosystem, check out mantle at mantle.xy-Z and follow them on Twitter at ZeroX Mantle. You know Uniswap. It's the world's

largest decentralized exchange, with over $1.4 trillion dollars in trading volume. You know this, because we talk about it endlessly on bank lists. It's Uniswap, but Uniswop is becoming so much more. Uniswap Labs just released the Uniswop mobile wallet for iOS, the newest, easiest way to trade tokens on the go. With the Uniswap wallet, you can easily create or import a new wallet, buy crypto on any available exchange with your debit card, with extremely low Fiat on-Ramp fees,

and you can seamlessly swap on main net, polygon, arbitram, and optimism. On the Uniswap mobile wallet, you can store and display your beautiful NFTs, and you can also explore Web3 with the in-app search features market leaderboards, and price charts, or use wallet connect to connect to any Web3 application. So you can now go directly to Defy with the Uniswop mobile wallet, safe, simple custody from the most trusted team in Defy. Download the Uniswap wallet today on iOS. There is a link in the show notes.

I'm kind of just getting the intuition that if we re-rolled the dice on this universe and maybe we plopped out Sam Altman and Open AI, something else would be there instead. And so, like, we could go down the rabbit hole of just like, hey, let's talk about the personality. disposition of each of these founders. But I think it's really just about the nature of the problem itself, that it doesn't really matter. Like, take out Google, take out Anthropic, take out OpenAI, and you'll just find three other companies that will take their place in the race. Like, the race conditions are race conditions that expand beyond the current set of players. Do you agree with this intuition? I'm more skeptical about that than I think you are. I think it is true in the limit. Like, eventually someone will figure out AGI. But I think you're under as a.m.,

estimating how much ideology is actually at work here. GPD4 was not a coldly calculated, you know, business decision. It was an ideological decision. GPD4 costs like $100 million or billion dollars to build. It's not meant to make that much revenue. It's like extremely expensive, extremely risky. It could have blown up at any time. You know, this is not what a Goldman Sachs does. No Goldman Sachs is going to build a GPD3 or GPD4. You know, no coldly rational organization like this is going to take risks like this. Right. You're saying the rational economic actor would not have chosen to produce Chad GPT for. Yeah. After they have chat GPT, sure, they might have liked it, but no rational actor

would have, at least not now, you know, maybe when the cost comes down, you know, 10 more years of progress, maybe then. So I do think eventually would have happened, but we do have people who are accelerating. It's not just that like, oh, this is the one time where it happens. No, it could have happened in 10 years. It probably could have happened a few years earlier, but like not many years earlier. You know, maybe other people would have arisen. It's possible. I'm not saying it's impossible. Like, you know, we can't re-roll history, really. But I don't like these arguments too much about like, oh, it's just incentive.

It's just a race. Also because one of my maybe controversial beliefs is that I think that people have way more control over reality than I think they do. I think that actually the world is way more plastic and way more controllable than people think it is. I think that individuals with high agency can get a lot more done than people think they can. I think that individual actors matter a lot. individual great people, you know, or politicians or activists or whatever, can make a huge difference, actually. There's a saying, I forgot who was from. I was like, never underestimate the

ability of a small group of dedicated people to change history. In fact, it's the only thing that ever has. Okay. What I hear you saying is that because of the commitment to building ChatsyPD4, even though it was economically just non-rational, you're saying, therefore, it is ideological, as in therefore these people are motivated by something else, an ideology, maybe some sort of glory, I don't know, maybe the idea of just like creating the AGI is so attractive that they want to be the ones. And so that is a little bit more of a parnicious problem because that is like harder to tinker around with external incentives, correct? Yeah. And to be clear, it's not just that. It's also that they literally state this on their own personal blocks. Right.

Like I am not doing just pure psychoanalysis here. Like this is literal actual statements you can actually read written by these people that will. be confirmed by their friends and so on. Like, not all of it, but like these are not wild speculations that I'm pulling out of nowhere. You know, maybe they've changed their minds. You know, maybe there's some subtlety to it. I'm not dismissing these possibilities by any degrees here. But I'm saying ignoring that there is an ideological component, I think, is not correct. I think it is very reasonable there is an ideological or glory or a aesthetic preference in some sense.

There is a, in retrospect, really rather chilling interview with Geoffrey Hinton, one of the, you know, godfathers of a from, I think, like, 2015 or 20, I don't remember. And the article ends on him asking, like, well, you know, if you think these things could be dangerous, like, why would you do this? And Hinton, basic answers, well, you know, sometimes the lure of discovery is just so sweet you can't resist. And this is very grimly prophetic in the sense that now Geoffrey Hinton has disavowed his life's work and has now come out in favor of that actually this is an existential risk. This could kill everybody. I was in a lecture with him that he gave in Cambridge a few weeks ago, and it was a very lovely lecture.

But it basically ended on this note of just like, well, it turns out, yeah, probably that this is, yeah, this is going to kill everybody. And I really don't know what to do about it. Yeah, damn. It was just like really this weird note. Like even Yashwabangio, one of the other like touring award winners, so one of the most senior, most respected people in the entire field of AI, has said that he feels lost after having come to realize,

It's just how it's regret his life's work. Imagine that. Imagine one of the most senior professors, one of the most senior scientists to exist in this field, who has built this field saying that he regrets his life work. Like this is unprecedented. This is unprecedented basically in history. I mean, you know, there's some cases, but this is a truly extraordinary scenario for something to be this clear. So I think a lot of it is just curiosity, beauty, fun, glory, etc. and you're correct. This makes purely rational incentives harder to control. I think this is a part of it. If we try to model this as a purely capitalist problem, this is a purely money problem, I think we would be not get the correct solution. I think this is, again, where we just have to be pragmatic. I'm not being judgmental. I'm just being like, okay, let's be pragmatic. What can we do? Like, what do things need to come? And the truth is just this is where government has to step in. Like at this point, like these people are not going to stop. They've had many opportunities too.

and they will not do so, government just has to make them. I was going to ask you what are the next steps here because I resonated with the quote that you said where the man feels lost because at the end of any sort of AI podcast, and again, our AI podcasts are primarily alignment and safety podcasts. I feel lost. I feel a little bit hopeless. What you just suggested is like the easiest thing, the lowest hanging fruit that you see possible, which is that governments have to step in.

What does that look like to you? How do we get that process started? Like, what are your ideas here? Yeah. And so again, I want to call back to the beginning of this podcast and just be like, this is not about government good or bad.

I don't want to talk about, you know, you're from a crypto background. I assume many of your viewers are quite skeptical of the government. I think there's very good reasons to be very skeptical about many things. General distaste, yeah. Totally understood.

I run a business. Nothing makes you a libertarian faster than founding a business. You know, and noticing all the red tape you have to go through. Like, I understand. But this is not about good or bad. It's not about ethics judgment.

This is not about, You know, did they mess up housing policy? Spoiler, yes. But it's way more about, okay, let's be practical here. What are the options? What can we do? So this is the kind of thing that government exists for. If oil companies are polluting, they're just, you know, putting poisonous to the river or whatever,

and you ask them nicely, they don't care and they keep doing it. You know, maybe they cite some reasons about, you know, acceleration or, like, races or whatever. Then you send police officers to make them stop, you know, like, you give them a polite knock and a polite letter. and you tell them, look, knock this off, you're in deep trouble. So this is going to have to happen. This is not a solution to be clear. I want to be very clear.

None of this is a solution. This buys you time. This doesn't solve the whole problem of technology and, you know, the future of mankind. No one would claim that, you know, us, you know, shutting down some of these dangerous experiments is going to solve us. But it buys us time. We can talk about this in a second. But the longer story, of course, is that if we really want to solve this,

If you want a good future for humanity, I'm going to have to do a lot more than this. I'm going to have to solve a lot of technical problems and a lot of political problems and a lot of philosophical problems if we want to get there. And it's unfortunately not optional. But the first most clear step from my perspective is that, to be blunt, I don't currently see any good timeline in which there is not a pause. If we don't buy more time, if we just continue accelerating, if we're just pushing as hard as possible, as fast as possible with all the resources our economy can muster, we're not going to make it. We're going to type 1 system is going to pop out. And that's just going to be it. And, you know, we're not even going to make it to a type 2 system.

If we make it to a type 2 system, meta is going to immediately make it open source and then we die that way. But that aside, I don't even think we're going to get to type 2 systems. So the government needs to step in and slow these things down. Luckily, there are actually very good levers that the government does have access to in current legal frameworks that can be used with quite small impacts on the wider economy. There is, luckily we are in a scenario where really 99% of AI is completely fine. You know, love, you know, medical research, you know, AI or, you know, using particle physics or, you know, doing like picture generation stuff, even so that has some copyright issues.

Okay, let's not open that can of worms. But like 99% of AI is great. It's progress, it's technology. It will provide great benefits to society, you know. Fantastic. There will be ups and downsides, but a blast radius is contained. 99% of AI things has a small blast radius, not saying zero, but the kinds of blast radiuses that our society knows how to deal with, like we can handle. The internet had a huge blast radius. We're still here. We can handle the internet. So we can handle 99% of AI just fine. There's only really a teeny tiny percentage of these like hard, huge, you know, GPT4, GPD4 plus, you know, large language models, general purpose reasoning, agents, RL, you know, like far.

stuff that where basically all of the risk comes from, that all of the existential risk comes from. Not that there's no risk from other things, there are other risk, but it's not existential. The existential risk is really focused on this teeny, teeny tiny subset of like, you know, three maybe companies in the United States. That's kind of it. And the government totally can just stop that. The United States government can just unilaterally tomorrow put out an executive order and just put it into this. It's that easy. And this exact, suggestion I would give to the U.S. government is to simply ban or require strict oversight and regulation of any individual training run, which takes more than 10 to the power of 24

floating point operations. This is a unit of measurement for the amount of computing power used. And luckily, this is very easy to measure. We know GPUs have this and this much computing power, run for this and this long. You can calculate how much computing power is going to run. This is very easy to measure. It's very clear. There's very full. few companies that have access to computers this big. They are known to the U.S. government. They are easily tracked down. And they are, you know, settled in the U.S. These are, you know, large cloud providers, Microsoft, Google, Amazon, et cetera. And they will comply. Of course they will. You know, they're not going to risk their entire multi-billion dollar business for these,

you know, few eccentric clients. So if the U.S. government tells Amazon no more large training runs, no more things above 10 to the power 24, I expect Amazon will just instantly comply. And that seems actually to be kind of a long-term solution, my limited understanding of AI, constraining how powerful our computation can be, constrains how powerful the AIs can get universally across the board. And so while that law would be in effect, we have as much time as we need. Is that correct? That's my intuition.

Fortunately, no. I wish, though. Yeah. Oh, yeah, indeed. The truth is that we are not on exponential. We're on a double exponential. There's two exponentials. Is it hardware plus software?

Yes, that's correct. We are both on exponential and hardware and an exponential in software. If we can cap the exponential on hardware, again, you know, if people want access to their small AIs at home, they want to do stable diffusion, they want completely fine, you know, maybe the government gets involved for copyright reasons or something, but like, from my essential risk perspective, I'm quite fine with that. But not these frontier runs for calibration. 10 to the 23 is roughly how much to estimate GPT3.

used. GPD4 probably 10 to the 24 or 10 to the 25. And we expect that, you know, GPD 5 will probably take another, you know, factor of 10 on top of them. And so if a cap at 10 to the 24, I personally am confident that algorithms will improve that you could still build an existential risk in that regard. I actually expect as possible 10 to the power of 23. It's probably powerful, you know, capable of even less than that. It's probably possible. I expect to do that. You need algorithms that we don't currently have. Like, I don't think currently you can make GPT3 existentially dangerous. I think it's currently

possible, but I'm not confident that's going to be true in 10 years, or in 20, or in 30. We will develop better algorithms for bootstrapping, for more sample efficient learning, for better RL planning things for bootstrapping, and there'll be like all these things that we will discover and that we will continue to improve.

There's lots of low-hanging fruit here still. But this does buy us time. This does buy us a lot of time. Right. It cuts down the acceleration by half. Exactly. It makes a huge difference. This is like, it's not going to save us, makes a huge difference. It gives us the time to do safety research, to build gliders, to build rockets. It gives us time to, you know, actually study these systems, like actually understand, to integrate them to society, to figure out what is the right regulation? Like, how do we

want to integrate these things into society? Like, Open AI likes to say, it's like, oh, we think the safest thing is to do iterative deployment, and so they can integrate into society, which is code word for, lull, we release it immediately and just, you know, throw it on the market. If Open AI had built a GP3 and then completely stopped, And, you know, and done only state of GPD3 until they fully understood it. They had full safety control of it. And, you know, all society had, like, totally integrated it culturally. And we had, like, regulation that, like, totally handles it.

And we're all comfortable with it. And then they had produced GPD4. Yeah, you know what? Fine. That's fair. Honestly, totally fair. If this was the way our society handled AI, fantastic. I think this is great.

You know, I want technological progress, too. Of course. I just want it to go well. And this is not the shape. of a technological path that goes well. Right. What you're saying is that instead of testing chat GPT3 or 4 inside of a contained environment in the lab

so that we can fully unpack it and understand it, OpenAI is just yeeding it out into the world and using the world as the test bed, not the contained environment as the test bed. It's even worse than that. Never mind the contained environment. They're yeeding out, you know, the one thing and then eating out the second one

before we've even, you know, recover from whiplash from the first one. And so just to really unpack these two exponential curves, we have the hardware curve and the software curve. The hardware curve is the thing that you're saying that we can easily and objectively place a government-enforce cap on that curve

with that specific certain to the power of a number that we can totally measure. And so that seems like a solved problem if we can actually get the governments to do that. Then there's the other equation, which is like, we are still in the early days of these AI models in the first place. And so even if we do cap the hardware,

we still have at least a handful of orders of magnitude's efficiency in the algorithms that have access to that hardware. And so we have like, NVIDIA, the GPU supplier on one side. Maybe there's others in the game as well. I don't know how relevant AMD is in the world of AI. But NVIDIA is the big player that I think most people know. On one side, supplying GPUs into the market.

And then we have the consumers of those GPUs, which are the people racing, right? The open AIs. And so we need to, in order to have a complete containment of the problem, there needs to be control on both sectors. And you're saying the hardware side of things is relatively objective and easy, and we can prescribe action steps for governments to take those steps. And I'm assuming on the software side of things, it's a little bit more gray,

and we don't know how to proceed that well. Is my intuition correct here? Yeah, that's definitely the case. I mean, you know, easy is a strong word on the hardware side. I have talked to many people in policy positions and many of them are like, this is literally impossible. You are insane. Other people, though, have said, oh, no, this is like totally something the government can do, especially using national defense arguments.

Like, you know, we already have a chip ban where we don't export H-100s to China and stuff like this. We already track this. Like, the U.S. government totally has the affordances and has the capabilities and the incentives to do this. This is totally within their wheelhouse. This is not some crazy new thing that has to be invented. but getting the government to do anything is hell. So this is a massive undertaking. That being says, so software is much more complicated.

I don't think we can or should try to ban mathematics. That seems insane. I don't think this is a kind of level of coordination that humanity is capable of. You know, maybe we were some kind of like super smart enlightened aliens. We could maybe like all handshake on it. But like I still don't think humanity can really work that way. But there are things that could be done here. there are actual straightforward, very easy, and clear things that can be done here, actually,

which is, again, doesn't fix everything, but it does buy time. So in particular, the reason I think there are such easy wins here is because things are so bad, because things are so bad. No one's even trying to control these things. There's a lot of low-hanging fruit. So one of the, like, very obvious things, which is not, again, I'm not saying it's easy, but I think this would make a difference is, I think, there should be strict liability for not just model deployers, but also model developers. If you

develop an AI system and someone uses it to commit a crime, you should be charged as if you commit the crime yourself. You should be liable. And strict liability, let's take a moment to unpack that. Strict's liability is when you, the individual, you cannot give your liability to an LLC. Strict's liability is like, yo, you, the individual, that falls back on you. Yes. Just want to define that term before we move on too far. Yes, that's correct. I think this would be the ideal case. Again, I've had a common section, I'm sure, is yelling at me right now. I have heard

from plenty of policy people telling me how this is impossible, but I have also heard from very senior people that this is possible. Basically, the number one step is removing to Section 230 protections from AI systems. This is already being discussed by senators in the Senate. So Section 230 is the

internet neutrality thing where, like, if you host content, which is illegal, but you didn't know about it and you remove it, you're not liable for it. If your users were doing it, you had no hands in it. One of the obvious things is just do not apply Section 230 to AI. Simply, if you develop a model or a system, an AI system, which causes actual criminal harm, you are criminally liable for it personally. This is currently not the case. Currently, there's a, again, look, I love open source, I love my nerd friends, but it's insane, is that some nerd,

somewhere in a university, can develop a voice cloning system, you know, take 15-second audio of anyone, peripheral clone their voice, and use it to, you know, scam their parents, and use it to, you know, scam their parents to, you know, call in bomb threats, you know, whatever, to totally ruin their lives. And the person who developed that model who posted it to GitHub has zero liability. It's not only that they don't have any legal liability. They don't even feel responsible. The people who do this feel no shame. It's not part of open source culture to even, like, feel very responsible for this kind of thing. It's like, oh, you develop a cool thing. You know, throw it out there. It's fine. You know, let the lawyers figure out how society

thing. What's not your problem? Like, I had an actual conversation I have with some tech people. I was talking about how I think there should be straight liability. These kind of thing. And then one of the people sincerely made the point, well, that's impossible. We can't do that. Like, how would they possibly, you ever determine who is at fault? This couldn't be done. And then a friend of mine was like, courts, that is what courts are for. How do you not know what courts are? So there's a huge disconnect between a lot of the tech sector and like how civil society actually functions. Like, the infrastructure of like actual functioning government and societies and courts and so on, we have dealt with things that enable criminal activity in the past. And another great example is

automobiles. Automobiles. There was a huge fight. I think it was in the 70s, if I remember correctly. I might get the dates wrong, bad with remembering exact numbers. There is a huge fight, legal battle where automakers were arguing that they were not responsible for any deaths that occurred due to cars, and including if their cars malfunctioned. And a lot of people push back at that. They said, well, no, if your car malfunctions, if the brakes don't work, if it catches fire or whatever, no, the maker of the car should be liable for that, not, you know, the driver. It's not to his fault that the car caught fire. And this was a massive legal battle. Nowadays, we do hold car companies liable. If you build a car and it explodes and it kills somebody, it's your

responsibility. You build the car. You put the car out there. Of course, you're responsible for this. And of course, there is a spectrum here. There's a large spectrum. I'm not saying I know where on the spectrum is the right thing. You know, should everyone who developed Linux be held personally responsible if someone uses Linux for something bad? I think no, but it's a spectrum. It's not a yes or no.

It's not everyone's whole responsible or no one ever. It's a spectrum. And this is how laws work. This is how courts work. We as a society have to find where on the spectrum are we comfortable. And I'm simply stating, I think we're way on the one. one side, and we should move more towards the middle where there is actual accountability. That same conversation I had as a different person, actually, but same conversation.

When I talked about strict liability, someone called me out, and they're like, okay, Connor, well, you developed open source technology. Do you want to be held accountable? If someone you uses your language models, you developed a Luthori to cause harm? And my answer is, yes, I would like to live in a society where I is a technologist, if I cause harm to civil society if I cause harm to other people, that I am held accountable to this. I think this is good. I think this is the kind of society I would like to live in. I can see a large number of the crypto people, which are 95% of this audience, recoiling at this. And I'm sorry. I want to give an anecdote from the crypto world, which I think is universally accepted as like we are in a legal

battle with the United States Department of Treasury because they deemed, I don't know how familiar are with this, but this smart contract on Ethereum called Tornado Cash, they deemed using that software uploaded to the Ethereum blockchain, using that as an American citizen is illegal. And so I'm personally actually suing Janet Yellen and the Department of Treasury on that. And saying somebody, open source developer, they live in the Netherlands, uploaded this piece of hardware to Ethereum. And I was able to use that as a neutral tool. I was achieving some sort of financial privacy because Ethereum is transparent and open and everyone can see my transactions. and I would like to have created some sort of private version of my wallet that no one used that no one knew about.

So I used tornado cash in order to achieve that end. Simultaneously, so did North Korea. And so my money was right next to North Korea. And so the Department of Treasury deemed this piece of software to be illegal to use by all Americans. And so a few of us in the crypto world banded together to sue the Department of Treasury saying, hey, you cannot make a neutral piece of software illegal just because some bad actors are doing bad things with it, including perhaps funding the development of nuclear weapons. But like when you tell me this about AI, is like how else do we solve this AI alignment problem other than giving

strict liability to the people that create the models that ultimately destroy the world? To me, I'm like, well, I don't have a better solution. And so I don't know what else to do. And so I'm torn at this. I see the crypto code libertarian side of me on one side of things. and then the, hey, let's not die from AI on the other side of things, and I see the conflict here. And I'm sure you have run into these conversations a number of times. Absolutely. And thank you for sharing that. That's a great illustration of, again, coming back to the beginning of the podcast. It's not about good and bad. It's about finding the right thing.

I think you, suing in the US government, is fantastic. Like, I think you're doing a civil duty here. I think you're improving society for all of us by doing this. Whatever comes out of this case, like, someone has to do it. This is how society progresses. This is the mechanisms by which our society finds consensus on what is the correct dial setting. It's not that the government magically knows the correct dial setting. No, this is why in the USA we do sue, and this is great.

This is good. It's a huge pain. I know this must be so much pain for you to do all this shit. That's why I'm like thanking you. Not to the Coin Center lawyers, yeah. I mean, yeah, that's why I'm thanking you and these guys, you know, I don't have a horse in this race. You know, I don't know whether this is good or bad. I don't have a horse in this race.

But I think it's good that this is going to court. I think this is good that someone is holding the position, this is good, and someone is holding the position, this is bad, and this should be fought out. This should be debated. This shouldn't be taken for granted that things are obviously good or obviously bad. Again, beginning of this conversation, things are good or bad. It's about decisions. It's about consequences. I don't know. Maybe the software being banned is net good. Maybe, I don't know. Maybe it's not. Like the way you describe it, I don't know. And I would love for, you know, people who don't understand this technology much better than me, people like you, and, you know, people, you know, the government who hopefully

have, like, a better understanding of national security, something, to battle it out and, like, actually, hopefully find truth. And our courts are not perfect. You know, I'm sure there's going to be a bunch of bullshit. And, like, whatever the verdict is, no one's going to be happy with it. I understand. But this is how our society works. This is how we make progress. This is how we, as a democratic, you know, liberal, you know, judicial rule of law country make progress. And I'm saying that this is painful, but it just has to be done. It's a price of living in a complex society. It's a price of playing in a complex society with complex tradeoffs, with complex technologies, which I think crypto is. I think crypto is an amazing technology that has

incredibly complex benefits and downsides. You know, there is blast radius. You know, FTCS was a huge blast radius. It's a massive blast radius, right? And that doesn't mean crypto is bad. No, like, of course it doesn't. Just because they're scammers doesn't mean it's bad. But it also doesn't mean it's good. It's just a technology. It's neutral. The same way AI is.

It is not good or bad. It's about how do we as a society digest this technology? How do we get as much of the good stuff while preventing as much as the best stuff? And this is simply not easy. This is all I'm saying. I'm not saying I have the perfect solution and solves all the problems forever. Don't worry. Just do this.

I'm saying this is a hard process. This is like doing science. Like in a sense, going to court is like doing science. You're doing like lawmaking science or engineering. You're doing the epistemological labor that needs to get done to actually get to the right kind of civilizational software that we want to be running. Because we don't know what the right software is. We don't know what the right laws are. We don't know what the right liability or the right insurance schemes.

I don't know. It's right. And we have to find out. Someone has to come up with them. And they have to debate against people who come up with alternative things. and we have to try things out and some of them blow up, and then people sue, and then we try another stage and so on. So I think this is a natural part of how a functioning society should work.

And sometimes this is a huge, massive pain. I understand. But I've talked before about, like, okay, humanity can't coordinate that well. You know, we couldn't coordinate. But, like, this is what coordination actually looked like. Coordination doesn't look like, oh, we're all happy friends, and we hold hands and we all sing songs. That's not what I mean when I use the word coordination.

When I don't coordination, I mean we sue each other. I mean, we have debates. I mean, we fight, we argue civilly, you know, no violence, you know, we're very civil about it. We have rules. We follow laws. We also follow laws that we don't agree with. Like, there's a bunch of laws I don't agree with. But I am willing to make compromises on this. And I'm willing to compromise with the government and with my fellow citizens. You know, if I go to a country where maybe I don't agree with all their laws, while I'm there, I'm not going to break them. And this is not just because of like fear of retribution. It's also because of contracts. and just like making deals and like being fair and changing processes, I think violence is extremely bad for this reason. Because violence violates our contracts. Our social contract is this is not how we solve disputes. If you disagree with something, you don't use violence. You sue.

You have debates. You start a campaign. You start a, you know, talk about a podcast, you know, whatever. Like, we have mechanisms and we should use them. And they're not great. Like in case you haven't seen Twitter lately, our coordination mechanisms aren't great. They're not very good, but they're something. And we can do better.

We can coordinate. It is possible. It's not easy, but we can. At the beginning of this conversation, when we were trying to unpack the different categories for how AI might come to destroy the world, you called this a deeply meta problem. You talked about how we need to solve philosophical problems

about what it means to be human. It's a problem that really goes to the heart of the human condition. I've been talking to quite a large number of AI people in the last couple months or so. And it's been very interesting to me. every time the topic of AI alignment and AI safety comes up, which is like almost all the time, how different approaches the conversation can take across the board. And it seems to be that there are so many different problems that need to be solved in order to solve the AI safety

problem in the first place. I'm wondering if you could just unpack. I'm pointing you in a direction here and seeing if I can just unleash you here. But it seems to be like the AI alignment problem goes very deep as to what it means to be human and what it means to solve problems in the first place. And especially when we are approaching this technology that I think if we, as humans, just like developed as a society over and over and over and over again, like we re-rolled the dice of the human experiment, you arrive at AGI almost all the time, like 99.99% of the time. So it seems to be like, hey, humanity, we have arrived at what seems to be the largest problem that humanity has ever faced.

Nuclear war, nuclear bombs, big problem, not as big, right? Like, disease, big problem, not as big. This is like the big problem, and it seems to be posing very big questions. And so as we come to the end of this conversation, I'm wondering how to tie this off with just understanding the relationship

between what it means to be human and how to solve AI alignment. So what does this vector of conversation sound like to you? Sounds like the real question, you know, the ultimate question. alignment is an important field and sometimes its definitions get stretched to the point where it starts encompassing these larger and larger

meta problems. I think this is not a coincidence. I think really a potential rephrasing of the alignment problem is the general question of how do you control a powerful system using a weaker system? This is something that is not unique to AI. This is a problem that exists

organizations in multicellular organisms. You know, cancer is a form of misalignment. Some of your cells start having different values from the rest of your cells, and they start taking actions, maximizing actions, which are harmful to the rest of your organism.

Our organism has alignment mechanisms. It has stuff like apoptosis or, like, you have cell suicide. There's mechanisms that, like, if it detects that a cell is going down a path, which might be dangerous, the cell kills itself. And sometimes these mechanisms don't work, you get cancers. So this might be a bit of a torturous metaphor, but, like, I do think there's actually a deeper truth here in that alignment and power and coordination

are in a sense all facets of the same coin, are all facets of the same problem, of how do you organize chaos? How do you create stable systems, equilibrium? How do you resist entropy? How do you resist just the eternal cold noise of the universe? Because by default, that's all there is. By default, there's only cold nothingness.

That's what the universe is. It's space. There's nothing else. So space and just random particles. But then things can cohere into complexity. They can cohere, they can form, you know, whether by simple mechanisms such as gravity or chemical things, to, you know, create planets and stars.

And then we have higher and higher level things. We start, you know, seeing complex molecules. Then we start seeing, you know, the first self-replicators. We see, you know, RNA molecules and, you know, protocells. We see early bacteria and archaea. And then they start to coordinate among each other, you know, a lot of theories is that the, mitochondria in the cell was once a separate organism that got absorbed into our cells, into eukaryotic

cells. So there's a coordination happening there in a very abstract sense. And then eventually these cells learn to organize at even larger scale. They organize into multicellular organisms. And then first very primitive, you know, probably like sponges or like jellyfish or something, probably are very simple organisms. But then stuff started to specialize. But to specialize, you have to coordinate better. You have to have better coordination mechanisms. For a brain cell to live, it needs a lot of digestion cells to help it out and a lot of blood cells to help it out and a lot of like the amount of infrastructure that needs to be in place and coordinate it to support a human brain is ridiculous. There's a massive, huge system and everyone has to play their part. You know, some muscle cell can't

just be like, oh, you know, screw it, I'm going to be a neuron now, you know. If that happens, we call this cancer. This is a disease. This is not coordinated. It's not controlled. Mike Livin has a lot of great work on this on like bioelectricity and like cell identity and stuff like this. And then it goes further. And then we get to the meta level to the, you know, we go from the genes, the memes, you know, as Dawkins might say, where we go from purely genetic evolution or like multicellular evolution to memetic evolution to cultural evolution. There becomes now it's about cultures and tribes and civilizations and companies and gods and religions and these kinds of things. And these can also coordinate and mutate and war and change. So there's this recurring motif of life, of these like higher order patterns of complexity

and coordination and stuff emerging. And the way for these things to emerge is through coordination. It is through coherence of smaller patterns, cohering into larger emergent patterns on higher levels of scale. And this is also where we humans are. were nestled here, somewhere in the hierarchy, you know, somewhere above multicellular, somewhere below, you know, pure memetic, you know, religions or stories, somewhere in between. We're kind of like, you know, one foot we're in the realm of animals of, like, the earth. And one foot, we're in

the realm of the gods. We're in the realm of memetics, of information, of stories, of religions. We're kind of this interface between these two things. And so what is the next step from here? Like, where do we go? Like, what? what does it mean? Like, what does it mean for us to want something? Like, in a sense, if we build AI, this is pushing us further into the realm of memetics, of knowledge, of pure abstract software, of something that exists not as a brain, not as a specific piece of hardware, but as a purely, like, as basically a spirit, you know, just like an immaterial code. It's the next step. You know, it goes up.

It goes, you know, from the transition from purely biological, purely physical, biological, then. half memetic, half biological to purely mimetic. They can run on any substrate. Whether there's a CPU that can run its code, the AI can copy itself, and it can exist in its purely memetic realm. So what does this mean? Well, this means many things.

It means, could be mean fantastic things. Culture allows us to do a fantastic thing. Our control over memetics, over culture, of technology, over science, allows us to build this wonderful place we're in. Like, look at this. You know, we've got internet, and they got air conditioning, like, eat all the

food I want. It's great. My animal is feeling great about this whole like civilization stuff. Seems pretty cool. But things can also go wrong. Blast radius in the powerful. If we dabble in magic, you know, in like the, you know, technology. As we dabble in this, we blast raters increases, power increases. And if we, we need to get control over that thing. So this is all just a very nice poetic reframent of where we start at this podcast. It's about how do we deal with, the upper level, you know. There's also a question of down level, like the question of cancer control

or like, you know, medicine is the question of how do we align the levels below us? Or engineering or like physics are the questions of how do we align the levels below us with our values? Now we have the question, how do we align the thing above us with our values? Because we want some things, you know? There's many things I like and there's many things I don't like.

I don't like my friends to die. I don't like being in pain. I would like a beautiful universe full of cosmopolitan art and adventure and all these nice things. And this is possible. But it's not something that an animal can do. It's not something me or you as half animals could even do. It is something that God could do. It is something that a very, very powerful intelligence, a memetic supercreatures could do. It's the same way that the memetic supercreature is that we call civilization has built technology that you or me could never build by ourselves.

And this is great. But what are our alignment mechanisms? You know, our cells have apoptosis. They have a bunch of stuff. And even that's not perfect. But now I'm saying, okay, medicine is great. You know, physics is great. You know, we're developing a lower level alignment text. We also need to develop the upper level alignment text.

I think it is possible that we can understand memetics and AI and optimization. Unless this is magic. That's a cool thing in a sense. It is both magic and it's not magic. It is like information processing. It is things that can be understood. it is coordination. You know, humans could solve many coordination problems, you know, me and you can coordinate, you know, me and all the other people down the street, we can all live together

peacefully. This is coordination. This is controlling something, you know, working as part of something bigger than myself. And if we want a really good outcome, if you really want the world to go well, if we really want, you know, our far descendants, you know, whether they're human or not, to, you know, live a great life, but we have to not blow this all up. We have to actually make sure that the next step we make is not just some random thing that, you know, someone cooked up on a server somewhere, but it's actually something we like, actually something we endorse, actually something we want. We don't want something to inherit the universe and just, you know, blow it up or use it for nothingness. We want the universe to be something that we like. And this is really the full problem

of alignment. It's not just the question of, okay, how do we solve this narrow problem? How do we regulate this thing. It's how do we think about what does it mean for humans to want things? What do we want? A lot of our preferences are culturally constructed. A lot of people want things because their friends want it or because they saw it on TV. Is that a real preference or is that a fake preference? I don't know. This is something we should be asking. What does it mean to be good? Is that even a well-cohered concept? If we can't agree in any of this, is there some way we can bargain? Is there some kind of, you know, Harsani's veil type solution where, you know, we build some super aligned type four system. It goes to all the humans, figures out what they all want, and makes some kind

of like fair distribution. I don't know. What we really want is these type four aligned systems that are so aligned that they figure it out for us. You know, they're like, I'm going to figure out all humans. I'm going to understand their psychology, their traumas, their life story. I'm going to deeply, deeply understand them in ways that no human could understand them. I will use all this incredible superhuman intelligence in order not to do something random, but to do what these poor half animals actually would like, what would actually make them happy, would actually be good for them. And I have no idea how to build such a thing. I mean, I have a few of ideas, but I don't know. This system is not forbidden by physics. There's no reason that such a system

cannot be constructed. But we are so far away from that. We're not even going to get to there. If we can even get to the point where we can just have like systems that we can carefully use, we coordinate on, and we use them to, you know,

a cool world, you know, where people are happy, where people don't have to fear for the next day, where people aren't hungry, where people can just chill. That'll be pretty nice. We can get there.

It's technically possible. But it requires not just technology. It's like technology is not free. Progress is. not free. If you just take the shortest path on the tech tree, you don't get to the final tech, you blow yourself up on

the way. There is a final tech, somewhere deep, deep, deep down the tree. And if we're very careful about how we traverse this, if we improve our coordination tech, if we, you know, work carefully about this and we avoid the existential the traps, you know, the traps along the tree, we can get there.

And that's what like Eliezer or such would mean is truly solving alignment. That final tech. They were like, have understood what are values, what does humanity mean? How do we bargain between values? How do we, you know, think about reality? How do we think about, you know, meaning? How do we think about all these things? And not just that, but never mind the philosophy here. This is not just about this. How do we turn this into code? How do we turn this? How do we build this? I think that I like to think about philosophy that I think is often lost in modern philosophy is that good philosophy

should make you stronger. Philosophy should solve problems. If your philosophy isn't ultimately going to solve problem, why are you doing it? If you really do philosophy, if you really go to the end of philosophy, you should come out on the other side with something that allows you to do these things, that allows you to build these systems, to create this great future, to make the universe good, to make it great. And I don't know what that means. I'm not claiming that I do. Oh, God, no.

You know, I barely have a few technical ideas about how we can get. get to type two systems. A little bit of an idea. But even that, it's going to be tough. We just, all of us, you know, humanity, our technology, our politics, everything, it's going to be heroic. It's going to be heroic task to get to that point. But hopefully we just have to be heroic one more time. Connor, I really appreciated all of that. And that was a very cerebrally and deep conversation. If bankless listeners are out there and they want to continue this conversation, where should we send them? I know that your web company, Conjecture, has a fantastic blog. I can definitely put those links in the show notes.

Is there any other place where people who want to continue this conversation might want to end up? So our website is a bit not super up to date, but I thank you for your kind words anyways. I'm on Twitter at MP Collapse. I'm not super reactive, but I exist there. Sometimes hang around to Luthorai's Discord still. Not super common, but if you pick me, there's a decent chance I'll respond at some point. I go on a lot of podcasts. I do a lot of stuff like this.

If you're interested in learning more about some of the regulation things I'm talking about, there is a website we have made called stop.a.i. There's an ask risk after the AI is that we're not trying to stop all AI, just a specific type of AI. There's a lot of content on there about like fleshing out some of these things. Yeah. And in general, you know, I'm around. And this is not the last conversation, I think, that, you know, will be had on this topic. Oh, I definitely think that's correct. Bankless Nation, you know the deal. Crypto is risky. Ethan is risky. AI alignment is risky, and apparently we need a heroic effort to solve that problem. But nonetheless, you can lose what you put in. We are headed west. This is the frontier.

It's not for everyone, but we are glad you are with us on the bankless journey. Thanks a lot.