The Great Simplification with Nate Hagens - Algorithmic Cancer: Why AI Development Is Not What You Think with Connor Leahy

Episode Date: June 25, 2025

Recently, the risks about Artificial Intelligence and the need for 'alignment' have been flooding our cultural discourse – with Artificial Super Intelligence acting as both the most promising goal and most pressing threat. But amid the moral debate, there's been surprisingly little attention paid to a basic question: do we even have the technical capability to guide where any of this is headed? And if not, should we slow the pace of innovation until we better understand how these complex systems actually work? In this episode, Nate is joined by Artificial Intelligence developer and researcher, Connor Leahy, to discuss the rapid advancements in AI, the potential risks associated with its development, and the challenges of controlling these technologies as they evolve. Connor also explains the phenomenon of what he calls 'algorithmic cancer' – AI generated content that crowds out true human creations, propelled by algorithms that can't tell the difference. Together, they unpack the implications of AI acceleration, from widespread job disruption and energy-intensive computing to the concentration of wealth and power to tech companies.  What kinds of policy and regulatory approaches could help slow down AI's acceleration in order to create safer development pathways? Is there a world where AI becomes a tool to aid human work and creativity, rather than replacing it? And how do these AI risks connect to the deeper cultural conversation about technology's impacts on mental health, meaning, and societal well-being? (Conversation recorded on May 21st, 2025)     About Connor Leahy: Connor Leahy is the founder and CEO of Conjecture, which works on aligning artificial intelligence systems by building infrastructure that allows for the creation of scalable, auditable, and controllable AI. Previously, he co-founded EleutherAI, which was one of the earliest and most successful open-source Large Language Model communities, as well as a home for early discussions on the risks of those same advanced AI systems. Prior to that, Connor worked as an AI researcher and engineer for Aleph Alpha GmbH.   Show Notes and More Watch this video episode on YouTube   Want to learn the broad overview of The Great Simplification in 30 minutes? Watch our Animated Movie.   ---   Support The Institute for the Study of Energy and Our Future Join our Substack newsletter Join our Discord channel and connect with other listeners  

What I personally am most concerned about are extinction risks. If we have systems or things or species that are more intelligent than us, and they are not wise, kind, often called aligned with humanity, well, then I think humanity just doesn't have a long future. If there was a more intelligent species on the planet, and that species wanted our resources, who would win? And I expect the more intelligent one wins. You're listening to The Great Simplification.

I'm Nate Higgins. On this show, we describe how energy, the economy, the environment, and human behavior all fit together and what it might mean for our future. By sharing insights from global thinkers, we hope to inform and inspire more humans to play emergent roles in the coming great simplification. Today, I am pleased to be joined by Connor Leahy, an expert in AI development and a leading advocate for awareness and regulation of the existential risks that unfettered, artificial intelligence presents for society in our future. Connor Leahy is the founder and CEO of Conjecture, who is working on aligning artificial intelligence systems by building infrastructure that allows for the creation of scalable, auditable, and controllable AIs.

Previously, Connor co-founded Elluther AI, which was in the earliest and most successful open-source large language model communities, as well as a home for early discussions on the risks of those same AI systems. If you've ever wondered about some of the basics behind how artificial intelligence software is created, or what exactly about that process makes AI innovation both rapid and risky, this conversation with Connor provides a fantastic primer on these topics, although it is not for the faint of heart. This episode also complements our previous content on the discussions of AI, including the recent episode with Zach Stein on artificial intelligence and education.

With that, please welcome Connor Leahy. Connor Leahy, great to meet you. Thanks for having me on the show. So as longtime viewers of this show know, I am no AI expert. But as anyone awake and paying attention to the world, AI is part of our future. and our present, whether we like it or not. We've covered the topic of AI on the podcast. I will have my team link those episodes in the show description.

But I invited you on the show because you've become a vocal leader in raising awareness about the risks to humanity and the biosphere from the race to develop, artificial general intelligence. So let's start there. For those who might not be aware yet or aren't convinced, start with a definition, brief definition of artificial intelligence, artificial general intelligence, and artificial superintelligence. And from there, use that as a springboard to explain what you think are some of the worst case scenarios for AI and humanity. Yeah, starting with easy questions. Just define intelligence.

So it already starts with that these terms are contentious. There's no universally agreed upon definition even for the word AI. Like, people disagree quite a lot on what the word AI means, on what intelligent means, on what AGI means, on what AGI means. So instead of digging into this whole, you know, valid controversy, I'm just going to ignore all of that and just make my own definitions. And if people don't like them, that's fine. These are just the words I am using for the sake of this podcast. Please, I do that all the time. Exactly.

So for me, AI is generally software that can do things we don't do. know how to write as code. Very simple definitions. It's like everything humans do or learn or whatever that we don't know how to write as like a formal algorithm. You might be confused how can there be software that we don't know the algorithm but still does something. I'm sure we're going to get back to that in just a bit. AGI artificial general intelligence is what I would define as something that can do anything a human can do, at least as good as a human can do it. But not play football or or dig a trench. Okay.

Yeah, sorry. I mean like on a computer. So any intellectual task that like a human could do on a computer, AJ I could do that. Obviously, I think such systems could also have, you know, robot bodies or whatever, but it's not necessary. I don't think it's required. Some people think it's required.

I don't think it is. We can talk about that as well. A cuter way of defining AGI is it's a thing that has what humans have and chimps don't. humans build nuclear bombs and go to the moon, chimps don't. Despite us sharing 99.9% of her genome, having mostly the same brain structure, like down to the lobes and everything. But for some reason, something happened with our ancestors where suddenly we can build nuclear weapons.

Chimps can't. So a lot of people expect that there is like something that differentiates from chimps that allows us to do that. But the truth is we don't actually know what that thing is. A lot of people have theories about what that thing is, whether it's language or the merge operation or whatever. But the truth is we don't know. Like, no one actually knows because we don't understand tellers that well. So the difference between chimps and humans, even though we share most of our DNA, you're making a comparison that between humans and AI, there will be a similar sort of leap?

I think between AI and AGI. That would be such a leap. So, like, for me, AGI is a thing. Like, if AI is a chimp, then AGI is a human. Got all. Like this kind of how I think about it. Got it. So an AGI system was something where you would believe there is nothing that a human could do that it couldn't do, you know, in a reasonable similar time frame.

And then there's ASI or artificial super intelligence. So this I define as a system, which is more intelligent, more competent at all relevant tasks than all of humanity put together. So this system, it would be more capable. than the economy. It would be more capable than all states put together, not just more competent than an individual person. This is kind of how I tend to use these words. Now, there's a lot of, you know, valid questions to be asked of like, how AGI exists? Can AGI says, can ASI exist? I personally think that it's overwhelmingly likely that AGI is definitely like. I mean, it's possible because we have an example, humans, of a system that is generally intelligent.

And we have, you know, very powerful systems such as the economy. Like, the economy can build semiconductors. I can't. You can't. No individual human can make, you know, complex semiconductors, but the economy can. Does that make sense? It does make sense, but the economy can as long as we have copper and silicone and international supply chains and peace in the war and all those things. Of course.

You know, this is a very simplified model. don't they is like literally, but obviously if you had a system, you know, where the system is made of silicon chips and robots or of humans that can extract or refine or develop new blueprints for semiconductors and whatever, pull up new factories, etc., then you can expect it will be able to do these, you know, you can have a system that can do this. And so the thing that I am most concerned about is that I think the step from AGI is very soon, and the step from AGI to ASI is very fast.

So what I mean by this is, is that AI has been on an exponential improvement curve. This is unintuitive in many ways. We're quite used to things progressing linearly. You know, every year it gets like two units better. But there are some exceptions such as Moore's Law, where, you know, every two years or whatever, the number of transistors on its chip doubles.

And AI can ride the Moore's Law wave but it can also write other ways and improvement in algorithms. Like we've seen just or less like five years, you know, things going from you can't talk to your computer to you can talk to your computer and it can solve PhD level math for you

and also generate full to realistic videos for you and, you know, manage your inbox. That's really fast. It took humans, you know, three million years from our ancestors from the first nonchimp to the first guy who could open Microsoft office, right? So this is an extremely fast-level development already happening.

And we have a lot of people, including, you know, the Bell Prize winners, Jeffrey Hinton, the CEOs of the major AI companies, who themselves say that they think we're going to get to AGI within the next couple of years. And this seems plausible. And it's something that I think we should take very seriously. And once you get to AGI,

I think you get to ASI quite quickly. Because, well, what if you had, say, the best program in the world, but they never need to eat, they never need to sleep, they never get tired, they never get bored, they can run 24-7

on a data center chip, you can have 100,000 of them in parallel. Every time one of them learns anything, they can immediately copy this knowledge to everyone else. How fast could something like that do science? And my expectation is it could do it really fast. And then, so we can get from,

if you had an AI system, just as smart as like the smartest scientist, and you run it at, you know, let's say a thousand X speed. Well, that means every day that system could do two years of scientific research. That's a lot. So I think you will get from AGI to ASI very quickly. So very quickly we will have systems that are just smarter than us.

I'm just going to show my naivete openly to you on a lot of these questions. If it's doing science, two years of science in a day, like scientists go out and they measure. plankton populations in the ocean and they take pH levels and the blood and all this. So it would still need the inputs from actual scientists to analyze, yes? For some things, yes, but there's a pretty clear exception to this real world, which is software. Software can run at the speed of software. And AIs are themselves software. So doing AI development itself itself is a task that AGI could do and could do very quickly. It could do it much fast than humans. We are very bottlenecked by the fact that

humans only work eight hours a day before they get grumpy, you know? So is that what people mean when they say recursive, meaning it improves itself? That's correct. So there's some, again, contention about the words. But like what I mean is, for me, when I use the word recursive self-improvement or RSI, it's the moment when you have a system that is as good as the best human engineer at doing AI research, and then you tell it to make a better AI. And then once it makes a better AI, well, that better AI can make it even better AI, and now this even better AI can make an even better AI until, you know, it'll bottom out at some point. But I think you'll be very far from where we currently are.

So what, given all that, given you think artificial general intelligence is quite possible within the next few years and that artificial superintelligence would come soon after that, plausibly, what do you see as the worst case scenario for AI and humanity and the biosphere if you want to go that far? I'm not sure it's worth us talking about worst case scenarios because I think the true worst case are actually kind of unlikely and they're truly horrific. How about middle of the distribution case? Yeah, exactly.

So this I think is much more useful. I mean, to speak quite frankly, let's be clear here, if you have something that is smarter than you and you don't control it, it doesn't end well for you. Unless it's wiser and kinder than me. Unless it wants to help you. If it wants to be good to you, if it wants to help you, if it is kind and wise, potentially, that's totally possible. Do you know how to make computers wise?

No. Yeah. The problem is no one does. And no one is even really trying, like some people are, but like not really. Even if the computer itself was wise, if it takes its orders from someone who's not wise, then there's a problem. That's a big problem. I think it's even worse than that. I think that what's happening right now is that it's not even that we're giving, I think there's many ways and way the world can go wrong.

Like there's kind of like three large categories of like, you know, middle distribution scenarios. One is the kind of like what a lot of people expect is something is called the dominance doctrine, which is this idea that, well, if you build a superintelligence, well, then you could tell it what to do and then you rule the world. A lot of people currently believe this. I don't think this is true. So this already leads to like all kinds of dystopias. Is there a single person on this planet who you would trust with full power? I don't. Like there's no person or group or entity or even ethical principle that I think should have that much power.

So this will already go wrong. And this I think is the optimistic scenario. What's happening right now is that we don't know how to control our AIs. So earlier I mentioned that. with AIs are systems that we don't really understand very often. So it's very important to understand that AIs as they exist today, which is quite different from like how they were done in like the 80s or the 70s, is they're based on neural methods. AIs were done in the 70s and 80s? Yes, actually.

The term was coined, I think, in the late 50s. They just didn't have the compute or the complexity of today's models. No, and used to mean a very different thing. So there's a really funny historical artifact from what's called the Dartmouth conference. which is where kind of the word AI first started, where, so this is like the 60s. And they were like, well, we think that we can make significant progress on, you know, analyzing images and generating, you know, good speech over a summer with about 10 students. They were slightly off, slightly off.

So they lacked compute. They lacked a lot of methods. Neural networks kind of existed, but like not really. and so at this time there was a different paradigm, sometimes called good old-fashioned AI or gofi, which was more based on logic. So it was more like you would code

logical rules that your system would follow. This had some useful properties, such as it made them much easier to understand because you would follow the logic trains that these systems would use. But they were very brittle, very, very brittle. They were very hard to make, and they just didn't really work very well

for the most part. Sorry to the gofi researchers out there. most of them just didn't work very well. There were some applications, of course. So the real revolution in what we call modern AI or deep learning kind of really started like the early 90s or the late 80s with the invention of an algorithm called backprop. This is an algorithm that you can use with neural networks to teach them things. The exact mathematical details don't really matter.

But basically, it's a machine learning algorithm. This is different from you sit down and you write down logic or you write down code. It's much more you give them a bunch of examples. You give your computer a bunch of examples of what you want them to do. And then the computer figures something out how to do that. And this is how all modern neural networks work. All modern AI systems work. What this means is that they're more like grown rather than written.

Like, AIs today are not lines of instructions that you can read, you know, like a human wrote those. It's more like a blob, like a blob of numbers that kind of like saw millions or trillions of examples of the problem and kind of like absorbed them and learned how to deal with them. But from a human perspective, we have no idea what's going on in these things. We know that if we execute those numbers, if we all like, you know, let them run on our computer, they do a bunch of great things. You know, they talk to us about poetry. They, you know, make funny videos and so on. But we don't really know on the inside what's happening. I have, I'm just going to pop in with some, some questions as my curiosity pops up. So, so I've got, I haven't used AI much. I use chat GPT or Claude to do a book summary on someone's research or, or things like that. But when I have like a Google Chrome or whatever browser open and I'm accessing the internet, but then I have Claude or chat GPT loaded on one of the browsers and I ask it a question. And,

it has to go somewhere and use compute to give me the, all the iterations that it researched on my behalf. Is that an additional draw and energy somewhere in the world when I ask a chat GPT browser on my Google Chrome? So it works very similar to other forms of software. So if you access a website, what your computer does is it pings a data center somewhere in the world. The data center is running, you know, hundreds and thousands of computers that are running the software of the website you're visiting. and it returns to you what you're supposed to see. And then every time you click a button or you submit something or you want new photos or whatever, the server that you're in contact with will give you that additional information.

You know, it will retrieve it from its database or it will calculate something and it will send it to you. So you can, you know, your computer can then show you. AI works in quite a similar way. It is unusually compute heavy compared to other applications. Websites are quite light. You know, they usually don't take that much. Even so then give for very large websites like Facebook or something, there's just these massive data centers.

Because, you know, you have millions or billions of people accessing your website at the same time. So you need massive amounts of computers to serve everybody. AI is kind of similar. AI is a little bit different in that they use very special chips called GPUs or graphics processing units, which are a bit different from the CPUs that most of our computers, you know, most of our computer tasks are done with because they are kind of focused. on doing this kind of like blob calculations for the neural network. These are very energy hungry.

They're extremely energy hungry compared to other forms of chips. They often take on the order of, you know, two to four times as much energy depending on the exact chip and so on. And often for the very, very big AI systems, such as the systems, you know, like open, like trashy, and so on, you often need like ungodly amounts of these chips to both make these AIs. So there's kind of two steps. There's the making of the AI or the growing system. stage, and then there's the deploying or using of the AI. This is often called training and inference. In the training phase, you is when you feed the AI all the data. You teach it all the stuff. This is ridiculously compute heavy. You need massive supercomputers, you know, that can take,

you know, megawatts, gigawatts of energy to run these things, you know, kind of similar to the supercomputers we use for like physics simulations and stuff like that, like kind of like a very similar thing. And we run those for, you know, months at a time to build one AI, like one big AI. And then once you have the big blob, then you have to, then you do inference. So you expose it to customers so customers can send it queries and get a response from it. Do you ever just get this deja vu, a wide boundary, shuddering sort of sense that this all is like our modern equivalent of the stone heads on Easter Island? Well, at least our stones actually talk back to us.

So back to my question, then I'll let you get back to your main point. When I ask a Claude a question, is that in the world burning more energy than if I ask Google the same question? Probably. Yeah. I don't know, obviously, but probably. Yeah. It's definitely, like, it's worth keeping in mind that the energy consumption here, per user is quite low.

Well, yeah, I know, but if everyone all of a sudden uses chat GPT for everything in their lives, that there's going to be an uptick. Oh, yeah, yeah, absolutely. Like, I mean, this already happened with websites and data centers and so on. Like, data centers nowadays are a massive, you know, sort of user of large energy. A convenient thing about data centers is you can locate them in, like, right next to renewable energy, which is a thing that happens very often. Data centers are often located right next to hydroelectric dams or in the middle

of deserts and stuff like this because you can guess where you can get the cheapest energy. So because it's okay if it's a little bit farther away, you know, that's not that big of a deal. They don't have these are very rarely like in or near cities. They're usually like out in the sticks. Does the path between here and ASI require a lot more energy? I expect so. I think the path between where we are now and any future society requires a lot more energy. You know, I am not an expert at all on energy infrastructure. But for what it's worth, I do think AGI can be. built with what is currently on the grid. I think we will likely want more because it's faster and more convenient, makes you more

money. But I expect just the current amount of compute that exists in the world is like enough 100 times over. That was very helpful, by the way, because I didn't know those things. Yeah, no, please. I think these things are like not obvious and are often not, like, kind of assumed but not explained very well. Exactly.

But I think they are very important to like understanding that AI is different from other software. It's not just another website. It's like, it is a different technology. It, of course, is on computers, but it has, like, some unique properties that make it, like, quite different. And if we totally understood our AIs, you know, we knew every line of code, we knew exactly how they worked, I would feel a lot better about that, right? Because then, because we, you know, okay, to be clear, I don't think we do a great job on

cybersecurity right now. It's not like our software is bug-free. But at least hypothetically, you know, we can build pretty good software if we really try. With AI, it's much harder. So we talked about this, like, idea of, like, the dominance doctrine. This idea that like, well, if I have a super, super genius that does everything I say, well, then I can run the world. I can, you know, gain decisive strategic advantage militarily, geopolitically, and I can use this to enforce my vision of utopia upon the world. With just a super genius, wouldn't that super genius also need to have access to power and might?

I expect if you have something that is intelligent, persuasive, you know, enough, it can develop technologies. it can outperform the market. It can gain political power. It can tell North Korea, hey, I'll give a super weapon in return. I want you to do this for me. You know, whatever, right? Yeah, yeah, okay.

I think it will probably be much less, like this sounds like a Hollywood movie. I think it's going to be way more boring. I think the thing that's probably going to happen, and we're already seeing happen, is that they're just going to make smarter and smarter and the eyes. And the ayes are going to get smart and smarter. And then they're just going to put the AI in charge of the corporations on purpose.

Like, they will just be like, wow, this AI is a better CEO than our CEO. So we're going to kick out our CEO and let the AI run it. And then we're going to let it run all our hedge funds. And we're going to let it, you know, give advice to all our politicians. Like we already had like examples of people in the White House and so on using chat cheap ET for policy advice, right? Isn't there such a thing as hallucinations, at least in today's level of, of AI's? And isn't that kind of dangerous in itself?

I agree. I think these things should not be in charge of any of these things. But those are getting better as I guess smart. Humans hallucinate too. Like, have you asked your buddy, like any random factual question? He will probably make some shit up, you know, right? It happens to me too. Sometimes I misremember or like, or I thought I remembered something, but it was actually wrong.

But a human does that out of self-deception and evolutionary peacock status, but an AI should be at least conceptually able to say, I actually don't have a good answer to that question, try again or help retrain me or something. Do they admit they're wrong or that? sort of thing. So this is a really fascinating one. So this brings us into how AI is different from other software. So for example, with like gofi with old AI, this is very much

how it worked. It could, it would take its premises and its logic and it would logically deduce. And if it didn't know something, you would tell you. It's like, up, my logic does not contain what you want. But this is not how our AI's work. Our AIs are much closer to copycats than they are to

logical reasoners. So the way they're trained, like chat GPT, clod, The way of train is actually kind of weird. The way it works is, is you take huge piles of just text, books, stories, dialogue, whatever. And you show it the start of the text. And then you say, what do you guess is the next word in the sentence? And then it makes a guess. And then you show it that word.

And okay, okay, now guess what is the next word? And then you so on, so on, so on. And you do this trillions of times. You just guess what the next word in the sentence is going to be. And for some reason, this gives you chat, GBT, Claw, et cetera, if you train on the right data. But what this also means is these things learn to emulate many, many things that humans do. So they, if you give them, like, for example, recently I've been getting a lot of email from crazy people telling me that AI told them that their theory of everything and consciousness and space travel is correct. Because the AI, of course, has seen this before and knows it's supposed to say, yes, that's good.

So they not only emulate human brilliance and science and inference, but also our delusion and self-deception. Absolutely. And overconfidence and macho testosterone, I need to be right for status and blah, blah, blah. They often do. They often do. And it's actually even crazier than that. Because with humans, you know, humans kind of have a personality, right? Like, you know, you have a personality. I have a personality, right? AIs are kind of even crazier than that. They kind of are like super schizophrenic, where if you give them, a slightly different phrase thing, they can just flip their personality entirely. Like sometimes they can be super nice and friendly, and like other times they can be super aggressive

and crazy. And a lot of what these companies do is they put a lot of work in hiding this from the user. So there's a thing called a base model. This is what you remember the training process where you crunch all the data and it spits out what is called a base model. Base models are crazy. There's very few of them that people nowadays have access to, unfortunately. It's really hard to get access to these things.

They're kind of like these crazy schizophrenic aliens. They're super smart, but they just regurgitate like everything they learned, all the emotions, all the different characters. It's like, it's like really weird. They're very, very smart, but they're not, they don't really talk. They kind of like pretend to be things more like. These things then get further modified, which is sometimes called chat fine-tuning, where they train, they then further modify these systems to act more like,

a single entity that you can talk to. This process is not super reliable. So sometimes there's something what's called nowadays usually called a jailbreak, where you can give certain instructions to a chat model, which will make it suddenly go crazy, or like do something it's not supposed to do, or totally change his personality, or output data it's not supposed to output,

it's stuff like this. And all this I know sounds crazy, and the truth is we have no idea really why these things happen or like how to prevent them, it's all super ad hoc. So is there some, not a kill switch, but some secret code words that the developers of chat GPT, open AI or whatever,

can put in that do something? Does any human have oversight over this or once the model is out there? It's kind of the AI on its own. So in general, of course, when these things are deployed in a corporate setting and so on, they will be monitoring the logs, with their users as they would with any product, right?

Like, you know, if use any product on the internet, the developers are watching very closely what you're doing and how they can improve their product further. So that's definitely a thing. Where this becomes a problem is with open source. So there are open source AI systems. You can download open source models and run them yourself. And with those, obviously, there is zero possibility

of overseeing anything going on there. Like, it's just out in the world, anyone can do anything they want, who knows. So there's kind of like, there's some amount of oversight on kind of like the output or like the interface between customer and model. But within the actual AI itself, there's no real oversight. Like there's no way to like bake in an instruction that it will 100% follow. You can try, you can prod them, you can train them in a lot of data that tells them if I'd say this, do that. And that works decently well, but it doesn't work 100% of the time.

So get back to your middle of the distribution. What are the big risks here, Connor? I mean, I can think of a dozen, but you're the expert. So please, please help me understand. I mean, unfortunately, there are definitely over a dozen. So for, I mean, at least a dozen here, right? So we already talked about the first one. It's like, you know, power concentration, dystopia kind of stuff.

I do think these risks are real, right? I do not dismiss these at all. I think these are like a very big problem. What I personally am most concerned about are extinction risks. if we have systems or things or species that are more intelligent than us and they are not wise, kind, often called aligned with humanity, well, then I think humanity just doesn't have a long future. Then I think the future will belong to these things. Please connect the dots on that. I've heard that before.

Can you just outline the sequence of how that potentially could unfold? And obviously, I don't know how it would exactly unfolds. Right. So I could only give a very rough, very, very rough idea of what happened. The way I think about this is mostly just like, if there was a more intelligent species on the planet and that species wanted our resources, who would win? And I expect the more intelligent one wins.

You know, maybe it'll take a while. Maybe they'll be nice at first, and then later they'll betray us. Maybe they'll betray us right away. I don't know. I'm not a smarter species, you know? I'm no chess grandmaster, so I don't know what they would do. I expect what's going to happen is kind of what's been happening for the last couple years. We'll just keep accelerating.

These things will get better and better at doing tasks. They will automate more and more labor, more and more programming. They will get better and better at doing science. They will get better and better at doing diplomacy at business, sales, persuasion, politics, etc. Over time, it will become a competitive advantage to have these AIs run more and more of the economy because they're just more effective than humans. They're cheaper, faster, smarter, more sociopathic. So they will make more money.

So if you replace more and more of your employees with AI systems, you will make more money. So more people will do it. More and more politicians will start taking advice from these systems because they're smarter than their advisors. So they'll use the AI's advice for how to run their political campaigns. And then as this continues, the information ecosystem will continue to degrade. Like, it's already almost impossible to really know what's going on in the world, right? Like you look at social media, like what's true, what's false. It's like already very hard to determine.

What do you think the risks are for job losses and how will AI replace many of today's common desk jobs? And what sort of jobs are most at risk for this? And then I have some follow-ups to that. Yeah. So my general opinion on this is kind of informed by my belief that I think AI and AGI is coming very soon and that ASI comes quickly after. Because what this means is, is that we can get to ASI long before people have bothered to automate all the jobs. Even if it was possible, people just maybe haven't gotten around to it yet. You know, firing people takes time, doing new processes takes time.

While going from Asia to ASI could take a year or less, it could take months. It could maybe only take a week. Who knows? Right? Like, who knows how fast those things can think? I don't know. So therefore, I think the answer to your question is, All jobs. By definition, AGI can do anything a human can do, including, say, develop cheap robotic bodies that can be mass produced.

So one of the, I think one of the reasons that there are a lot of people that follow this podcast is they know I'm flawed. I don't have all the answers. I'm hitchhiking a ride generally in the sapient, wise direction as a human alive in these times. But I'm real. And I make mistakes. And I tell people my feelings. and my thoughts and my opinions, how soon could AI replace even me, like in a podcast, or is that not likely? So I think there's two aspects to it. One is like, when could it do the literal craft that you do? Like, come up with questions, ask them in a, you know, charismatic way and, like, have a good conversation.

I think we're not, I mean, I think we're kind of there already, right? I think the thing that makes you good, white people like you, is because, you know, you're you. You know, we like, we like being with other humans. We like hearing other humans. We like, et cetera. But in terms of like the craft, you know, AIs are already like really good at like asking questions, organizing interviews, you know, stuff like this. But what I kind of meant there is a deep fake of me where people wouldn't be able to tell it wasn't actually. Oh, we're definitely very close to that. I mean, you can already, I say, dupe like 50% of the population for sure. Like, and I think. that number will keep rising. So here's actually last night I recorded what I call a frankly, which is my Friday thing. And I did out of the blue before even talking to you, I did the eight archetypes of human and AI. And I opined that AI is going to act as a filter, not for jobs, which was my initial or ongoing concern, but for our identities. And some people who live in uncontacted no internet areas of the world are going to have their natural human cognitive structure remain intact. But it's going to be like I think you say, algorithmic cancer is going to

spread around the world. And some people, because of their temperament and their insecurities or whatever else, are going to become totally, completely addicted to AI and their self, who they are as a human is going to dissolve the world. way. So I increasingly think that is a big risk. Like we lose what it means to be human because this is going to be so powerful and so ubiquitous. What are your thoughts on that? This is absolutely true. And it's so much worse than you can imagine. Like, it's already happened. Like so much of this has already happened. I know how much, for example, you spend time like the third world, like North Africa or stuff like this. Not recently, but I have in my life.

So, you know, I have friends and so on who have family in these areas of the world. And like, addiction to like TikTok and so on. It's like already a problem in the West. But it's so much worse in like many of these areas where there's just like no antibodies to these kind of things. Oh no. And now with AI, it's been gone into hyperdrive because, you know, now you can have characters that you can talk to. You know, you can have individualized, personalized friends. Recently, Mark Zuckerberg said an interview is that the average, I think he says I'm like the median American has three friends but has demand.

for 15 and this demand can be filled with AI. So not only would you have your favorite friend or girlfriend or boyfriend as an AI personality, but you could have 10 different personalities and call them Fred and Louisa and whoever and interact with them. Absolutely. And there are companies who do that right now and they make a shit ton of money. Why is it that I invite people on this podcast and in my first conversation I just get the sense that they are going to continually inform me that it's worse than that.

I'm surrounded by it's worse than that humans who are all swinging for the fences to make the future better than the default. So thank you for all your work on this. So what else are you? I mean, okay, so getting back to my point, I mean, is there, we're going to talk about governance and your work on regulation and the software hierarchy. of these things. But is there any resistance that humans can have in their own life? One of the eight categories I labeled simply AI Luddites, which is those that understand the things that you are saying. And I had Zach Stein on a couple weeks ago talking about the impact of AI and education that we understand that. So we use AI when we need to, but we cordone it off with like a

Chinese wall that we don't do it beyond XYZ and then we go out and go out in nature and meditate and cook our own meal and are not using it. Are there protocols? Are we going to be able to distance ourselves using

discipline or is this like social media and Facebook on crack cocaine and meth at the same time? I think it's like social media on crack cocaine and meth on the same time. That doesn't mean we can't do something about it. But it is like really worth understanding just like the fact that social media is unregulated is insane.

Like it is as addictive as gambling or like hard drugs. And yet it is completely legal to, you know, pipe it directly into a two year old's brain 24-7, advertise it, you know, no problem, right? Like we have a mechanism in Western democracies for handling dangerous pollutants and dangerous addictive substances. It's called regulation. We regulate gambling. Well, I mean, has been unregulated recently and has been a huge disaster, especially in the U.S., but also here in the UK.

Wait, just real quickly on that, what do you mean recently it's been a huge disaster? I mean, I'm aware of the history of gambling and sports books and all that, but what do you mean recently? Oh, recently they've just legalized sports betting. Okay. Which has just been, again, another rabbit hole. So there's a whole rabbit hole here where recently in the U.S., many states have legalized sports betting. And now it's being super optimized by, like, you know, social media apps to like ping you on your phone, like get a free, you know, bet right now. And not only that, but what I noticed last football season is they want you to bet on the next play.

What is the next play going to be? Exactly. And we've lost the action. It's actually a microcosm for this whole conversation because we've lost the inherent cooperative. sport team dynamic of watching the Packers beat the Bears and we're super focused on our fantasy lineup or the next play, which is a little bit of a microcosm of what AI is splintering our human experience. Absolutely.

Like fundamentally, humans have some just flaws in hard brains are designed. There are just a couple flaws. And one of the big ones is addiction. It's like everyone knows addiction is a thing. It can happen to anyone. It can happen for many different reasons. it's very bad for us, yet once you're in it, it's very hard to get out. Even if you know it.

Even if you know, right? Exactly. And it can happen for all kinds of things, right? And having addicted customers is so profitable. It is just so profitable. You know, have you ever heard the word like term whale before in the context of like, In Bitcoin, I've heard the term whale? Yeah.

So in gaming and betting and so on, you know, whale is the word that use for the big spenders. know, people who spend a large money. And everyone is targeting whales. And like whales usually make up like, you know, like 5% of the user base, but make up 99% profit or something like this. And so these are people who are, so all these apps are not optimized for, hey, let's make everyone have a fun time, you know, doing a little bit of betting for fun. Like, you know, gambling can be fun, right? You know, I have poker now with the boys, whatever, right? You know, it's good fun.

You're not the target audience, you know. If you have a little bit of fun, you know, putting a couple of dollars and a, you know, on a match here and there, you're not the target audience. The target audience is people who get their paycheck and immediately put everything into the next play within four hours. Those are the target audience. And these people are optimizing with like actual clinical psychologists to target, manipulate and groom these people

and create more of them. Like, it's so cynical. And is that, I mean, Connor, is this, so what's the other? Addiction was one of the flaws of the human brain? What was the other? There's a couple other ones, tribalism. is a big one.

Yeah, in-group, out-group. In-group, out-group, that's a big one. You know, there's various, like, biases in that direction. I mean, is this whole thing that as humans have ridden up the carbon pulse, which is drawing down all these fossil and material inputs millions of times faster than Mother Earth built them up, and we're applying that to technology. That technology is more and more faster and faster going right to our brainstem. and it's almost like we don't have control.

It's like the entire species is now going to be addicted once AI comes fully online. Exactly. And like these are the kind of things why I think it's not going to be a big fight. Like some people think like, oh, once the AGI shows up, humanity will ban together and we'll do the right thing. But then it'll be too late. And it's way too late. Like, what are you talking about? By then it's like we're already like, do we feel like we're in control of social media?

Like it's a healthy thing for the point. planet that we're on top of or like tech companies or oil companies. No. And I think those things are, you know, they're still human. You can still regulate them. You can go to their house and serve them, you know, a lawsuit. You can't do that with an AI, with an AGI, or never mind an ASI. Is it too late already? I don't think it is yet, but it will be soon. So keep going. Did you want to say anything else on my comment that not only will we lose jobs, but we're going to lose our identities and that that's unfolding with all the different categories and risks there. I think all of these things are happening. I think all of these things would happen even without

AGI. I think even if AGI doesn't happen, I think all of these things will still happen. I think we will still lose incomprehensible amounts of jobs and therefore people lose a massive amount of economic power. Without economic power, they will often also lose a lot of political power. I think there will be a massive loss in executive capacity to addiction and, you know, just like super optimized propaganda and stuff like this. Like a lot of people are losing their souls, their attention, their political power to Twitter propaganda, to optimize advertising, to pornography, to stuff like this. Like, these are all, what's important to understand is that this is adversarial. There is an attack. There is optimization happening to take away agency.

It's not like a natural thing that just happens in the ether. It's there are deliberate people whose job is every day to take away your agency as much as possible and to monetize it. That's literally what the attention economy is. That's literally what advertising is. Because it's their job description and it's the goals of the shareholders of the corporation. So it's fully approved and sanctioned by our culture. Yep. It's fully approved.

It's fully legal. It's like the fact that it's, and like it's even considered mundane. Like when someone says, oh, they optimize advertising. It sounds so boring, so mundane, you know. But like what they're doing is, is they're optimizing to take away people's attention to something useless. Well, this is this overlaps with my work, which is that we have outsourced our wisdom to the financial market and have become an unthinking, unseeing, energy hungry, mindless economic superorganism. And the more I talk to people like you and Zach Stein, I think that the algorithms are the beating heart of this superorganism.

And maybe let's unpack there. You have coined a term called algorithmic cancer. Can you explain what that is and why it's relevant? So, yeah, so we can dive right in. So kind of as we're talking about this, so like there is a sense where when you build software or when you build an engineering product, Well, let's say you build an airplane.

If you don't know how to build airplanes, it will crash and you will die. So pretty bad idea. So you have to actually understand a lot about airplanes to build a good and safe airplane. Like this is quite hard. You know a lot about aerodynamics, you know, a lot of materials. You have to do a lot of tests. You have to be, you know, it's a hard thing to do. But eventually, we can build planes that fly across the Atlantic and don't crash,

which is amazing feat, you know, that we can do this. Well, software, there are similar things. where as you build software as it becomes more complex, at some point it tends to break once you don't understand it. And this happens all the time, especially with like big corporations, where they have software that is so big, it is so old, it is made of so many parts that no one actually knows how it works. No one actually knows where all the pieces are,

and if something breaks, they're stuck. And it can be like almost impossible to get it back running. This is super common. I sometimes call this software senescence. Senescence is the state of like, you know, when you, when cells stop dividing, when they stop functioning properly when they age. And there's kind of a thing here as well where it stops growing, it stops gaining new abilities and it kind of becomes defective. But there's a strange thing that has happened with AI and not just the most modern AI. It goes back to even earlier forms of AI systems like recommender algorithms from, oh, by the way, I mean, have I mentioned?

All those TikTok algorithms, all those YouTube algorithms, those YouTube algorithms, those are all AI too. So this was actually the first major use of AI. A lot of the funding that led to the creation of the current batch of AI actually comes from the profit made by social media companies from using AI to optimize their social media algorithms. This is where a lot of the funding and all the science originally came from. So there's a direct line between, you know, recommending more slop to your children to charge EBT and. all these modern systems replacing jobs. So there's a weird thing that happens with AI, which is what we talked about a bit earlier,

where it's grown, it's not written. And so this decouples your understanding from the capabilities of the system. In the past, if you wanted to build a software system to do something really complicated, you had to know a lot about software and you'd have to understand the problem very well. Otherwise, you will fail, as happens a lot, right?

But with AI, there's this crazy thing where we can build systems that can do crazier and crazier things that we don't understand. We can build systems that can solve math problems that we can't solve. We can build systems that can, you know, write Shakespeare in ways I could never write Shakespeare, you know, that can do all these things. All we need is more data and more computing power, bigger supercomputers. Sure, there is some engineering that goes on there, but it's like very different from building an airplane. So there's no one who understands all these parts, yet it keeps growing. And it often grows in ways we don't understand.

Like sometimes these weirdnesses are amusing. Like I recently heard an example where Chachybti started refusing to speak Croatian because creation uses kept downvoting the Croatian answers. So it just stopped talking Croatian and just refuse to talk Russian anymore, which is quite funny, you know. is it the end of the world? Probably not. But there's more sinister examples.

So, for example, as I've said earlier, I get a lot of emails of like crazy people telling me about their theories of quantum and consciousness or whatever, right? You know, just like random stuff. And what I found is the style and the amount has drastically changed since an update hit chat GPT where it made it much more sycophantic. There was an update for a couple days that made it

super sycophantic and agree with everything. And that very day, I got like literally 10 times a number of schizophrenic emails where they all had screenshots of chat GPT, telling them how true their ideas were. Dude, that is so, like, personally scary to me because I've been kind of a public. I used to run something called the oil drum and I'm social. So I have a huge network. And you cannot believe over the last 20 years how many, most. mostly male, almost exclusively male, people have some special secret thing that they've discovered. That is the answer to all of our problems.

And they are 100% confident and charismatic. And I'm sure they're well-intended people. But I almost think there's something in our genome with male social primates, with technology, and a status accordion that they want to move up, that this is one of the generator functions of our. cultural predicament we're in because there's tens of millions of these people that are pushing ideas and some of them get funding and it's just disconnected from reality. And you're telling me that AI is going to turbocharge all that. Already is. Like I used to, when I get, when I used to get emails from schizophrenic people, it would often be like long, rambling word documents that

they written themselves. Nowadays, I only get chat GPT screenshots. It's always them with chat GPT talking about their delusions and like reify their delusions and so on. This makes me ill. It makes me feel ill. Yep. And so I think this is the correct reaction. You are having the correct human reaction to this. Meanwhile, the reaction in Silicon Valley is, wow, look how engaged our customers are.

Look at our engagement metrics. Look at how many returning users we have. And this is algorithmic cancer. Like to me, this is algorithm pollution. And what is happening here is there is a massive cost being pulled. put on society that is not being paid by the people causing the harm. I think it is bad that these schizophrenic people are being harmed, right? But who's responsible for this?

Like, who's going to go to jail for this? No one. In the same way that CO2 and the pollution from, I mean, fossil fuels are amazing because they allowed us to do things with machines that we couldn't do with our draft animals or our hands. But the trade from hand labor to fossil fuels and machines is an analog to the trade from human cognition to AI cognition, but the externalities are not included in the prices. So we don't include the price, the externalities of CO2 and the fish having to swim poleward in the oceans because there's less oxygen, any of that in our prices when we buy a coffee cup or go on a

plane ride. And in the same way, the externalities on our brains, our cognition, our psychological development, our humanity are not included in the technology. Exactly. Would AI or social networks still be profitable if those externalities were accounted for? I think probably not. Yeah, probably not. Yeah. I think there's a way, like, could there be a world where all, you know, CO2 and like pollution externalities are included with our cost and we still have a thriving economy? I think it's imaginable, right? It would probably be different from how ours currently is. Well, that's the premise of the great simplification. It's not going to look like this economy and it's going to be much less material throughput per person. But yeah, I mean, at least conceptually

that that's where I would like to head. So is there something like algorithmic chemotherapy? Yes, but it is not an easy solution and it will destroy the business models of some of the largest corporations in the world because their business models, similar to oil companies, is dependent, is only profitable because of unpriced externalities. If we held people responsible for the harm that social media causes, I don't think these social media companies would be making this profit. I think they would all fall apart. So let's stop right there. Is it possible that, like I know Jonathan Haidt, who's been on this show before, has recently been tweeting and and writing a lot about this study that came out that I'm sorry I don't have the details on the top of my mind,

but you're probably aware of it that showed that cognitive abilities of people in the world have declined over the last decade, like precipitously. And the culprit is probably social media and addictive use of screens and such. So on social media alone, let alone AI, let's set AI aside for the moment, wouldn't a precursor to regulating AI and including the externation, of it in the prices, wouldn't we first have to do that successfully in the social media field? I think it would be a very sensible thing and we should have done it 30 years ago. Yeah. Like, I think, I mean, so to be clear, to anyone in the world who wants to truly regulate social media, you have my full support.

Like, I think this is an incredibly important thing to work on. It's a thing I put some work into as well. I completely agree that if we had successfully and like healthily found a way to regulate and deal with social media in a healthy way, in a healthy way 20, 30 years ago, I think the world would be in a much worse, like much less bad place than it is now, and we would be much better prepared for dealing with AI. But it's basically a massive pollution crisis

that is still ongoing that we just never dealt with. So, and now we're getting another crisis on top of that with AI. It's actually the biggest pollution crisis, because to solve the pollution crisis on the biosphere, we first have to solve the pollution crisis in our brains. Yeah, I think this is very deeply true. I think there's like a really, really deep thing here, right? Where sometimes there's a view that like conspiracy theorists like to have of the world is that the reason the world is bad is because there's like some evil cabal.

You know, there's some there's some evil, you know, whoever they think it is, who's in charge and they're making things go poorly. And in a sense, I think this is kind of cope. I think it's cope in the sense that like it's kind of more comforting to believe that somewhere someone is in control. This is our in-group out-group algorithm. Exactly. So even if they're bad, even if they're evil, at least, you know, someone knows what's going on. Someone could, you know, could do something if something truly terrible happened. But my, but the truth is there are no adults in the room. No one is in charge. It's just chaos. I don't think anyone is, I don't even think the social media companies are winning. I think their kids are getting addicted to. Like, I think they're suffering as well from the decay of their society. So this is, this is exactly what I say in my superorganism movie and my work is we truly. have outsourced our wisdom as a species to the financial markets because every company has to have shareholders. If the CEO all of a sudden gets religion and wants to do what's right for the biosphere and future generations, he or she will be booted out by someone who

maximizes profits according to the shareholders. And it's the same with a fossil field company. It's the same with any company, including an AI company. So until that changes, all of these intentions and well-meaning and wisdom and kindness are downstream, lower in the decision-making hierarchy than this optimized profits, optimized growth, optimized GDP. And so it is, in that sense, it's no one's fault, but it is also all of our responsibilities once we understand it. Yep. I think this is exactly correct. And this is the deeper, like, rabbit hole reason for why I think ASI will not be kind to humanity. Because humanity isn't building it.

That thing is. That process you described. That's the thing building ASI. And what do you think it's going to do with it? Well, it could say, look, there are, we're already transgressing six of the nine planetary boundaries. Things are accelerating. There's a long, long list of ecological woes. there are 8 billion humans on the planet who consume 19 terawatts of energy continuously.

What do you do, ASI? And the answers are probably not good for humanity. I think that's exactly correct. It doesn't even have to be evil necessarily. It just has to be like efficient. And logical. And logical, right? And, you know, no emotion sociopathic, which is what we expect an AI to be.

It will just be optimizing. But it would also know that it's going to need humans to procure energy for it and all the like. For a while. Like I think, like the way I think about it personally is I think the point of no return, like game over, is quite a while before humans actually go extinct. Like I think we'll probably stick around for a little while, right? But like it's already to be too late.

I'm going to just put you on the spot. And of course, it's all speculation based on your expertise. I have a, I have a distribution in my mind of this century. How likely do you think it is that humans are extinct or largely extinct in the next 50 years? Depends in a condition on we do something about it or not. If we, because I do think things can be done. And I think this changes the probability. We're going to get to that.

We'll get to that. I think the probability can change dramatically, depending on what we do of the next two years. If we do literally nothing. If we do literally nothing. If we do just, you know, I would say like, I don't know, 50 to 80 percent. Yeah. Okay. So I want to get into your, uh, your, your,

your work. But one more question about AI. How is the way that AI solves problems and performs cognition different from the way that humans do? And why is that misconception shaping the discourse on AI and what you referred to as AI alignment? So the true answer is that we don't really know how either of us really do cognition, right? Like, we don't really understand human cognition, not on a very deep level. And we don't really understand AI cognition on a deep level. We understand a bit more about humans for sure. In particular, humans tend to have a universal kind of like set of emotions and priors and bases that, well, not all of us, but that AI lacks. And this is usually a thing that trips off people the most with AI, where, for example, most humans are, don't really want to hurt other people.

They don't, they're not, you know, if they have enough food, they're safe, you know, respect and whatever. they don't really want to just like randomly go out and hurt people. There are exceptions to this rule though. And people, you know, would sometimes come up with theories like, well, if you're just intelligent enough, then you become peaceful. And I think this is nonsense. I think this is just humans have a thing in their head, you know, designed by evolution that says like all things equal, be nice to the people around you.

Be nice so they, you know, so you can build a tribe, whatever, right? But there are, for example, sociopaths who just, just don't. don't have this. Like, I think the way to think about sociopaths is not that they have an additional evil gene. It's more that they lack various emotions that drive us towards prosociality. Is there a magnetic attractor between sociopathic humans and the power that AI provides? I mean, quite clearly so. Like, I mean, there's a clear drive here, right? Like, a lot of the people that are building this technology are the same ones who admit that it could kill everybody. And they do say this clearly, like Sam Maltman, Darry,

Amadeh, etc., have said that it could kill everybody. And they're willing to take that risk. And like, I've talked to people in like San Francisco who say stuff like, yeah, there's 10% the AI is going to kill everybody, but 90% it'll go great. So it's totally worth it. Which is just like, like, ugh. Like, like, like, like, like, like, I remember Dario Amadei saying, for example, he thinks there's a 20% chance of things going poorly and potentially everyone going extinct, which is worse odds than Russian roulette. But aren't they all in this race that they understand the risk? But if they do nothing and just grow potatoes and write poetry, that other AI companies are going to try to go towards AGI and ASI anyways? So this is, I think, really getting to the heart of the thing of like, why can things be done?

the true thing is, is like, obviously this is a fake answer. Obviously, these super powerful men with billions of millions of dollars, all these connections, super political club, extremely intelligent, extremely charismatic people, of course they can do something to slow down the race. Are you kidding me? If Sam Altman tomorrow, a charismatic, intelligent, super well-connected, super rich person decided, I'm going to stop this race, obviously he could do something about that. Could he single-handedly solve it?

I don't know. But, like, obviously, he could go to politicians. He could lobby for things to slow down. He could, you know, found new organizations. You know, he could just say clearly to every journalist that would listen, hey, we need to stop this. Or he could just himself not make it worse. And, like, there are obviously things that people, like, you know,

there's a thing that happens a lot where powerful people pretend to be oppressed in order to do the thing they wanted to do anyways. Look, sometimes there are people who are genuinely oppressed. Like, you're genuinely in a shit situation. You need the paycheck. There's no other way for you get the paycheck. So you just have to do something not so great. If you're in the situation, I'm sorry, sucks.

Like it does happen, right? But that's not the situation these people are in. So what's the main driver? Just status and power? They want to build ASI. It's not, like, you can read their blogs from even like 10 years ago where they just really want to build AI and build Utopia. Like, it's not deep.

It's like, wow, I can build a super cool thing, make tons of money, and maybe become God. Epic. So who do you see as the primary actors that are most driving this acceleration to AGI? And how does this backdrop contribute to the malignant outcomes in the middle of your distribution you're discussing? Or is it everyone? I mean, it's definitely everyone who's trying to build AGI. Like at this point, people aren't even hiding it. Back then, people would be a bit coy about it.

They wouldn't, like, use the word directly. But now we have, like, fucking Alibaba saying on their Quinn III report that this is part of their goal to build artificial superintelligence. They just, like, straight up. It's just on their blog, right? It's like, it started with deep mind, moved on with open AI and anthropic. Like, those are kind of like the big ones, right? But now it's everyone is fully on board. everyone is pushing as hard as possible, right?

You know, whether it's, you know, I mean, mostly in the U.S. and San Francisco, they're definitely the ones furthest ahead. So like OpenAI, Google, Microsoft, et cetera, are definitely the furthest ahead in the race and are the best capitalized. But there's a bunch of tier two actors as well that are gaining ground, including in other countries. So if there was an AI actor that did have some positive outcome for humanity and the biosphere aligned in their training and their algorithms or their objectives or their stated

corporate laws, would they be outcompeted by these other ones? And is that a possibility that there could be an AI that has some morality either embedded in the training or in the people that oversee it? So, hypothetically, could we build a software system that if we execute it would lead to a good, morally aligned, like, happy future for humanity. Physically, of course. Like, there's no fundamental barrier to building such a system. But let me be clear here. This is kind of like saying, we're going to build a system that solves all problems humans face day to day, all problems that the government's face, that the economy face, that corporations face will solve all moral problems, will solve all. you know, voting theory, all like governance problems. And we're going to stuff all of this in one shot using software that will have no bugs.

And I'm like, okay, hypothetically, like we are so far, so unbelievably far. If we had all of our greatest scientists, all of our greatest mathematicians, you know, in a massive Manhattan project, work on this for decades, you know, for a century, all generations of our greatest mathematicians, work on this problem, our greatest philosopher, worked on this problem. And, you know, unlimited funding, you know, highest security standards, extremely careful with, you know, international oversight. Could this work? Yeah, I think so.

But this is not what's happening right now. How could we make that more likely? And maybe you could bridge the answer to that to what, what you're doing with your organization and your work? I think the truth is that given how close AGI is, the number one most important thing is to buy time. You know, political change, social change, building large projects takes time. I think we can, for example, solve alignment. I think this is a thing that could be done, at least good enough, right? That we would, like, endorse it morally, that we would, like, globally, we poll everybody.

And, like, mostly everyone's, like, all right with this, right? Like, look, if we poll the entire world and, you know, 90% of people said, you know, who cares, just let it rip, we don't care about safety. Fair enough, right? Like, this, I would disagree, but I think this would be like a much more morally acceptable, you know, state. But this isn't how it's happening. Like, we've done polling. And like, people do not want to take these risks. It's a very, very small group of quite sociopathic people, especially in San Francisco, that are extremely driven to build these systems, even at these extremely high risks. And the number one thing is to stop them. You know, it's just to be like this is illegal.

To be clear, I'm making very clear, the way to do this is regulation. The way to do this is legal, lawful legislation. This is the mechanism we use for chemical companies in the past when they were polluting rivers. It's the mechanism that we used to, you know, deal with nuclear proliferation across the world. We have mechanisms. So the sequence for that to happen is, first of all, there would have to be education. education and awareness. I went for a walk yesterday with a local friend of mine who is basically just a farmer. And she said, oh, boy, I wish I had some extra money because if I did, I would

put it into the market because AI is going to make everyone so rich. She just doesn't understand these risks that we face. So the first step is to make more people aware of the loss in jobs, the loss of autonomy, the fact that ASI could extinct us all, the loss of identity and the cognitive entropy that AI is going to do. So we get more people educated and aware and feeling it. And I know a lot of people are feeling it, but I think that's probably still a minority. And then, you know, maybe there are things that if we have a super majority of humans in a population that vote for something or put an opinion on it, President Trump would be a super majority sort of person

that might want to do the will of the people and others in the world. So what are the possibilities of that happening? Yeah, I think you've got it exactly right. I truly think the solution here is not some crazy, hair-brained, extremist, you know, scheme. I truly think the right thing is, to do our job as citizens of democratic and free nations, which is to see the risk that it is to bring you to awareness of our fellow citizens and our policymakers to discuss and suggest policies

nonviolent to solve these problems and then actually implement them. This is what we have democracies for. I think a lot of people have just forgotten what it means to be a citizen of a democracy. As being a citizen of a democracy doesn't mean, oh, your policymakers know everything and they figure everything else and you sit around. No. If you have a citizen of a democracy, it doesn't mean, oh, your policymakers know everything else and you sit around. no, if you have a problem, you go to your policymakers and you say, here's an issue I care about. This is the problem. We want, like, how can we figure out a way to solve this? You know, you talk to, you organize, you talk to other people in your community. You'd be like, hey, we all care about this issue. What can we do about this? All of this is kind of like non-glamorous compared to like, you know,

building, you know, silicon god, you know, in San Francisco. But this is what made the world great. This is what made democracy great, right? Like, is the fact that this is how. we solve problems. We don't do violence. We don't do crazy hairbrain schemes. We do, we educate, we bring solutions, and then we use state capacity to do things. Do you know Audrey Tang or of Audrey's work? I know of her work, yes. She was recently on the podcast. It was really inspiring on what some of this open source, open democracy

things could do. I think it would be interesting to have you and Audrey and for instance, Daniel Schmachtenberger or someone like that on a roundtable to discuss this, something like that is going to have to happen. So right now, most people, most people that I know, only intentionally interact with AI using things like chat TPT or Claude. Are there any moral hazards that should be considered for individuals who use just those simple tools, but are also concerned, understand and are concerned about the risk that you've outlined today.

So the way I personally see things is, is I think the moral cost for you personally using an AI tool is quite small. You know, like I wouldn't put it much higher than you using social media. Or taking a flight. Or taking a flight or whatever, right?

Like, I think there are people who have stronger, you know, ethical principles on these than I do. I personally fly quite frequently for work and stuff. So, like, I respect people who have, you know, stronger, you know, you know, thoughts there than I do. I do think there are risks that are worth keeping in mind, such as that AI systems just do make things up and they often make things up in subtle ways. And it can be very

easy to get lazy in your own thinking around these kind of things. I think these things are similar to how social media is dangerous for your political alignment. Well, I actually think the energy use and material use, that's one thing. The output, which is that the GPs get lazy and might be diluted, like, that's another. But the eventual day-by-day slow dependence on something external, we're outsourcing more and more of our own skills to the AI cloud. And then what's left in here and in here? I think this is a real issue. I don't have a good answer to like, what is the correct balance here? How do you, you know, deal with these things? Same thing with social media, right? Like, what is the correct way for society to interface with social media? I think a good society doesn't have zero social media. Same way I think a good society doesn't have zero gambling, you know? Like, I think there are ways to have these things that can even be net positive. Like, especially social media. It seems obvious to me that we could build social media that is good for people. Like that is like, it helps build communities that help build connection that help people become more independent. It's just, I don't think it would be profitable to do that.

So what are you doing with your organization, your work, and given how concerned you are about this and this is your field, what are you doing? So it's a few things I work on. So my day job is I work for a startup that I founded, which works on technical AI alignment kind of stuff, kind of stuff we talked about, like teach something very, very briefly only. I've become a lot more pessimistic about this direction, though, over the last couple of years. I think it's possible, but just extremely expensive and slow. And so I am a close advisor to the nonprofit control AI based here in London, which is advocating exactly for the kind of policies that need to be passed. And briefing policymakers, so over the last couple months, we have briefed over 80 members of parliament here in the UK. And we were told by every political consultant in Westminster that,

no one would ever sign our statement. No one would even give us the time of day. We were talking crazy. And this was extremely not true. In our experience, people are often surprisingly reasonable. It's just that they are uninformed. That's why I like that you use the word education. So most of our meetings are just politely informing or lawmakers about information they don't know about AI.

Let me ask you this. You just mentioned that a lot of these Tier 1 AI plays are in, not only the U.S., but in San Francisco, a lot of countries in the world, there's probably 200 countries in the world, a lot of them don't have any horse in the race in the tier one AI. So this becomes a global West and a UK-US sort of thing. I mean, I would think even Europe is less sanguine towards AI scaling than the U.S. So is there, like how is that, with the geopolitics and all that? I have. haven't followed that closely. Yeah, I'm glad you bring that up. My colleagues are actually going

to be publishing a paper on exactly this topic very soon, hopefully, which is that, yeah, I think there is a huge opportunity here for middle powers to work as brokers of peace and international agreement where, like, quite frankly, just from a purely, let's forget all the optimism and humanism for a second here. Let's talk purely real politic, you know, NATSEC. There are companies in the world building systems that will plausibly be powerful enough to disarm and disempower the United States government and military. These are systems being built. Superintelligence can disarm and disempower the United States government and military. This is happening on American soil today. They don't exist yet, but they are being built. So from a purely logical

perspective, as a national security, this should obviously concern you. You're not. You're not going to should obviously care about this. What do you mean private companies or building systems that they themselves say will be used to disempower? Now they've changed their rhetoric over the last year. Now they don't talk about creating utopia and so on anymore because that implies disempowering the US government. Now they keep saying, oh, we'll give it to the US government. You know, we'll help the US government do this. But what does this mean but the disempowerment of all other nations? If there is a super-intelligence, that is built on American soil.

Well, first of all, I think it destroys the U.S. It doesn't empower it, but let's ignore that. Even then, well, it disempowers all other nations. All other nations now have a massive national security risk that is exclusively located in foreign borders. This is a big problem. Do you know what my honest reaction is when I hear this? My honest reaction is I really want to buy a puppy.

I'm serious. Because a puppy is AI proof. They're not going to be affected by this. Yeah. I mean, isn't there this deep yearning for authenticity and somewhat maybe romanticized conditions of our ancestral human? I mean, that's how we're all wired anyways. We're trying to go through our daily routines,

getting the neurotransmitters and emotional states of our tribal great-grandcestors, 10,000 generations. ago and AI is is like the mother of all speed bumps between me and the future that I would like. Yeah. And this is how I see it personally. This is why I think the first thing is very clearly it should be illegal to build ASI. Like, you know, just like straight up. Like this should just not be a thing. But no one's building ASI. They're building AI that will eventually become ASI, right? Exactly. And then the problem is how do you do this? And I think the thing is you have to be. ban precursors to ASI. Because once ASI exists, it's already too late, right?

It's like the moment an ASI exists that is not perfectly aligned and wise and blah, blah, blah, it's game over. So our number one policy objective must be to never get into the situation in the first place. So we have some suggestions for how to do this. We've written a piece called a narrow path. You can look it up narrowpath.com. We'll post it in the notes. where we talk about our principles for what precursors could look like and how banning them could work. We have recently also drafted a bill in the UK for how this could work.

When you say we, what is that? Control AI, sorry. So still talking about control AI here. But I love the thing you just said, is that fundamentally, this is a speed bump between the future we want. I think this is exactly correct. This is how I think about it as well. There's been a third project that I've been like starting to spin up lately, which is, Control AI is focused on preventing that speed bump.

Like we need to like flatten out that speed bump. Otherwise, it doesn't matter. But then there's another question. All right. Let's say we ban ASI and this buys us 20 years or something, right? Because like at some point someone's going to figure out how to build AGI on a laptop. Like at some point someone's going to figure out how to do it. I think it's going to take a couple decades, but someone's going to figure out how to do it.

And so, or you know, some rogue terrorist somewhere will build AGI. or something. So I think a maximum we can buy is like 20 years or something. Okay. If we have 20 years, how do we get to a good future? Because I think we all have this feeling in our heart that it doesn't feel like the world is on track. And what I mean by that is I don't feel like we have the expectation that next year is definitely going to be better than this year. Like people don't have this expectation. And like what would it mean to put the world on track? We're like every year, we're like next year is going to be better and like we're confident that next year is going to be better.

And this is the thing I've been thinking about a lot lately. I've been thinking a lot about like, what would it mean to build a humanist future? Like a future for humans that we like, you know, whether it's pastoral and puppies or something totally different, right? Like I'm not committing to any specific utopia. What I'm committing to is I think there needs to be a process. You know, there needs to be a process of,

of how do we make iterative improvements? How do we get better at morals? How do we get better at values? How do we build better institutions? How do we coordinate as a species? And I think this is something that really needs to happen. I agree. And I don't think there's a direct one thing now,

but the whole purpose of this platform is to change the hearts and minds and the initial conditions of the future so that the next cycle, whether it's a month from now or six months from now or two years from now, the conditions have changed such that we have better options. And that's the goal of my work. So one of the problems, I think, with hearing about climate change and geopolitics and all the issues we discuss, including now AI and ASI, is hearing the story, at least on the surface, gives people an ill feeling, but also a lack of agency that there's nothing that they can do. So for those who are following and tracking what you're saying today, Connor, how can they start engaging and taking impactful action on these issues in their own lives or in their political spheres?

There's two big things that I think are the most important things that can be done right now. The first, it's got the control AI kind of like policy side of things. So you can go to the control AI website right now, go to take action and you can send a letter to your representative right now. If you're in the UK and the US, it gets auto-filled. It's exactly what you need to do. Minimum thing you can do. Tell your representative, I care about this issue. If you want to do more, fantastic.

We are following what we call the direct institutional plan or the dip. Also on our website, you can give it a read. The plan is very simple. We write down the policies that need to be passed to prevent this problem. And then we go to every policymaker in the world and in good faith make the case. We inform them. We educate them. And to do this, especially internationally, we need people.

We need people from all across the world, normal people, who put in a couple hours of work a week, who just are willing to phone their senator, talk to their friends, do stuff like this. Also, in particular, if anyone from Hawaii is listening, it's the only state we currently have not had someone send a letter to their senator yet. So if you're in Hawaii, please, we need your help. We have a lot of listeners in Hawaii. Let me just ask a subset of that, which could be.

be an awkward question, but what if every citizen in the world outside of the 330 million people in the U.S. were 100% on board with regulating AI and making it illegal to build an ASI, would that matter since most of the Tier 1 plays are in the USA? Absolutely. This would matter immensely. I think there is a lot of bravado about like, oh, the U.S. can do whatever it's want. It doesn't care about other people. This is not true. Obviously, other countries can put pressure upon the US, both soft and hard power-wise. Obviously, like, you know, there are trade relations and they're just cultural relations. There's friends, there's family, there's culture, there's wanting to be the good guy. You know, there are, it really does matter. Like, people all across the

world, all countries, like, sure, you know, obviously if, you know, President Trump would like to have a chat about this, that would be great. But also other people in other countries, it matters. Presumably China has some tier one AI plays underway. Close at least. Okay. Okay. This is the big issue, right? Like fundamentally, we need a world where we have international agreement between China and the U.S. There's no way around this. So there's, I mean, I care about the natural world and the other species that are on the planet and climate change and the oceans and all that. And 20 years of researching that made me realize that we have to deal with our entire economic system. issue first. And then I am starting to come to terms with we have to deal with AI first. But underpinning all those things is governance. I think governance is the single issue in the world that will allow us to navigate the narrow path on many of these issues. Do you have

thoughts on that? I think this is definitely correct. I think we have seen a lot of loss in state capacity in many places in the world, especially in the U.S. It's become very hard. to pass legislation, to have even just rational and like calm debates about controversial topics, especially on social media is like almost impossible these days. I think these are things that make this problem much harder. Like it's crazy, right? Like I'm sorry to complain about social media again, right? But like imagine it's the 1960s and Soviets came to Washington, D.C., put up a radio tower and started broadcasting propaganda. What would happen?

Immediately immediately be arrested. Yeah, hell would break loose. Like, are you kidding me? We would have never allowed the Soviets to do this. But now, everyone goes on social media, TikTok, whatever, and gets directly blasted, you know, with like propaganda, you know, those things, you know, maximizing outrage, etc. So this does make it a lot harder to have good governance,

especially global governance. And there are real tensions here. Like, I want to acknowledge here. Like, the rivalry between the U.S. and China is very real and it's very serious. And I'm not trying to. say people here are being irrational or stupid. Like, there's some shit. Like, I get it, right? Like, this is heart. I remember fondly once I spoke in the House of Lords here in the UK, and I talked about AI risk, and someone in the audience asked the obvious question, but what about China?

You know, China will never agree to this. They will never disarm. It's never going to happen. and then this old Scottish lord Lord Desmond Brown who's a good friend basically said what the hell are you talking about we did the same thing with nukes with the Soviets

it's diplomacy yet it's hard obviously it's hard but like it has to be done like what are you talking about it was a great moment I wish it would have caught on camera it was such a wonderful moment so I think

it is very important to acknowledge that the odds are stacked against us like you know the odds were never in our favor here. Like what we're trying to do to build a good future for our children, for our grandchildren, for our grand-grandchildren, indefinitely into the future with powerful technology, well,

it's not something's been done before. And I'm trivial, like, obviously, technology hasn't existed before. There's not something's been done before. This is something totally new. It's something extremely hard. It's something extremely ambitious. But damn it, it's worth trying. Like, damn it, it's worth giving it everything we can to try to actually build a future. Because we can.

There's no physical reason why this can't work. We can build governance. You know, we didn't nuke ourselves to death, right? Like, we actually managed to get through the Cold War without nuking ourselves, you know, except dropping that one bomb in South Carolina accidentally, but that was an accident. You're right. It is physical and technologically possible.

It's just that we have the perfect monkey trap has been developed where we're grabbing hold of the banana and not willing to let go because of our curious George social primate brain algorithm nature that we described earlier. And that's the issue. I think it's a very important thing here also, which is one of the core reasons why I have not given up. And when I say, I think it's not over. I do really mean this. I don't just say this, you know, for content.

Like, I do really believe. One of the core things is that people really don't want this. Like, we've done polls with the UK, US. You know, we've talked to the general population everywhere, and people hate AI. They're scared of it. If you tell them, hey, there's some guys in San Francisco building things that are smarter than humans. They don't really know how to control.

How do you feel about that? The answer is universally bad. Like, what? How is this legal? Are you kidding me? Currently, there's more regulation in the UK. I got this checked by a lawyer. There's currently more legislation in the UK.

There's less legislation on building powerful superintelligence systems that might kill everybody than there is on selling a sandwich. So this is a thing that people do care about. And they, once you inform them of this, like, obviously, obviously this can't be legal. Like, obviously this should be highly regulated. Like, we just need to actually do it. Yeah. I think the education curve is starting to catch up.

Because to be honest, a year ago, I was a babe in the woods on these issues. and even three months ago. So personally, I've become a lot more aware of these things. And when I talk to people like Zach Stein and yourself, even more so, I would love to have you back. If you have a few more minutes, could I ask you questions that I ask all my first-time guests? Just taking off your AI hat for the moment, which for you is probably difficult to do, you're aware of some of the issues we face on the planet. Do you have personal advice to the viewers of this program at how to manage all this in their own lives?

Like just recommendations for behaviors coping or engaging or whatever you think. Yeah. I think the most important thing is to not overdo it and to not do nothing. This would be my first piece of advice. Okay. I think it's a very, very good thing. if you spend, you pick a some amount of time, you per week, can be an hour, can be half an hour,

can be five hours, whatever, where you try your best in good faith to think about the problem, think about how you would solve it, reach out to other people, read more about it, and then don't really do more than that. Yeah, that's great advice. Yeah, like, I think, and I not just say this,

like, to like, you know, protect your soft heart or something. I truly think long-term, stable investment is way more useful than these like burning yourself out and getting like super depressed and working over time. I think this is super useless. It is much, I would much rather work with someone who gives 100% for two hours per week than someone who burns out 80 hours a week and is like useless after three months. So if you want to do something, this is a power condition. Here, here. And you said that was one piece of advice.

Do you have others? There's always many advice. another important thing is to not forget to have fun. Like, you should still do things. There's a thing that some people do, including myself in the past sometimes, where you feel guilty for doing things that are fun because you know bad things are happening. And sure, we can argue about Catholic guilt. Like, I was raised Catholic, you know, until Kazga home.

So, like, I know how this is. But, like, this is just not a productive way to be a human. Like, both, like, burnout-wise, like, you just won't handle it. And two, if you give up on this, then we've kind of already lost what we're fighting for. Like, I, some people are like, oh, I won't have kids because of AI. And I just, like, so strongly disagree. If we don't have kids, if we don't love our kids, if we don't do things that are fun, if we don't eat good food, then we've already lost. We've already lost what's the point?

So still have fun. So speaking of kids, how would you change your advice for young humans kind of 15 to 25 who are watching this program and learning about this? stuff, do you have advice for young humans? Depends on the exact young human. Mostly is try many things. You have a lot of time. You have a lot of energy. So you should try many things. You should read many things. You should talk to a lot of people. You should make a lot of mistakes. People will forgive you for your mistakes. This is a great benefit that you have for being young. It is okay for you to do something embarrassing or stupid. People will forgive you. We've all done stupid things when we were young. you should use this privilege, you know, by trying to, like, I find it very endearing when I get, like, 18-year-olds or something sending me these emails with all their theories and, like, stuff they want to do. And I'm like, man, kid, none of this makes any sense. But, like, I'm really glad you're trying. Like, like, but I say this genuinely, right? Like, I'm not trying to be facetious. Yeah. Like, you're, yeah. You're a good dude. You embody the, the zeitgeist of this program. I'm so glad that, uh,

that we were introduced. A question I ask all my guests, personal question, what do you care most about in the world, Connor? I think the true answer is, I just want everyone to be okay. Like,

not even like utilitarian maximize happiness and just like, I just want everyone to be okay. I want everyone just having a good time, you know, spend time with their family, their friends. You know,

if you want to do some crazy stuff, that's fun. That's also fine. But I just want everyone to be okay. Thank you for that answer. If you had a magic wand and there was no risk or recourse to your reputation or status or safety, what is one thing you would do? And I could guess at what you might say.

What's one thing you would do to improve human and planetary futures? Well, I mean, depends on the limit of the magic wand, obviously. But I mean, I think a big one would be I would summon the textbook from the future that has the solutions to AI alignments. and coordination and all of moral philosophy, all written down in a nice, explain like I'm five format that I can understand. That might be a pretty good start. Some version of time travel.

So if you were to come back on this program in six to nine months or something like that, what is one topic that you have expertise and are passionate about that is highly relevant to human futures that you'd be willing to take a deep dive on? We should definitely talk about humanism values coordination, like how do we build institutions, what are human values, how do we like build a good future, stuff like this. As that project I talked about earlier that I've been starting, also for the people interested in potentially putting a couple hours of volunteer work into doing something, it's called Torch Bearer. It's a brand new project. It's not super public yet. But if you want to put a couple hours into trying to do something for the world, you know, building a humanist,

future, reach out. This has been great. Do you have any closing comments or thoughts for our viewers? I would just say, as I've said many times before, it is not over. Like, I think the cynicism of our age, like this belief that, like, things can't be changed is kind of like a self-fulfilling prophecy. And it's often a much bigger hurdle than many other things. There's a thing where I've talked to many incredibly smart people, like geniuses, you know,

like, you know, super, super smart. talk to super rich people to super powerful people. And one thing they all have in common is they all feel like they can't do anything. They all feel like they're powerless. Everyone feels like they're powerless. And this can't be true. We can't all be powerless. And like we were talking about those, you know, like I was talking earlier, when we

started our campaign in Katrina, like briefing politicians and getting them to sign, you know, onto our campaign, we were told it's impossible. No one will do anything, you know, blah, blah, blah. And this just wasn't to you know. We have 35 supporters who have supported our campaign against extinction risk from AI parliamentarians, like real elected officials, right? This is not a simple thing, right? And so one thing is it's just like so much of the narrative that like nothing can be done. Oh, AI must be built, blah, blah, blah, is a complete self-fulfilling prophecy. It's not true. This is not an external force coming from outer space

onto Earth. It's being done by humans today. It's like the people who say, oh, we can't do anything, we can't stop the AI race, are the ones built? holding it right in front of you with their own hands. Like, we can do things. That doesn't mean it's easy. Don't take me as someone who's saying like, oh, this is going to be easy. Everything's going to turn out fine. We do need to do something.

But we can do it. So there's one thing to take away from it. We can't do it. We just need to actually do it. Thank you very much for your time today and your very important work and your passion about these important issues to be continued, my friend. Thank you. If you enjoyed or learned from this episode of The Great Simplification, please follow us on your favorite podcast platform.

You can also visit The Great Simplification.com for references and show notes from today's conversation. And to connect with fellow listeners of this podcast, check out our Discord channel. This show is hosted by me, Nate Hagan's, edited by No Troublemakers Media, and produced by Mistor. Stinnett, Leslie Batlutz, Brady Hyan, and Lizzie Siriani.