The Munk Debates Podcast - Be it Resolved, AI research and development poses an existential threat

Episode Date: June 29, 2023

With the debut of ChatGPT, the AI once promised in some distant future seems to have suddenly arrived with the potential to reshape our working lives, culture, politics and society. For proponents of AI, we are entering a period of unprecedented technological change that will boost productivity, unleash human creativity and empower billions in ways we have only begun to fathom. Others think we should be very concerned about the rapid and unregulated development of machine intelligence. For their detractors, AI applications like ChatGPT herald a brave new world of deep fakes and mass propaganda that could dwarf anything our democracies have experienced to date. Immense economic and political power may also concentrate around the corporations who control these technologies and their treasure troves of data. Finally, there is an existential concern that we could, in some not-so-distant future, lose control of powerful AIs who, in turn, pursue goals that are antithetical to humanity’s interests and our survival as a species. Arguing for the motion is Yoshua Bengio, one of the leading worldwide experts on AI whose pioneering work in deep learning earned him the 2018 Turing Award, often referred to as “the Nobel Prize of Computing. Yoshua’s debate partner is Max Tegmark, an internationally renowned cosmologist, global leader in machine learning research, and a professor at the M.I.T.  Arguing against the motion is Yann Lecun. Yann is an acclaimed computer scientist of mobile robotics and computational neuroscience, the Silver Professor of the Courant Institute of Mathematical Sciences at N.Y.U. and Vice-President, Chief AI Scientist at Meta. His debate partner is Melanie Mitchell, a bestselling author and world-leading expert in the various fields of artificial intelligence and cognitive science at the Santa Fe Institute.   The host of the Munk Debates is Rudyard Griffiths - @rudyardg.   Tweet your comments about this episode to @munkdebate or comment on our Facebook page https://www.facebook.com/munkdebates/ To sign up for a weekly email reminder for this podcast, send an email to [e-mail].   To support civil and substantive debate on the big questions of the day, consider becoming a Munk Member at https://munkdebates.com/membership Members receive access to our 10+ year library of great debates in HD video, a free Munk Debates book, newsletter and ticketing privileges at our live events. This podcast is a project of the Munk Debates, a Canadian charitable organization dedicated to fostering civil and substantive public dialogue - https://munkdebates.com/ Senior Producer: Ricki Gurwitz Editor: Kieran Lynch  Become a Munk Donor ($50 annually) to get 72-hour advanced access to the full length editions of Friday Focus and Munk Dialogues. Go to www.munkdebates.com to sign up. Hosted on Acast. See acast.com/privacy for more information.

When you're a journalist and people don't trust you, it's always your fault. These people need to be represented. They are Canadian. They deserve to have a voice and a seat at the table. It is time to go back to the office, and the time is now. Russia had reasons to be concerned. They had reasons to be fearful. We're at an absolute turning point in reproduction. This is the problem with realism. They just treat all countries the same. They don't distinguish between dictatorships and democracies. Welcome to the Monk Debates. Our mission is to,

provide you with a civil and substantive debate on the big issue of the day to arm you the listener with enough information to make up your own mind. Today's debate, be it resolved. Artificial intelligence research and development poses an existential threat. How history of AI has been a history of failed predictions. It's naive to think that just because you make something smart, it's only going to suddenly care about humans, you know, ask some willy mammoths if they feel so reassured they were smarter than them. The printing press was going to destroy society. And it did. It totally did.

For the better, it enabled the enlightenment. I see where we are going and I don't like it. Hi, I'm Rudyard Griffith's moderator of the monk debates. Welcome to this special edition of our podcast. On this episode, we're going to feature the best moments of

the sold-out monk debate on artificial intelligence, which took place in front of an audience of 3,000 people in downtown Toronto. at Roy Thompson Hall. The motion before the house, be it resolved. AI research and development poses an existential threat. Arguing for the motion was Joshua Benjillo,

one of the world's leading experts on AI, whose pioneering work in deep learning earned him the prestigious Turing Award, often referred to as the Nobel Prize of Computing. Joshua's debating partner was Max Tegmark, an internationally now cosmologist, a global leader in machine learning, and an influential professor at MIT. Arguing against the motion was Jan Lacoon,

chief AI scientist at Meta, parent company for Facebook, Instagram, and all those other apps that over 3 billion people use daily. He too was a winner of the Turing Award for his contributions to computational neuroscience. Yan's debate partner was Melody Mitchell, best-selling author, world-leading expert, in various fields of AI and cognitive science at the Santa Fe Institute.

As with all our Monk debates, we pulled the audience prior to the start of the back and forth to see where they stood on the motion. 67% believed that AI research and development did pose an existential threat, whereas 33% disagreed. How would you vote on this debate if you were in the audience? Well, now's your chance. Write down, yay or nay, pro or con. be it resolved, AI research and development poses an existential threat.

You can compare how you voted at the end of the debate versus your vote now and see what our audience did. How much did they change their minds over the course of this fascinating debate? Well, let's join the debate in progress with Max Tegmark, giving his opening remarks in favor of the motion. Thank you. Technology can be used for good. and for bad, and its power is growing exponentially,

which means, of course, that its blast radius, the amount of damage it can do to our society is also growing exponentially. Back in the Stone Age, with a rock, maybe someone could kill five people. 300 years ago, with a bomb, maybe 100 people. In 1945, with a couple nukes, 250,000 people, with bioweapons, even more,

with nuclear winter, according to a recent science article, over 5 billion people. Now the blast radius has risen up to about 60% of humanity. And since, as we'll argue, superhuman intelligence is going to be way more powerful than any of this. Its blast radius can easily be 100% of humanity, giving it the potential power to really wipe us out. Now, to stir things up, I have a tweet here from,

friend John Lacoon, who says that making AI safe is going to happen, as with every new tech, and it's going to be a process of iterative refinement. Now, I think iterative refinement is great for less powerful technology, like jet engines and less powerful AI. Well, you have the luxury you're trying many times, but I think it's a terrible strategy when you're talking about things as powerful as nuclear winter or some future AI that we could completely lose control over, because then you really only get one chance.

You have to get it right the first time and can't come back and iteratively refine. Now, why is it that superhuman AI can be so much more powerful than all those other technologies? It's because, by definition, it can do all the intelligent things that we humans can do just better. For example, it can do goal-oriented behavior.

It can persuade, manipulative, hire people, start companies, build robots, do scientific research. It could, for example, research how to make more powerful bio-weapons, or how to make even more intelligent systems so it could recursively self-improve itself. It also could do things that humans cannot do at all. It could very easily make copies of itself. So if you imagine, for a moment, a superhuman human. AI that can think, say, a thousand times faster than a human researcher, so it can in nine hours

do a year's worth of research. But now instead think of a million of those, a swarm of a million superintelligent AIs, where as soon as one of them discovers something new, it can instantly share that new skill with all of them. That's the kind of power we're talking about here. And finally, superhuman AI will probably be a very alien kind of intelligence that lacks anything like human emotions or empathy. So, of course, it would have the power and potential to wipe us out if it had those goals.

But why would it possibly have those goals? Well, the most talked-about way in which this could happen is if we've given it some sort of goal that it actually faithfully obeys, but that goal just isn't fully aligned with human goals. This is the way we humans have usually wiped out other species. For example, when we wiped out this guy here, the West African black rhino, you know, it wasn't that we hated rhinoceroses and were deliberately trying to kill them. It's just that some humans had this weird goal that they thought that if they ate ground-up rhino horn,

their sex life was going to get better, even though this was totally debunked in the medical literature. But the humans were smarter than the rhinos, and so it was the human's goal that prevailed. Or if you chop down the rainforest for the goal of making money, you know, when you extinct species that was living there, another example of what could happen to us, if it's rogue AI. At the top here, we have a second route to extinction, which could be malicious use.

So to stir things up more, have another tweet from Jan, where, you see, he says, we would obviously never design AIs to have that sort of goals. But who is we? I'm sure you wouldn't, Rion wouldn't do it, because I know personally he's a very nice guy.

And to you in the audience, it might seem really weird that anyone would do it because you're all so nice because you're Canadian. But in the U.S., we regularly have mass shootings where people deliberately try to kill as many people as possible. So if there is some sort of super-powerful open source AI, you know, a person who wants to kill as many as possible might well use that. And the AI is safe in the sense that it's totally obeying its owner now, right? Still causes disaster. The third route to disaster I placed here is just that we get out-competed. This is the one that's the least discussed, and I would love to hear from John and Melanie what their plan is to avoid the one in the middle

and the one at the top. Out-competed. Well, by definition, AI can do, if it's superhuman, all our jobs better, right? So companies that don't replace their humans by AI, will get out-competed by those who do. Similarly, it's about delegating not just jobs, also decisions, so companies that don't have an AI CEO will be out-competed by those who do. Militaries that don't have AI generals will be out-competed by militaries who do.

Countries that don't have AI government will be out-competed by countries who do. And we end up in a situation where we humans have quite voluntarily just given away more and more power to AI. And we have this future where all sorts of stuff is happening, but it's no longer our future. We have been disempowered, giving away control to machines. that don't even need us for anything. That's a very bad situation. Let's not go there. Thank you, Max Tegmark.

Terrific opening statement. Jan Lacoon, you're up next with your opening statement. Okay, well, I first want to thank Max for showing some of my tweets. I won't have to mention of that. Okay, so first of all, we are still very far from having human-level AI. The current technology is very limited. We have systems that can pass the bar exam, but any 10-year-old kid can learn to clear up the dinner table

and fill up the dishwasher in minutes. Convincing them to do it is a different story. Any 17-year-old can learn to drive a car in about 20 hours of training. We still don't have domestic robots. We still don't have cell-driving cars, at least level-five side-driving cars. So we still have some basic, you know, major things to progress to make to reach machines that can reach human-level AI. We're missing something big.

There are immediate risks to AI, but those risks are not new. There are things like, is AI going to be able to generate a lot of misinformation or convince people to do bad things or give information to people that they shouldn't have. We already have the Internet. We already have social networks. We already have countermeasures against things like disinformation and hate speech and things like that. And by the way, all of those countermeasures

make massive use of AI. AI, for those problems, is the solution. It's not the problem. So now, is there a basic design for AI that will make it safe? It's terrible. Now, if we extrapolate the capability of the current AI system,

systems, I would agree that we might be worried about the fact that they may do bad things and be non-controllable. So most of you probably have played with Autoregose ULLM, chat GPT, etc. And you know that they confabulate things. They make up facts that don't exist. They can't really reason. They don't plan. They don't really understand the reality of the world because they're all purely trained on text.

And most of human knowledge has nothing to do with text. or with language, for that matter. So if you believe that future AI systems are based on the same blueprint, I think you're entitled to believe that they might be dangerous. And my prediction is that within five years, we're not going to be using those things anymore. So what I'm proposing is something called objective-driven AI.

So this is a type of AI whose behavior is controlled by a set of objectives. So those AI systems cannot produce an output unless it satisfies a number of constraints, safety constraints, for example, and objectives that measure whether the system answers the question they're being asked or satisfy, accomplish the tax they want to do. Those systems are controllable. They can be made safe, as long as we implement the safety objectives. And the surprising thing is that they will have emotions.

They will have empathy. They will have all the things that we require. entities in the world to have if we want them to behave properly. So I do not believe that we can achieve anything close to human level intelligence without endowing AI systems with this kind of emotions, similar to human emotions. This will be the way to control them. Now, one set of emotions that we can hardwire into them is subservience. the fact that they'll be out of service.

So imagine a future, 10, 20 years from now, perhaps. Every one of us would be interacting with the digital world through an AI assistant. This AI assistant would be our best digital friend, if you want, who will help us in our daily lives. This would be like having a staff of people who might be smarter than us. And that's great. Having, working with a bunch of people who are smarter than you

is the best thing you can ever do. People, right? So if that's the case, we would want those AI systems to be transparent and open. Kind of like Wikipedia, if you want. We trust Wikipedia because the content is contributed by millions of people who have been vetted, and there is some editing process.

And so we have some level of trust in the veracity of the content. It's going to have to be the same for AI system. the way they learn about the world, because they're going to be the repository of all human knowledge, will have to be crad sourced, which means they're going to have to be open source. And if we are afraid that they are a danger to humanity, they cannot be open source. So we're on a fork in the road. Are we going to go keep this under lock and key as if it was a weapon? Or are we going to go open?

And I really argue for going open because I think a... is going to open sort of a new era for humanity, a new renaissance, a new era of enlightenment, if you want. Everybody would be smarter for it, and being smart is intrinsically good. So I have a positive view, as you can tell, and I think there is a very efficient way or a good way of making our system safe. It's going to be arduous engineering, just like making turbojet safe. It took decades. And it's hard engineering, but it's doable. Thank you very much.

Thank you, Jan Lecun. Two terrific opening statements. Let's go for another, Joshua Benjo. You're up. Six minutes on the clock. Thank you. Let me start by stating something that's probably obvious

for many researchers in AI and neuroscience, but for many people, it isn't, which is that your brain is a machine. It's a biological machine. and that's what neuroscientists are trying to figure out. And what it means is that there's no scientific reason to think that we couldn't build machines as smart as humans in the future. Well, when exactly is another question.

But as Jeff Hinton has explained in a recent talk, digital computers have advantages over analog brains. Analog brains also have advantages in terms of computer energy efficiency, but in terms of the ability to absorb large quantities of data and to share information at high rates between computers so that they can learn in parallel as Max was talking about, so you can have thousands of computers sharing what they've learned. For example, right now we already have AI systems

that can read the whole internet very quickly, which a human couldn't do in many 10,000s of lifetimes. So this gives them advantages if we are able to, to build machines that have the same principles of intelligence as we have, and that means it's very likely that we'll build machines that are superhuman. Now, when is that going to happen? Well, Jeff Hinton and Jan Lequin and myself, who won the Journey One for Deep Learning, actually all agree that it may be anywhere between just a few years and a few decades, something like five to 20 years with some confidence interval. Now, if it's a few

decades that may be reassuring, but as if a few years I think we should be concerned. And in any case, it doesn't change the question of tonight, which is, is there existential risk? It's not about how far into the future it is. I changed my mind about this whole question because the time scale changed. If you had asked me just a few years ago, I would have said, well, maybe a few decades or centuries, because I thought, like Jan, like Melanie, that it was a few years. it was too far away.

Now, I'm sure many of you have tried ChatGPT or GPD4, and you can't really avoid noticing that these machines are incredibly powerful. In fact, they pass what Turing, who gave his name to the price we won, defined as the Turing test. In the sense that when you dialogue with these systems, you may not be sure if you're talking to a human or a machine. Now, it's been also the job of many scientists like myself to try to figure out what is missing.

And I agree with Jan and Melanie that there are important ingredients missing, but are they something we'll figure out in just a few years or a few decades? It's hard to say. I can tell you that the things that I'm working on, which is something on the order of reasoning, right now these systems are really good at something like intuitive intelligence, but not so much at reasoning and thinking through before they're saying stupid things, is something that could potentially be solved very quickly, or maybe there will be obstacles on the way. We don't know. So this is a problem, and as Max said,

once the recipe for building these things is available or even worse, the parameters, the weight, so somebody could download this, and give it instructions that could be very, very harmful. Because imagine systems that are many times smarter than us could defeat our cybersecurity, could hire organized crime to do things, could even hire people who are legally working on the web to do things, could open bank accounts, could do all kinds of things

just through the internet, and eventually due to R&D to build robots and have its own direct control in the world. So in a way, where we're going is, and as I'm channeling Jeff Hinton here, you know, coming up with a new kind of entity in the world that may have its own self-preservation goals, which is something that also Max explained why it may happen. It may happen because somebody asks it to do something bad, or it may happen because of what I call a Frankenstein's tendency of us wanting to build machines that are like us.

And once we have machines that have a self-preservation goal, well, we are in trouble. Think about what happens when you want to survive. You don't want others to turn your off, right? And you need to be able to control your environment, which means control humans. So existential risk isn't just, well, we all disappear. It might be that we're all disempowered, that we are not anymore in the control of our destiny. And I don't think this is something we want. I'm writing a blog post, which I'll post soon, where I've tried to take all of the comments and questions

that people have asked me about this question in trying to answer all the critical. about, oh, we should not worry about existential risk. And up to now, I, unfortunately, I haven't been convinced by these arguments. I would really like to be convinced and lay these concerns to rest, but all of the arguments I've heard and we'll discuss tonight have not been good enough.

Thank you, Joshua Benjillo. Let's have Melanie Mitchell to give our final opening statement, six minutes on the clock, Melanie. Fears about machines unleashing human extinction have deep roots in our collective psyche. These fears are as old as the invention of machines themselves. But tonight, we're debating whether these fears belong in the realm of science fiction and philosophical speculation, or whether AI is an actual real-life existential threat.

I'm going to argue that AI does not pose such a threat in any reasonably near-finding. future. First, I'll argue that the possible scenarios that people have dreamed up for AI existential threats are all based on unfounded speculations rather than on science or empirical evidence. Second, while we can all acknowledge that AI presents many risks and harms, none of them rise to the extreme level of existential, saying that AI literally threatens human extinctions, that's a very high bar. Finally, claiming that AI is an existential threat is itself harmful.

It misleads people about the current state and likely future of AI. Such sensationalist claims deflect attention from real immediate risks and further might result in blocking the potential benefits that we could reap from technological progress. Let's look at the three scenarios

that people have posited for AI to be an existential risk. The first is that a malevolent, super-intelligent AI somehow emerges and uses its evil genius to destroy humanity. We've all seen that movie. I believe that no one here takes that scenario seriously. For the foreseeable future, AI systems will not have their own desires or intentions for good or for evil the way that living beings do. They're not alive.

The second scenario is also that a super-intelligent AI emerges, but it's not malevolent. It just misinterprets our wishes and accidentally kills us all, sort of like a sorcerer's apprentice gone nuclear. For example, Yelsho Abenjio wrote about this thought experiment. We might ask an AI to fix climate change, and to solve the problem, it could design a virus that decimates the human population. Presto. Humans dead. No more carbon emissions.

This is an example of what's called the fallacy of dumb superintelligence. That is, it's a fallacy to think that a machine could be, quote, smarter than humans in all respects, unquote, and still lack any common sense understanding of humans, such as understanding why we made the request to fix. climate change and the fact that we'd prefer not to be wiped out. Intelligence is all about having insight into one's goals and the likely effect of one's actions. We would never give unchecked autonomy and resources to an AI that lacked these basic aspects of intelligence. It just does not make sense.

The third scenario is that a genocidal group of humans uses AI to help them destroy humanity. Indeed, humans often use technology to do very bad things, but we can't conclude from this that AI research and development is itself an existential threat. A terrorist group could conceivably carry out a nuclear biological attack that kills millions of people with or without AI. There's information online right now about how to make weapons, one might go about killing millions of humans.

AI systems could make it easier to get that information, but the threat is still there without AI. But more importantly, our society, our institutions, and our technologies are enormously complicated, diverse, and resilient. They create a barrier of complexity that puts the brakes on such an attack, which would require a cascade of highly improbable events. Everyone here would agree that there's a long list of actual near-term risks and harms of AI, such as the spread of disinformation or job losses. We should take those risks very seriously. But it's also harmful to take unfounded speculations about existential threats too seriously.

Consider vaccines, which provide immense benefits in mitigating diseases. But we've seen calls for them to be severely restricted and even halted due to unfounded speculations about their risks. We don't want to kill off potential benefits from AI in science, in healthcare, or education. Would you call science itself an existential risk? It gave us nuclear weapons, after all.

But not pursuing science is an even greater existential risk. Just as for vaccines and other technologies, our assessments of AI threats need to be founded in science and empirical data, not unsupported speculations. There's no evidence that AI research and development poses an existential threat, now or in the reasonable future. Thank you. For a reminder to our Monk Debates supporters and curators, right now on our website, www.com.

You can access a high-definition video version of the entire main stage. Monk debate on artificial intelligence, which took place recently at Roy Thompson Hall. simply log on to the website using your membership credentials. Go to the page for the artificial intelligence debate. You'll see a big red arrow there. Click on it and you've got 90 minutes of in-depth insight and analysis of artificial intelligence. If you are not already a supporter or a curator and you'd like to get access to on-demand versions of not just the artificial intelligence debate, but our 10-plus year archive of,

debates, dialogues, and conversations do that right now by becoming a monk debate supporter or curator. We'll also throw in advanced ticketing privileges to all of our in-person debates and other great perks and privileges. Again, you can do that right now on our website, triple-w monkdebates.com. Now back to our program. Thank you, debaters. The table is now set for rebuttals. So we're going to go around the horn, three minutes each in the same order. Max, you're up first. Thank you. This is very interesting.

So first of all, Melanie, we saw here that 67% of the audience already thinks this is an existential threat. So just coming out and saying it's not is an extraordinary claim, which I think requires extraordinary evidence.

And I would love to hear from you and Jan what your actual evidence is that there is no risk. So I would love if you can get a bit more nuanced and safe, instead of just saying, oh, it's unfounded speculation, you know, and it's things like, and it's sci-fi, tell us, what do you actually think the probability is? They are going to get superhuman intelligence, say, in 20 years, say, in 100 years.

And what is your plan for how to make it safe? What is your plan for how we're going to make sure that the goals of, and AI are always aligned with humans. How are you going to avoid the malicious use case that I brought up? None of you said anything about that. John, you again reiterated that you want to make sure that we give good goals to AI. I'm completely confident that that's what you would do. But if someone is a terrorist or just wants to take over the world for their own benefit,

what is your plan, actually, for stopping them from not putting very submissive goals into their AI and putting in any goals that are going to make that AI take over the world for them. I would like to hear some details. And finally, what do you each think, actually, that the risk is? We're here to debate whether there's a threat, right? We're not debating whether there's 100% certainly they weren't all going to get wiped out. That is not what Joshua and I are arguing. We're just arguing that the risk is not 0% and that it's too high for our comfort levels.

It would be wonderful if you can actually tell us, what do you each think? Do you think it is really 0% that this wipes us out, or do you think it's one in a million that it's going to pose an existential threat, or 10% or 1%? When we can get by, past slogans, like just saying it's speculation and sci-fi and get into these nuances, I think this is going to be really, really helpful and productive. And finally, I wanted to say to you, John, And it was actually great to hear that you actually admit that you feel that it is a real existential risk

if we just go with today's auto-regressive technology and just, like, for example, GPT-5, GPT-6, C-P-T-7. And what you see is the hope is that we switched to some better technology, right? So if I misinterpret it, please clarify. If not, isn't there still a risk, though, that some companies are going to continue doing it? the GPT6 kind of way because they already are. How do you ensure that people actually only use the Safeway? Yeah, and Lacoon, your opportunity for a rebuttal. Okay, let me answer this one just right away.

No, GPT whatever is not at the existential risk. It's a risk. It might be dangerous, might be not as useful as people make it to be, but it's not going to be the existential risk. It's not going to wipe out humanity. No. You'll need something considerably smarter than this, and that's what I said. We're missing something big to make systems really intelligent.

So the first thing I want to say is that, yeah, the science fiction scenarios of, you know, the Earth being wiped out, humanity being wiped out, this sounds like a James Bond movie, right? It's like the supervillain who, like, goes in space and then, you know, kind of puts, like, some deadly gas and eliminates all of humanity. It's a James Bond movie. And I can't disprove it. The same way, if I tell you, I use the Bertrand Russell. idea. If I tell you there is a teapot flying between the orbits of Jupiter and Saturn,

you're going to tell me I'm crazy, but you can't disprove me, right? You can't just prove that assertion. It's going to cost you a huge amount of resources to do this. So it's kind of the same thing with those doom scenarios. They're sci-fi, but I can't prove that they're wrong. But the risk is negligible, and the reason it's negligible of extension is because we build those things. We build them. We have agency. This is not superhuman intelligence. It's not something that's going to just happen. It's something that we are building. And so, of course, if it's not safe, we're not going to build it, right? I mean, will you build a bomb that just blows up randomly? No, right? Okay. So I think a lot of the fears around AI are predicated on the idea that somehow

there is a hard take-off, which is that the minute you turn on an AI system that is capable of human intelligence or super-intelligence is going to take over the world within minutes. And this is preposterous. It's one of the tweets that you posted that I wrote. The reason it's preposterous is that it's just not the way anything works in the world. You build something, you build it small, you don't make it super-intelligent right away. You make it as smart as a mouse. And then you figure out if it behaves properly, and then you make it as smart.

as a cat, and then a dog, and then something a little bigger, right? And of course you do this progressively, iteratively. That's what I was referring to in that tweet that you posted that I wrote. So, you know, it's going to be engineering. It's like asking today if we're going to, you know, precisely how we're going to make super intelligence system safe. It's kind of like asking in 1930 whether we're going to be able to be able to be able to be able to whether we're going to be able to build turbojets that are incredibly reliable,

so irreliable that you're going to be able to cross the Atlantic at near the speed of sound. This would have sounded impossible, but we did it. Thank you. I'd like to go back to the teapot. Well, there's no evidence for a teapot, but there is an asteroid coming to us. It's not clear that it's going to hit the Earth, but we're seeing a very clear trajectory of improving,

improvements in intelligence of the systems we've been building steadily over the last two decades or more. And you and I have been part of that. And yeah, I agree with you. It's not yet arrived. But does it mean that we should, you know, do nothing? In fact, it's interesting that you've been proposing solutions to the safety problem, which means you believe that we need to build safe AI. Yes.

which means that there is a problem that needs to be fixed. Yes. I mean, also, you just said that if it was dangerous, we wouldn't build it. Well, let me remind you of a few things that we've done. I mean, collectively, not you and I, of course, that are existential risks, you know, fossil fuel companies. for many decades have known and hidden the fact that their activity could be highly destructive for the planet.

And, well, it was the profit motive. Companies are actually acting in a way that is not quite aligned with what society needs. And in fact, there's an interesting analogy between that kind of behavior and AIs. We are asking companies to do things that are useful for society, you know, produce goods and services we need. But we can't exactly tell them to do that in a formal way.

So we tell them maximize profit and stay legal, pay your taxes. But that recipe, although in principle, if everything, you know, old companies were microscopic and everything was good and there was no environment would be good. But there's a mismatch between what companies are trying to achieve and what society really needs. So how do we deal with that? Well, we need governments to intervene to try to reduce that. And we need to understand the problem.

In fact, what Max and I and others are saying is not necessarily there's going to be a catastrophe, but that we need to understand what can go wrong so that we can prepare for it. We can build safe AI systems. The other thing is that you seem to think that everyone is like you. But there are a lot of people out there who can have all kinds of motifs that could be very dangerous for everyone. So, Melanie, you said that we don't yet have existential risk. And I kind of agree, but how do you know that in two, three, five, 20 years it's not going to be the case? Joshua, we're not debating whether or not there's a problem to be fixed or whether there's any harms.

we're debating whether there is an existential threat from AI. That's the resolution. And if the existential threat is, you know, 100 years, 500 years from now, I don't think then we would be here debating. You're talking about something very near term. And Max asked, like, what is the risk? I don't know. I mean, the risk is non-zero.

The risk of anything is non-zero. You know, we could talk about any kind of scenario, speculative scenario, like, you know, malicious aliens coming to Earth in their spaceship and destroying humanity. That's an existential risk. And then we could say, wait a minute, maybe we should ban radio broadcasts because that's how they're going to find us. But clearly, the probability of that is quite low.

It's not high enough to justify the kind of attention that you're seeming to ask for. Now, I want to say, you know, there's this word, superintelligence, smarter than humans, these phrases that are being glibly thrown around as if we understand what they mean. Yahshua said, you know, your brain is a machine. Sure, I agree with that. So we could build a human level intelligence in principle. I also agree with that.

I think we're quite far away, but in principle. But human intelligence is not any old machine. It's a very special kind of biological machine that's adapted to our human. problems. We're different from octopuses. We're different from rats. We're different from viruses. We have our own specific human problems, needs, and motivations. And we're fundamentally embedded in a physical, cultural, social system. And that's the kind of thing that gives rise to human-level intelligence. AI systems, while they learn from human data, and that's exactly all they learned from is the data that humans created. And so they have captured some aspects of our intelligence. They lack

fundamental, important aspects of what it is to understand the world. And I think that when we throw around words like superintelligence, we're using sort of our intuitive kind of associations and assume that these things could be easily built and that we're on a trajectory to build them. But really, that's not been shown at all. I don't think it's something that science really gives much evidence for now. And I'll just conclude to say that the whole history of AI has been a history of failed predictions. Back in the 1950s and 60s, people were predicting the same thing about super-intelligent AI and talking about existential risk. but it was wrong then, and I'd say it's wrong now. Thank you.

Well, let's thank our debaters for a terrific opening for this debate. I'm going to join the conversation now and try to think up some questions that are top of mind for our audience. And speaking with the debaters before, we all agree that we really want to center this debate on that key word in the resolution, which has been brought up many times tonight, which is existential. So let's throw a definition just up onto the screens here, just to kind of center ourselves for this portion of the debate. This is from Nick Bostrom, a kind of leading thinker on existential risk. His words, an existential risk is one that threatens the premature extinction of earth-originating intelligence or life, or the permanent and drastic destruction of its potential for desirable future development.

So I think we all agree that's a relatively fair formulation of what we're debating tonight. So let me come to you first, Max, because you were the first of talk in the last round, and just to ask a question I think that is on the minds of a lot of the audience members, which is to try to help us understand why would, whatever we want to call it, an AI system, machine intelligence, why would it want to harm us? Why would it have an intention to do something to us? Can you give us that answer? Yeah, let's not anthropomorphize by talking about wanting.

Why might it have goals to do bad things like this? It's either because of malicious use that some user actually gives it those goals. Not everybody is as kind as Jan and Melanie. Or because we specified some other goal that we thought was kind of getting it right. but it turned out when it's pushed to its extreme, does great harm. So in both cases, the AI is very loyal to us, and it's just that the goal was off,

or the human goal was put into them off. The third one, which I have heard no counter-arguments to whatsoever, is the one we just get out-competed. That's kind of the trajectory we're already on right now. Everybody just wants to make a buck. Companies have a goal to make profit, and we gradually get more and more disempowered. So those are three roots to it.

And I would love again if Jan and Melanie can answer this question. What do you think the probability actually is that we're going to build machines that can do most of our stuff in 20 years? What do you think the probability really is that we're going to have an existential risk? And what's your plan for avoiding these three different threats? Because we can't just dismiss arguments as being ridiculous or sci-fi when people like Jeff Hinton are making them. The CEO of Open AI is saying it's an X-risk. The CEO of Deep Mind is saying it.

What is your plan actually for how we avoid each of those things? So which authorities do we want to pay attention to? I don't think we have proof by authority here. That's not a fair argument. Right, so let's say we don't know. And I'm agnostic actually. I don't know what's going to happen. But given the stakes, shouldn't we pay attention to this?

Right. So, yeah, this is an interesting analogy, let's say, Joshua brought up earlier, climate. There's an idea of a precautionary principle, Jan, that you do certain things because you want to be careful. Maybe not what you want to do, but you take steps and actions now to avoid to close off certain worst-case scenarios.

Why isn't this an example, like climate, AI, is one where we want the precautionary principle to be in effect to deal with, maybe it's a tail risk only, but this tail risk of an existential threat. But that's exactly what we're doing. We're developing technology and testing it, making sure it's safe before deploying it. I mean, current AI systems are pretty much deployed this way.

There's no reason for this to change as the power of those systems progresses. But, you know, there's several questions. You know, if bad guys can use AI for bad things, there's many more good guys who can use the same, more powerful AI to counteract it. Right. So in the end... How do you know the good guys are going to win? Sometimes the attacker has the advantage.

There's absolutely no reason to believe that's the case for AI. Really? Really? We are facing this interaction right now. How do you know? I mean... Because it already exists. Because people are currently using AI.

No, no, we're talking about machines. Hold on a second. This is a really interesting point. So let's have Yan sum up on this, and then I'll come to you, Yosh. No, no, it's good. But I like the crosstalk. But these are complicated ideas. I'm struggling to keep up, so I know the audience might be too.

Because some of the dangers that you talked about of people using AI for bad things, they already exist. And Melanie said that before. Those threats exist with or without AI. AI might help a little bit. Let's say misinformation. Okay, is AI going to help with misinformation? Well, you know, QAnon is two guys.

They don't use AI. They have a big impact. The counter measures against misinformation, hate speech, et cetera, and propaganda attempts to corrupt the electoral process and democracy. Again, as I said before, the solution is AI to take down this content on social networks and various other channels. AI is used massively. So to take a precise example, the proportion of hate speech taken down automatically by AI systems about five years ago on Facebook was about 25%.

Last year, it was 95%. Right, but we need 100% to avoid extinction. This is, no, we don't. There's never anything that is completely perfect. That's the issue. That's why we need to do more than what we're doing now. Again, it's the good guys AI, which is superior to the bad guys AI. But that's like saying that the way to stop a bad guy with a bioweapon is for it to have a good guy with a bioweapon.

That's not what you do. The way you stop a bioweapon attack is with vaccines and banning bio weapons and having various forms of regulation and control. No, the way to stop bioweapon is to have counterintelligence because you can't do this in secret too much. And it's the same if you use AI. It's not going to make your... Exactly. And how do we build that counterintelligence? We're going to build...

we're going to need, sorry, the infrastructure to protect ourselves. And so for that, we need to recognize that there is a risk. There is a risk. Oh, thank you. No, there are a lot of risks. But the question we're asking is, are they existential? Are they going to end civilization? It could.

And that's, and I think, you know, Roger brought up the precautionary principle, which says, you know, well, if there's a risk, we should probably do something about it. And I'm, you know, I'm not opposed to that. but I think that we have to balance. We only have so much resources, so much attention, and the whole narrative on existential risk in AI has diverted a lot of the attention of the world, maybe part of the 67% that are here saying

that they think is an existential risk. It takes our attention away from some of the real immediate risks like disinformation and bias. So are you saying it's a 0% risk, 0% existential risk? Is that your claim? Do I think there's a 0% risk of any scenario?

No, of course not. What do you think it is then? 1%. What we're really debating? I can't put a number on, and I think it's quite low. And I think what we're debating, is there a reasonable

existential risk, a risk of end of civilization in the reasonable future? Otherwise, we wouldn't be up here debating. How high is too high to you? I'm not going to like say 0.000,000. I mean, I can't say that. How high is too high for you then?

How high is... How high a probability of a list is too high for you? I don't think we can put a probability on it. We don't know. We don't know enough. No, I'm asking what you think is acceptable. Okay, it's lower than the Earth being white-tile by a meteor? And by the way, AI can help with that problem.

By the way, Jan... But let's just bring the audience here. So in a sense, what we're hearing here from the cons of the debate is that the risk is so, in a sense, negligible that we should move on to other things. The development of AI with a variety of controls, with regulation possibly, but to assume that there's any demonstrable existential risk of a quantum that would require a pause, a moratorium, as both of you has argued, is not on the cards. Let me argue a little bit, if you're pleased, about these probabilities. Let's consider very simple scenarios. There are more complicated ones, for example, that Max and I have been talking about.

But let's consider first the probability that we will know how to build intelligent machines that are powerful enough to be dangerous existentially to us in, say, 10 years. Okay? That probability, when I ask a lot of AI people, they agree that it's significant. You know, it's 20, 30, 40, 50 percent. And people like Jan and Jeff and many others that I've asked agree that it's clearly not like tantamize something, right? It's not like asteroids coming and destroying the Earth. That's the first thing we need.

Wait, let me finish. Second, what's the probability that once this is available and known and open source, so that anyone can play with it, what's probably that there will be someone with misguided intentions or malicious intentions, which uses it and we get catastrophic consequences, but possibly leading to extinction, or at least the same scale as we're talking about

nuclear weapons, explosions and things like that. So what's the penalty for that? Most people I asked is like 99.9% Once it's available, somebody's going to misuse it. So I don't understand how they can get with very small probabilities. Now, the Jan's answer is, oh, it's okay. He agrees more or less with what I'm saying, but he says, we'll fix it before we get there. Yes, let's do that.

Okay. So I think, you know, you say you talk to, the people you talk to in AI have this belief. Well, the people I talk to an AI don't have that belief. I think the field is quite split, I would say. And I think there's quite a... Flip a coin at 50% that we're all going to die? Have you talked to any of the people who signed this letter saying that this is an existential risk?

There's many people who have signed letters saying they think it's an existential risk. They don't say over what time scale. But this was Jeff Hinton. Yeah, Jeff Hinton is a godfather. but he doesn't know everything. You don't talk to him. Yeah, I think that he's smart guy,

but I think that a lot of people have way over-hyped the risk of these things, and that's really convinced a lot of the general public that this is what we should be focusing on, not the more immediate harms of AI. And that itself is harmful. Let's, good answers, everybody. Let's just follow this risk question

and come to you, Jan, on this, because you're a scientist and you're at meta and you've got 3 billion active daily users. It's hard to wrap my mind around that number, but you've got big data and all kinds of really interesting ways to deploy this technology. Historically, scientists at times have made decisions to engage in risky behavior because they want to do things.

They want to innovate and they want to discover. There are hair-raising tales around the Manhattan Project, and probabilities of igniting the entire atmosphere of the planet with the first Trinity test, a very low risk, but they still went ahead and did the test. Why are you confident that this time scientists in the scientific community, writ large, around the world, is going to approach this with the prudence that you are,

and with the care and caution that META is going to bring to this challenge of safe development of AI? Okay, so first of all, it's not a question for MEDA because the progress, I mean, the question of really what is intelligence is a very deep scientific question that will take the effort, the combined effort of the entire scientific community. So this is not going to happen within the confines of a small lab or a big lab. This detects everyone. There's one reason we need this research to be open, by the way. But the point is humanity is still around, despite all of playing with all those toys like nuclear bombs. Now, the assimilation of AI with nuclear bomb, I think, is a complete fallacy. The assimilation of AI with nuclear weapons is a complete fallacy. Nuclear weapons are designed to kill people and destroy entire cities.

AI is designed to basically amplify human intelligence. Like, this is intrinsically good, right? Well, it depends who uses it. Of course, it can be dangerous. The same way of flying on an airplane that is not properly engineered is dangerous. Or a submarine. Or a submarine, for that matter. Yeah.

Not properly maintained. Ooh. Okay. That's a sad. So to ensure that things that are deployed in the public is safe, you need safety testing, you need certification. and I'm all for that, you know, we need regulation for various things. They already exist. We all have automated systems that can drive our car in the highway.

You know, soon they'll drive our car completely automatically. Those have to go through certification. They don't get into your car just because the manufacturer decides it's good. AI systems are used to help medical diagnosis for, you know, imaging or for various other applications. They have to go through approval by the regulatory. agencies. So we already have those regulations. That's good. And, you know, we're talking about the risk of AI, but like, what about the benefits? You know, everything is a trade-off between risk and benefit.

And what Melanie said is, you know, if the risk are infinitely small or negligible, you know, we have to consider the benefits. Like, what about killing all the benefits? The progress that we can make in science, in medicine, in technology. Well, let's bring in Max. You want to get in on this point. Yeah. So you just said that intelligence is intrinsically good. I disagree.

Intelligence is a tool that makes you able to accomplish more good things or bad things. Let me finish. What are you talking about? If Hitler had been more intelligent. You're a professor at MIT. Yeah? You are a good person, so your morality combined with more intelligence makes the world better. But if Hitler had more intelligence, I think the world would actually have been worse.

How do you know you don't have Hitler on your class right? It's naive to think that just because you make something smart, it's only going to suddenly care about humans. Ask some willy mammoths if they feel so reassured that we're smarter than them. Then, therefore, we would automatically adopt mammoth ethics and do things that were good for the mammoths. We didn't. That's why you probably haven't met any. I also want to say that if anyone in the audience works in biotech, maybe you could raise your hand, you know, this whole discussion in the last 10 minutes must sound kind of weird for you.

where we're like, we're going to keep building this unless someone can prove to us that it's dangerous. Because in biotech, it's exactly the other way around. If you come up with a new medicine, you'd say, this cures cancer, it's awesome. You can't just go sell it in the supermarket until someone proves it is dangerous. It's your job to convince the Food and Drug Administration,

the U.S. or the Canadian authorities or whatever, that this is safe, and that the benefits, yes, the benefits outweigh the risks. You also cannot, if you build a... someone can't just come in and build a new weird hitherto unseen design for a super powerful nuclear reactor right underneath the CN tower just because nobody else can prove that it's dangerous. It's their job to prove that it's safe. And we need to totally flip it around like Joshua said that with future very powerful AI systems, it should be the responsibility of the companies that to first prove that this is safe before it gets deployed. We need to become like biotech. And we will. I think we will. So you support regulation?

No, I don't support the resolution, but I think we, you know, I never said AI has no risk. Sorry, you support regulation. Oh, regulation, absolutely. I totally support regulation. And no one ever said AI has no risks. We're talking about this extremely high bar here of ending civilization, of ending humanity. But why is it our role to explain to you why it's dangerous? You still haven't answered my question.

I never said it wasn't dangerous. I've asked you twice. What is your plan for avoiding mitigation? You haven't told me. You have asked you twice. What's your plan for solving the alignment problem? I think... I haven't told me.

I've asked you twice. Let me just finish. To tell me what your plan is for avoiding the scenario where we get out competed in disempowerment. You've said nothing. So, unfortunately, I don't get paid enough to solve all these problems about AI policy. But I think... But I think... I don't have to have a plan.

I don't have to have a plan. I think the AI community is developing plans to mitigate risks. But that that's not what we're going to... we're debating here. We're not debating whether AI has risk or whether the community is going to solve them. We're debating whether AI research and development poses an existential

threat. That's something very different. And you're not answering my question. What is your plan to make sure it doesn't have an existential risk? I don't think that there is an existential risk. I think there are many risks, but there are people who are working extremely hard and including, I think, many you know, Yoshua, particularly, on mitigating the more immediate, real, real world risks.

We need, Melanie, you and I know each other, and we need regulation. We need to make sure that we get all the upside that Jan was talking about for AI. I've been working on AI because there's all this upside that, you know, is already starting to happen. And there's also downsides that you and I have been talking about, you know, for many years now, and we need to take care of that. But we need to take care of all the downsides, including the ones that seem maybe a little bit extraordinary. But I want to say that you're saying, oh, existential risk is a very high bar. So if I tell you, oh, maybe it's only going to kill one percent of humanity, you consider that's not important? That's extremely important, but I wouldn't call that existential by the definition we're talking about.

it can kill 50% and it can kill 100%, right? I think that's very, I think killing 1% would be a complete and utter catastrophe, and we should do everything we can to prevent it. But that's not existential. It's hard to kill 8 billion people. Well, but the reason why it could kill so many people, whether it's only 1% or 100% is because these systems, we're talking about systems that don't exist today,

that I think might be coming in a few years or in a few decades, and that these systems would be smarter than us. And so they find ways that we don't have easy defenses against. That's the scenario that I don't know is going to happen, but I think it's plausible enough that we need to worry about it, and so there is a risk. Let's have Jan come in on this point, because he wants... Thank you, though.

Two points I want to make. First of all, this is complete fallacy that the desire to dominate or destroy is linked with intelligence. This is false. It's not even true within the human species. Some humans. This is not necessarily the smartest among us

who want to become the leaders. In fact, we have plenty of examples to the contrary on the international political scene. And this desire to dominate is something that is, intimately linked with human nature because we are social animals, we are social species, and nature has evolved us to organize ourselves hierarchically like baboons, like chimpanzees. Not orangutongotongs.

Orongotong's have no desire to dominate anybody because they're not a social species. So this desire to dominate has nothing to do with intelligence. They're almost as smart as we are, by the way. So this is nothing to do with intelligence. We can make intelligent machines that are superior to us, but have no desire to dominate. I need a research lab, and I only hire people who are smarter than me. None of them want my job. Now...

But, Jan, isn't just to be fair... Second point. Just to be fair to the other side of the debate, because it's something I think the audience would appreciate understanding. It's not so much the desire to dominate, it's the control problem. It's that you've set them some goals, maybe very noble and great goals,

but they start doing other things to achieve those goals which are antithetical to our interests. It's not that they're trying to dominate. It's that there is a tragedy of the commons that goes on. And it's the same thing with companies. This is the goal alignment problem. So how do we design goals for machines so that they behave properly? And again, this is something that's a difficult engineering problem.

But this is not a problem that we are unfamiliar with because as societies, we've been doing this for millennia. This is called making laws. Right. And does it work? We design laws. Of course it works. Because we design laws to align. align our objectives with the common good so that we prevent people from doing bad things

by telling them you're going to get this punishment if you do this bad things. We even do it for super intelligent entities called corporations. It's not perfect, but it's not existential. Okay, but here is another point to, I want to respond to Max. You're a professor at MIT, which means you educate, really smart kids, you are building super intelligent machines, I mean human machines. in that case, right? Because you're educating them.

What is the probability that in your class, one of those students is the new Hitler and is going to wipe out liberal democracy? Those, as Joshua mentioned, what we can build with machines is going to be vastly more powerful, even than my MIT students. And I still love to brag about them, right? It's that scaling and also the entirely alien nature

of their minds, which is so intimidating. Not the students, the computers. Yes, exactly. And I feel a little bit like we're on this big ship sailing south from here down in the Niagara River. And Joshua is like, hmm, I heard there might be a waterfall down there. Maybe this isn't safe. And Melanie is saying, well, I'm not convinced that there even is a waterfall, even though Jeff Hinton says there is.

And moreover, I don't know how far away the waterfall is. It might be like really far away. There's big uncertainty. So I'm not really going to worry and tell you about my plan for keeping it. And then, Jan, it feels like you're saying that, yeah, it was definitely a waterfall there. We'll get there. But we're going to figure out how to make things safe, how to stay in control of our boat when we get close. We don't know yet how to make it safe.

I think we all admit this. Some of us have worked very hard on that research for close to decade. But it's going to be easy. We'll figure out when we get there. And I think in both of your cases, it would really be, if this were, again, biotech, this kind of argument is not reasonable. If a tech company says, oil companies told us, oh, climate, existential threats, whatever, from climate change, it's not a thing. That's sort of like what I'm hearing from you, Melanie. So therefore, they shouldn't be obligated to tell us why what they're doing is safe.

Sure, they should be. It should be their responsibility to tell us how they're going to. to mitigate this problem. And similarly, with biotech again, just because almost all big companies will always claim that there is no risk with their product initially. The tobacco industry told us that, too. The asbestos industry told us. Max, I think you're misstating what we're saying. We're not saying there's no risk. And, you know, I think the people, this analogy with the waterfall is, well, okay, we all know about waterfalls, that seems very reasonable. But this idea of super intelligent AI,

is not something that, you know, you're sort of extrapolating wildly from what we have to something that's smarter than us in every possible way. No, no, he doesn't need to be every possible way. Hold on. A hundred times smarter than us, maybe, and yet still misinterprets our goals. But this is a chartered cruise ship towards the fall that said that on the ticket. This has been the goal of artificial intelligence since its inception to build superhuman machines. And since the beginning, everybody's been saying, oh, the waterfall, we're almost at the waterfall, we're almost at the waterfall. We're almost at the waterfall. That happened in

1960, not by Jeff Hinton, but people like Claude Shannon and Herbert Simon, and they were just dead wrong. And I would be happy if I'm wrong, but I don't know. And you know who else was wrong? I don't know what other kinds of existential

risks are going to happen. Maybe we're going to add lots, you know, genetic engineering is going to add... But we are moving in that direction, right? That's the thing. It's not like some crazy, crazy thing that's unrelated to what we're seeing. It's very much related to the work that I'm doing. I see where we are going and I don't like it. And stop doing it.

Yeah. That's what I'm doing. And we have to be, let's be, we have to be more humble here because we made another terrible prediction, epically failed one. About three years ago, most AI researchers thought we were decades, 30 years, maybe 50 years, away from passing the Turing test. We just heard Joshua answer, argue now that this already happens. There is, no Turing test. But the prediction... Turing did not... Hear me out.

I'm just making the point that three years ago, most AI researchers that I know did not think we were going to get CHATGPT4 in 2023. This took a lot of people by surprise. Wouldn't you agree? No, absolutely not. Maybe not you, but there are actually

serious polls out there that show that most people... The basic technology for it existed three years before. So I didn't know what was going on there. And from the outside, it was a big surprise, really. It's a big surprise.

No one's denying that chat GPT and GPT4 and so are amazing, but that doesn't point in the direction of this sort of misalignment. Follow the line. I don't know if there is a line that I can follow. Well, look at the last 20 years. There is a line. There's a line in a certain direction where these systems are learning from huge amounts of human data

to be able to synthesize human knowledge. They're not able to. to feel, they're not able to think, they're not able to form their own goals. Well, that's not reassuring. That won't come soon. That's harder than intelligence, I would say. Right, and that's the problem. We don't know how to program a computer, unlike what Jan seems to claim. Right now, it's an open problem that economists, reinforcement learning researchers, AI safety people have been studying for many years, and essentially we don't know how to make a computer that will do what we intend.

And there are several reasons for this, but one of them is that we're not even able to express it in a clear way that a computer for sure will understand. And then the issue that they might go into something else that turns out because they want to achieve those goals, that they might want to preserve their existence. They don't want to do anything. They're not alive.

No, that's very easy to do. No, you're wrong. You're wrong. The systems that, for example, Chad GPT is essentially like an Oracle. It doesn't really have a want, although a little bit it wants to please us because of the reinforcement learning. Yes, it's been trained to please us. Then it's actually easy to put a wrapper around them, to turn them into agents that have goals. It's actually easy to do.

They have goals that humans gave them. Yes. Not their own goals. Yes. And in order to achieve those goals, they're going to have subgoals. And those subgoals, for example, may include things like, deception, and this is happening. That never happened. It happened already.

It did not happen. I actually studied that the example where there was an example where the New York Times in fact reported that GPT 4 had deceived a task rabbit worker. But if you actually look at the actual paper and look at the details, and I

ask you all to think more critically about what you hear in the media, a human prompted it to push it in that direction. It has happened. That was the point. They were trying to make a test to see if the system could deceive a human.

When it told this journalist that it loved it. When it told... Safe AI systems. Yeah. That was deception. When it told the journalist that it loved him and he should leave his wife, that was deception. Do you really believe that it loved the guy, or was it deceiving him? It was not deceiving his very anthropomorphic idea.

It's an intention. I don't believe it was deceiving him. It's a natural consequence of trying to achieve a goal. In order to achieve the goal, you try to find, you know, the means to the end. And just, it's not just humans. That system has no goal. Other animals do it as well. This has been a terrific debate because the moderator has been completely obsolete for the last 10 minutes.

I love that. I mean, let's do this more often. I just are remaining moments, though. I want to touch on one point that Max raised in part of his three kind of critiques. And it's to go back to the second point, maybe we can have that definition again up on the screen that we agreed on for existential risk. And it's the second point that an existential risk isn't simply an extinction event. It's a permanent drastic destruction of kind of human potential, our inability to return

to our previous track of development. Again, a very high bar. But I guess the question, Melanie, that interests me, and I know your views on this, is this idea of losing our agency. As Max presents it, the corporations that adopt AI will be more successful and outperform the ones that don't. The governments that adopt AI will be more powerful and outperform the ones that don't. The citizens, the individuals that do. And all of this will encourage us to hand over our decision-making to machines because performatively,

the outcomes will just be so much better for us individually and collectively. Why isn't that an existential risk, a slide into the end of human agency and an inability to return to our full human potential? No, I don't. So I think that, first of all, it's not clear to me at all that AI, companies that use AI or humans that use AI are going to outperform those that don't. You know, we have lots of, AI has lots of limitations. And, you know, for example, there was recently a lawyer who used AI. to put together a case, and it actually made up all kinds of cases cited, and he got, you know, pounded by the judge.

He did not out-compete the other lawyers. Now, maybe AI will get better, but I think that, you know, these sort of suppositions, the assumptions are not obvious at all, and humans are very reluctant to give up their agency. You know, we've had lots and lots of technologies in the past, and we've had, you know, people used to think that, you know, having the introduction of, say, writing and calculators would make people lose things like their memories, their abilities to reason well, the fact of, like, Google Maps, lose our ability to navigate and stuff. But humans actually kind of adapt to those technologies and go beyond them. So, again, you're nodding to this. You agree with the, there isn't an agency problem here?

Well, as Melani just said, the point has been made for every technological revolution or evolution for writing. Socrates was against writing. It said people are going to lose their memory, right? The Catholic Church was against printing press, saying they would lose control of the dogma, which they did. They could do nothing about it. The Ottoman Empire banned the printing press. And according to some historian, that's what accelerated their decline. It took then 300 years to, or at least 200, to authorize it again. And this was just because they wanted to control their population. And so every technology that makes people smarter or enables communication between people, facilitates education, again, is intrinsically good. And AI is kind of a new version of this. It's the new printing press.

So long as it doesn't blow up in our face. And the thing is that AI is... You can smash your hand with a printing press, right? Yes. If it's... The problem is the scale, right? So long as we build technologies that could be harmful, but on a small scale, the goods, the benefits overwhelm the dangers.

But now we're talking... Wait, wait, wait, wait, wait. Now we're talking about building technologies, unlike any other. technology because its technology can design its own technology. No. Yes. No. You're talking about superhuman AI.

This is the subject. It's under control. It's under control. We remain under control. It's very much like previous technology. It's not quite a different. All the experts have been studying this question say it's going to be very hard to keep it under control, and that is why I'm here today.

That very argument has been made with four computers, you know, 60 years ago. This is not a new issue. I recommend the audience to go to a website called the Pessimist Archive. It's hilarious. It's newspaper clips of all the stupid things people said whenever a new cultural phenomenon or a new technology appeared. Okay, the train. Oh, you're not going to take the train.

It's going to go 50 kilometers an hour, and you can't breathe at that at speed. And it's full of this, right? Everybody has said this kind of thing every single time. there was a technological evolution or even a cultural revolution, you know, jazz was going to destroy society, the printing press was going to destroy society, and it did. It totally did. For the better, it enabled the enlightenment, philosophy, science, rationalism. So let's, before we go to closing statements, I want to give Max the last word in the segment. An enlightenment, Max, we're on the verge of a reimagining and blooming of human thought powered by AI.

Stock traders will tell you that past performance is not an indicator of future performance. Future results, yeah. Yeah, future results. And it would be a huge mistake in an exponential technological growth to assume that just because something happened one way in the past is going to continue being this way. What's actually happened is, first during the Industrial Revolution, we made machines that were faster and stronger than us. So we started working less with our muscles and more with our brains, and that was fine. We typically got paid better.

Now, in this disempowerment scenario that you were asking them about, we're now instead building machines that can think better than us gradually. And we're not there yet, so we shouldn't be basing arguments based on today's pathetically bad AI. We should be looking at the tomorrow's, the superhuman AI that some people think might happen in five years or 20 years and you think maybe will happen in 300 years. That's what we should be talking about. That's the elephant in the room.

Because if that happens, it's going to be very different from the Industrial Revolution. Now we can neither compete with our muscles nor with our brains. And we are really going to start seeing a true disempowerment. And I don't think it has to be that way. I can be a bit optimistic, too. If we stop dismissing these existential threats and taking them seriously, dismissal of them is exactly the thing which is preventing us from doing the right thing. Okay, fantastic, four-way discussion.

I've learned a lot, and I know our audience has, too. So let's go to closing statements. We're going to put three minutes on the clock. These will be in the opposite order of the openings. So, Melanie, you're up first, and the stage is yours. Thank you. So as an AI researcher, I'm in awe of what our field has accomplished. You know, we've accomplished computers that can describe images to blind people.

We've developed computers that can help doctors diagnose diseases, predict the structures of protein and fluently synthesize enormous amounts of human collective knowledge. The potential for benefiting humanity is breathtaking. But hear me on this. We can acknowledge the incredible advances in AI without extrapolating to unfounded speculations of emerging super-intelligent AI. We all know that science and technology are double-edged swords with potential risks for misuse. There are many risks of AI technology, to be sure. But hear me on this as well.

We can acknowledge the risks and harms of AI without extrapolating to the mythical specter of existential threats. There are risks, there are harms, they're not existential. While talk of superintelligent machines destroying humanity or even helping evil humans to do so may resonate with our base fears, everything science tells us about the nature of intelligence and about the resilience of our society argues against the existential threat narratives that we've heard here today. I'm worried that overstating this so-called existential threats of AI takes our collective attention and focus away from the very real harms and risks that modern AI actually presents.

And this is not an idle worry. Remember Jeff Hinton, who we've all been talking about, he was asked on CNN why he hadn't pushed the concerns of AI ethicists at Google, who had long warned about risks of AI spreading misinformation and magnifying bias. He said, they were rather different concerns than mine. Their concerns aren't as existentially serious as the idea of these things getting more intelligent than us and taking over. So to me, this minimization of the real risks of AI

encapsulates what's so dangerous about this existential threat narrative. It takes all the oxygen out of the room and leaves no space for the real, evidence-based risks that we need to address. Let's not allow ungrounded speculations about the future of AI to inflame our emotions and fears, to distract us from real harms that we can address. Let's design ways to make AI safe, fair, and beneficial

based on science, not science fiction. Great closing statement. Melanie, thank you. Joshua Benjo, you're up next. I've been working on machine learning for all my life, and I've seen the progress. I did not expect the advances we've seen in recent years. And we now have machines that we can have a dialogue with and they can pass for humans.

And this was something that for a long time was considered as a milestone, maybe of actual human-level intelligence. Now, when we look carefully, we see that it's not exactly that and there are some things missing. Melanie says, let's not extrapolate. I say, we have to extrapolate.

The reason is we have no choice. If something as awful is maybe a few years or maybe even one or two decades in front of us, we have to prepare for it. We have to do the social adaptations. We have to do the AI safety work that I've been talking about, that I've been talking about,

that I think we have a chance. We have agency right now to control our future, and for that, we need to accept that there are all kinds of risks. Melanie, I'm totally with you with all the harms that AI currently poses, but it doesn't mean that we have to deny the risks that are existential, that are already on our radar screen, at least on mine. Now, just quickly, why do I think that this is something that can happen and have dire consequences? It's because of humans. It's not just by AI getting crazy. It's because we have weaknesses. There are many ways in which we can get deluded.

There are conspiracy theories, and lots of people believe them. So there will be people who act in strange ways and will do things that can be very harmful. So long as each of us has just our hands, maybe guns, unfortunately, and damage can be local and not existential. But when we build very powerful tools, I think we really need to be much more careful, and that's why we're talking about this today. So just to end on a positive note, Melanie asked me, you know, why don't you stop working on this? Well, I've been thinking a lot about this. I want to do what I think is best to go in the right direction.

And I think that I'm going to reorient my research so that either I'm working on applications that are not dangerous, but very safe, like working on healthcare and the environment, or working on AI safety, in order to prepare and prevent the bad things that could happen. So thank you very much, and think about it. Ian Lecun, the stage is yours. So, yes, there are risks, as with every technology, and they are not existential. I'm just repeating what you said. I couldn't say it better.

You know, every technology is a trade-off between the benefits and the side effects. Some side effects are predictable, some aren't. There are going to be side effects, bad side effects of AI. There are going to be people who are going to try to use AI for bad things. And it's a game of cat and mouse. And welcome to the real world. This is the world we live in. Every time something bad happens, we find countermeasures.

This is true for defense and everything military. This is true for intelligence. It's true for terrorism, crime, just about everything. And the statement has been made for every technological evolution. Remember what people were saying where the Internet started coming online and be generalized? Oh, there's going to be cyber attack. People are going to steal your credit card number, and maybe the financial system would be brought down.

Remember where people were saying just before year 2000? Satellites were going to fall out of the sky and crash into cities, and the phone system was going to crash and civilization will end. It didn't happen. We're still here. So I think there's a little bit of the same kind of feeling of uncertainty. A lot of people have the feeling that bad things are going to happen because they're not in control.

There are the feeling that AI is just going to happen, and there's nothing they can do about it, and that creates fear, and I can completely understand that. But some of us, in what could be construed as a driver's seat, there are plans to make those things safe. And if we can't make them safe, we're not going to build them. Here is an example. In the 50s, people thought about building nuclear-powered cars. There were prototypes that were built.

They were never deployed, obviously, because it's not safe. So I think it's going to be the same thing for AI. We're not going to build nuclear-powered AI. But, although arguably in some countries, it is nuclear-powered. So I think there are risks attached to not developing AI. There's so many potential benefits. We have to think about that and sort of weigh it against the actual risk, but the risks are not existential.

AI is going to be subservient to human. It's going to be smarter than us, but it's not going to reduce our agency. On the contrary, it's going to empower us. It's like having a staff of really smart people working for you. That's the new Renaissance. Thank you. Thank you, Jan Lacoon.

We're going to give the last word and tonight's terrific debate to Max Tegmark. Max, take us away. I'm going to make a pitch for humility. Science is all about being humble. My definition of what it means that I'm a scientist

is that I would actually rather have questions I can't answer, answers I can't question. Now, what are you going to vote on here in a few minutes? You're going to vote on this question of whether this is an existential threat? What does that mean?

You're not asked to vote whether you're 100% sure or we're going to get wiped out. You're going to ask to vote on whether there is non-zero risk of an existential risk or a non-zero probability, right?

That's what you're going to ask to vote on. So if you think, yeah, it's probably going to be fine, but maybe there's a 5% chance that we're going to get wiped out. And if you wouldn't get on an airplane, if you're told that it has a 5% chance of crashing, that's too high for you. Then you should vote, yes, it is an existential threat. We all need to be more humble.

You were right, Jan, making fun of that pessimism website where people worried too much about stuff that didn't happen. You were right, Melanie, pointing out how ridiculously over-optimistic McCarthy and other Minsky were in the beginning, thinking AI would happen sooner. Scientists have lacked humility in that direction. Well, we've also just as often lacked humility in the opposite direction. The world's most famous nuclear physicists at the time, Lord Rutherford said, oh, getting nuclear energy out of atoms is like moonshine.

Very next day, Leo Zillard invents a nuclear chain reaction, right? And as I said, a few years ago, most people, including even Yoshobenzio, thought we weren't going to have GPT4, well, here it is. And we also have to be humble about these probabilities that I kept pestering you about there. So thank you so much, John, for actually. giving you one. You said it's about the same risk as getting struck by an asteroid. Since I'm

a nerd, I happen to know that the risk of that, wiping us out, is about one in a hundred million per year. And I think that's not a humble estimate. I think it's way too low, as Joshua explained. It reminds me a lot, actually, of when the Challenger space shuttle investigation happened, where many scientists said the probability of the spatial blowing up was 10 to the minus five, one in a hundred thousand. They sent up a hundred of them. but two of them blew up, right? Lack of humility. The designers of the Fukushima nuclear power plant

had said that the chance is less than 1 in 10,000 that you'll get that kind of bad tsunami in any given year. Lack of humility. Let's be humble. The truth is we don't know how soon we're going to get superhuman AI. The three of us think it might happen in five to 20 years. Melanie thinks it's going to take a lot longer. We just don't know who's right.

That means there's a risk that it is going to happen. soon. And we also don't know whether it's going to go well or whether we're going to get disempowered or wiped up. Let's be humble. It is an existential threat. That's what it means. Well, that wraps up today's debate. I want to thank all of our participants, Max

Tegmark, Joshua Benjio, Jan Lecun, Melanie Mitchell for a terrific exchange. We polled the audience twice. We pulled them at the beginning of the debate and now at the end, the 3,000 people in attendance vote in 64% in favor. at the end of the evening, 36% opposed, making the con team of Gann Lacoon and

Melanie Mitchell winners by a slim margin of 4%. Did you change your vote listening to this podcast? Do you agree with the audience that you're more likely to think that AI research and development isn't a threat? We'd love your reflections and feedback. Let us know how you voted and what your thoughts are on this debate. Podcast at monkdebates.com is the email where you can get. get us right now. Well, thank you for lending your time and attention to our efforts to bring

back the art of public debate one conversation at a time. I'm your host and moderator, Rudyard Griffiths. The Monk debates are a project of the Aurea and Peter and Melanie Monk charitable foundations. Rudyard Griffiths and Ricky Gerwitz are the producers. Be sure to download and subscribe wherever you get your podcasts. And if you like us, feel free to give us a five-star rating. Thank you again for listening.