The Munk Debates Podcast - Be it resolved: The quest for true AI is one of the great existential risks of our time

Starting point is 00:00:01 There are options, and that's why we need to take this opportunity seriously. There's no way you can prevent global warming unless China is part of the solution. This is not normal male behavior. This is predatory behavior. We don't know how bad this bug is. We don't know what this bug does. All of that was thrown away in those eight minutes and 46 seconds, and that's the moment that I became an abolitionist. Extraordinary claims require extraordinary evidence. Welcome to the monk debates. we provide you with a civil and substantive debate on the big issue of the day to arm you,

Starting point is 00:00:39 the listener, with enough information to make up your own mind. Today's debate, be it resolved. The quest for true artificial intelligence is one of the great existential risks of our time. The curse of the Ninth Symphony might finally be broken. Artificial Intelligence is set to complete Beethoven's tense symphony through the tormenting fragments he left behind. Google's AlphaGo computer software beat its human opponent on Wednesday in the first of a historic five-game match between human and computer. The victory is considered a breakthrough for artificial intelligence, showing the program has mastered one of the most creative and complex games ever devised.

Starting point is 00:01:23 The neural network operating the Volvo XC90 deliberately violated the rules, considering it right and safe. Apparently, it learned this from a human being. Hello, I'm your moderator, Rudyard Griffiths. Well, those are just a few of the technological feats that have occurred in the past few years, thanks to extraordinary developments in artificial intelligence. Perhaps not surprisingly, many scientists are pushing for a more cautious approach to the development of machine learning. They argue that we could be not so far off from creating super intelligent machines who capabilities far surpass that of humans, machines that might not come with an off switch. These scientists argue that we can prevent humanity losing control over intelligent machines,

Starting point is 00:02:14 but we need to act now. Here's the CEO of SpaceX and Tesla, Elon Musk. Mark my words, AI is far more dangerous than news. I think we should be really concerned about AI. AI is a rare case where I think we need to be proactive in regulation instead of reactive because I think by the time we are reactive in AI regulation, it's too late. Critics say this view of superintelligence highly overrates the abilities of machines today and deeply underestimates the challenges of creating truly cognizant reasoning machines in the future.

Starting point is 00:02:54 They say that AI is nowhere close to matching the human minds' talent. for understanding, organization, and generalization. This is scientist and AI entrepreneur Gary Marcus. How you open a door, machines don't understand the first thing. Every ordinary person in the West has opened a bunch of doorknobs and gets the idea, right? I need to turn something, jiggle something. It might be different for the next one until the door itself is freed. And yet nobody's ever built a robot that can do that.

Starting point is 00:03:28 Common sense is not getting attention that it deserves. On this installment of the monk debates, we challenge the essence of these arguments by debating the motion, be it resolved, the quest for true artificial intelligence is one of the great existential risks of our time. Speaking for the motion is Stuart Russell, professor of computer science and engineering at the University of California, Berkeley, and also an honorary fellow at Oxford.

Starting point is 00:03:57 He's the author of many important books on artificial intelligence. Most recently, Human Compatible, Artificial Intelligence and the Problem of Control. Speaking against the motion is Melanie Mitchell, Davis Professor of Complexity at the Santa Fe Institute. She also is an award-winning author on the subject of AI. Her most recent book is called Artificial Intelligence, a Guide for Thinking Humans. Stuart, Melanie, welcome to the Monk Debate. Thanks. It's great to be here. It's nice to be with you.

Starting point is 00:04:31 I'm really looking forward to today's debate. This is just a lovely moment for us to pull back and think some big thoughts about the nature of artificial intelligence, how it will reshape our society in the years and decades to come. And more importantly, what are the risks of AI as we develop this new technology to our collective way of life, to how. how we think about what it means to be human and what it could potentially mean to create an intelligence that rivals our own, maybe not directly, but in some way that is truly transformative to the human experience. So to have both of your kind of considered opinions, your sustained, kind of thought and engagement with this topic to benefit us and our reflections today is a privilege indeed. A simple motion before us, be it resolved the quest for true AI, artificial

Starting point is 00:05:26 intelligence is one of the great existential risks of our time. Stuart, you're arguing in favor of our motion. I'm going to put a couple minutes on the clock and hand the program over to you. Thank you very much, Rajan. Let me begin by agreeing with my opponent, Professor Mitchell, that the human mind is quite incredible. And so exceeding its capabilities to make effective decisions in the real world, which is what I mean by true AI, is not going to happen tomorrow. But it's very probably going to happen eventually, most people think, within this century. And we have to ask ourselves, then what? What if we actually succeed in our goal in AI?

Starting point is 00:06:09 One answer was provided by Alan Turing, who was the founder of computer science in 1951, he said, and I quote, it seems probable that once the machine thinking method had started, it would not take long to outstrip our feeble powers. At some stage, therefore, we should, should have to expect the machines to take control. So why did he say this? Well, it's obvious, right? Intelligence gives us power over the world, and Turing is asking, how can we possibly retain power over a more powerful entity than ourselves forever? And that question really has no answer,

Starting point is 00:06:46 at least if we design those powerful entities within the current standard mathematical model of AI. And in that standard model, we create machines that pursue fixed objectives. And we call this the King Midas problem. So King Midas asked the gods that everything he touched should turn to gold. He got exactly what he asked for, his fixed objective. And unfortunately, his food and his water and his family all turned to gold and he dies in misery and starvation. And now, not having learned that lesson, Facebook has asked that everything we touch should turn to gold for Facebook, of course. So their algorithms, their machine learning algorithms, select content to maximize clicks. That's the fixed objective, and we can see what a mess that's made of the world.

Starting point is 00:07:35 That's despite the simplicity of these algorithms, as they get more complex for any fixed goal, the machine is going to mess up the rest of the world because you have said that the rest of the world doesn't matter by giving it that fixed goal. So we will eventually lose control if we build a... systems in the current standard model. We cannot define our objectives completely and correctly concerning the entire future. So instead, we need to build a completely new foundation for AI that's provably safe and beneficial.

Starting point is 00:08:10 And we don't know how long it's going to take to create that foundation, and we don't know how much time we have left. Thank you, Stuart, for those opening remarks. Melanie, is same for you, two minutes on the clock to argue. against our motion, be it resolved, the quest for true AI is one of the great existential risks of our time. Over to you, Melanie. Thanks. The phrase, a great existential risk of our time, means that something threatens the existence of the human species in the relatively near future. Unfortunately, we face many such risks, such as the climate crisis, the threat of emerging

Starting point is 00:08:47 viruses and pandemics, as we've seen so clearly, and many others. But in comparison, what about current AI and its prospects for actually becoming intelligent at anything close to human level in our time, much less being an existential threat. Well, here's this actual state of AI. While there's been a lot of progress on specific narrowly defined tasks, research on how to build machines with the general intelligence of humans, not to mention what you might call superintelligence, is essentially still at square one. The biggest challenges for developing AI have been the same since the field, beginnings. That is, giving machines a common sense understanding of the world, one that's not written down anywhere, but that underlies all of our own intelligent behavior, getting machines to

Starting point is 00:09:34 reliably apply what they've learned to new situations, and getting machines to actually understand human concepts. We're nowhere near a machine that has anything like these abilities. In fact, one AI researcher predicted that human level AI is 100 Nobel Prizes away. Now, I'm not saying, saying that full-blown AI is impossible, but I'll just reiterate what John McCarthy, one of the field's founders, said late in his life. It turns out that AI is harder than we thought. The fact is that we humans have poor intuitions about the nature of intelligence. Now, Stuart in his book, Human-compatible, imagined some doomsday scenarios, like one in which a so-called super-intelligent machine is smart enough for us to task it with, say, reducing

Starting point is 00:10:21 worldwide carbon emissions, but at the same time, the machine is too dumb to know that killing all the humans is not a solution we'd endorse. In other words, this imagined machine is super smart and powerful, but at the same time, strangely dumb and mechanical. Now, to my mind, this view of superintelligence or true AI makes the term meaningless. It's almost certainly impossible to separate intelligence from basic common sense. Today's so-called artificial intelligence is actually not intelligent, precisely because it lacks this kind of common sense. So how should we deal with a possible threat of super intelligent AI? I'll quote Stuart's book, again, where he says, we should probably get a couple of bright people to start working on preliminary

Starting point is 00:11:07 aspects of the problem. This seems to me to be closer to the right approach, rather than misidentifying AI as a near-term existential threat. Thank you, Melanie. Great opening statements from you. both you've nicely set out a whole bunch of themes that I'm looking forward to exploring with you in more depth in the moderated middle of our discussion. But before we get to that, I think it's important for us to hear both of you react to what each other has said in this debate so far. So, Stuart, let me add another two minutes to the clock and give you the opportunity to respond to Melanie's opening statement. The main claim that I'm hearing Melanie making is that we don't need to worry because true A.I.I. is not here yet. And I didn't say true AI here was here yet, but the vast majority of AI researchers are pretty confident that it will arrive during this century. And I've been in the field now

Starting point is 00:12:02 for 40 years, and I've seen massive progress, things that even 10 years ago, we thought were decades and decades away, have come to pass, for example, beating the human world champion at Go. So saying, well, it's not going to happen. So we don't need to worry. That's sort of like driving a bus towards a cliff and saying, well, don't worry, I'm pretty sure we're going to run out of gas before we go over the edge of the cliff. It's just not a very good way to manage the affairs of the human race. The second main point that I hear Mellie making is that it's not possible for a machine to be intelligent, truly intelligent, and to not care about the continued existence of the human race. And you can just imagine, you know, the guerrillas, you know, having a meeting a few million years ago.

Starting point is 00:12:54 And, you know, they've seen that another species is going to happen down some evolutionary branch called human beings. And they're going to be much more intelligent than the guerrillas. And the guerrillas are saying, you know, it's just not possible that humans could be intelligent and not care primarily about the benefit of gorillas. I think the guerrillas were wrong about that. So it's entirely possible for us to create machines that are. perfectly intelligent, that perfectly understand that humans want to be alive, just as we understand that gorillas and ants want to be alive, but not have the benefit of humans as their primary overriding objective. And that's what I mean by creating AI on a new foundation.

Starting point is 00:13:37 Thank you, Stuart. Melanie, your opportunity here for a two-minute rebuttal. Take us away. Stewart says that the vast majority of AI researchers are confident that true AI will arrive in this century. I'm not sure that I agree with that. I know there's been some unscientific polling done. But even if that was true, I would say that the field of AI has had a long history of wrong predictions. So the leaders of the field, such as Marvin Minsky, Herbert Simon, Claude Shannon, many others, and even more recently some important leaders in the field have predicted that we will have true AI within decades. And this happened in the 60s, the 70s, the 80s, and so on. And none of these predictions has ever come close to being true.

Starting point is 00:14:29 And I would argue that the problem is that our intuitions about what intelligence is and how complex it is are very poor because the most important and essential parts of intelligence, our own intelligence, are unconscious and invisible to us. And this has been a recurring problem in AI. In fact, Marvin Minsky, one of the pioneers who predicted true AI within a few decades, said later that what we've learned from AI is that easy things are hard. That is, it's much easier for a machine to play, go, better than any human, that it is for a machine to do things that even a young child can do,

Starting point is 00:15:11 like carry on a coherent conversation, or understand commands that we give them, or suggestions, or have basic common sense. So what we consider to be the most intellectually challenging tasks for us are not necessarily the most intellectually challenging for computers and vice versa. So I think that we should take with a grain of salt any predictions about true AI and when it's going to come about. Thank you, Melanie. An opportunity now for us to engage together in a conversation and really think through some of the ideas and topics

Starting point is 00:15:45 that are top of mind for our listeners. And I guess I want to start with you, Stuart, and just push you a little bit on your assumptions here about the nature of super-intelligent AI. I'd like to hear a bit more from you as to why you think Melanie is wrong that, you know, with super-intelligence, wouldn't morality and ethics and a sense of the good, as we understand as humans, be part of that intelligence?

Starting point is 00:16:14 The intelligence that you've described so far seems very mechanistic, very alien, very different from what humans assume that intelligence is. Yeah, so I think alien is a good word. There's no particular reason to suppose that what machines do. I mean, machines, first of all, are physically completely different from the human brain. They didn't evolve under the pressures of evolution, particularly social evolution, that humans have evolved under. So in many ways, machines, there's much more freedom of choice about how we design machines. What I'm arguing is that the current understanding of AI is actually, in some sense, the danger. Because the way we understand AI is that we design.

Starting point is 00:17:03 machines that optimize objectives that we supply. And this is not just AI. This is also many other disciplines, control theory, statistics in economics. This is a well-known issue. You know, if you define GDP as the objective or you define quarterly profits as the objective, then you can get all kinds of problems arising from pursuing those objectives. But that's the current science of AI. And the problem is we don't know how to specify objectives completely and correctly. And so that leads us to the King Midas problem. And I think we take it as a given in engineering that the degree of capability for achieving an objective is unrelated to what that objective is. In other words, I could swap out one objective and replace it with another, even if I have a very, very intelligent

Starting point is 00:17:55 machine. And that's the source of the risk. Melanie, I'd like you to come back on that point and give our audience a sense of why you think that the superintelligence of an artificially created machine might not, in fact, be alien, that it may have human characteristics that we would see in itself because we see it in ourselves. and therefore the risk of this kind of goal-driven path dependency that Stewart is outlining, in your view, really isn't part of how we should think about the characteristics of superintelligence. Let's hear more from you on that. Yeah, I think there might be a difference in the way that Stewart and I would define intelligence. And I think this is one of the problems that we don't have a good agreed upon definition of intelligence in AI or even in psychology or neuroscience. But I do think that what we know about intelligence in humans and other animals is that it's very much tied up with a particular worldview, a particular set of perceptions and way of being in the world, even with culture.

Starting point is 00:19:09 And when I talk about common sense, I mean not just something you might say that your teenager doesn't have, but something much more basic, something about knowing about how the world works. knowing about objects, knowing about objects interact, knowing about how people, their goals, their desires, having kind of what people call a theory of mind. And nothing in our knowledge of psychology or neuroscience today gives any evidence that you could separate any of these things the way that Stewart claims that you can, that you can sort of stuff in new objectives, any objective, and the intelligence is kind of orthogonal from all of these other things. So that's really a different view of intelligence. And really it's something that we don't understand it all very well. So I think we have to separate two things.

Starting point is 00:20:01 The ability to understand and predict what humans want and what they're going to do to understand what the human's goals are and to agree with those goals. And there's no reason that I can see unless we figure out how to design them that way, that AI systems, have to have this characteristic that everything they're going to do is for our benefit. And it's particularly difficult to ensure that because we don't even know what our benefit is. So it's very hard for us to, first of all, articulate it correctly and then get that definition into the machine. And any attempt that we make to do it is probably going to fall short in one way or another. And it's precisely when it falls short that that creates a kind of

Starting point is 00:20:46 loophole through which the AI system will end up messing up the world in ways that we're actually going to be unhappy with. I think that there's two things going on in Stewart's argument. There's kind of two separate scenarios. One is a scenario where there's no sense of whether the machine agrees or disagrees, but rather it's trying to carry out some objective. And we didn't state it clearly. So the machine ends up causing a lot of damage.

Starting point is 00:21:13 The second thing is where the machine actually does understand. but it doesn't agree, so it's more nefarious in that way. I think those are two completely separate scenarios. And I don't think the first one is possible to have something that's super intelligent but doesn't understand. The second one, I think that if we had a machine that was like us in the sense that it had the concept of agreeing or disagreeing or it had its own sort of desires,

Starting point is 00:21:41 that those desires and morals and what have you would have come about through a development that is something like the development that humans go through in a society, in a culture. I don't believe that our concepts like the ones that we have can come about in any other way, that kind of understanding. So I'm not convinced that we would have a machine that had everything that we had, all the understanding that we have, but was so different that it would want to destroy us. So let me be clear. I'm not saying that it wants to destroy us.

Starting point is 00:22:21 I'm not saying that it has the goal of human destruction. It has the goal that in some sense we set it up with. And we're probably not going to set up explicitly with the goal of human destruction. But in making mistakes, and this is precisely the King Midas problem, he thought that what he was going to get was what he wanted. But it wasn't. He made a mistake. it's quite possible for us to understand that ants don't want to be dead, but we don't have to agree with that.

Starting point is 00:22:51 And most of the time, if we want to build a house and there happens to be some ants there, well, tough luck for the ants. So we understand that they want to be alive, but we don't agree. So it's obviously possible for this situation to arise that you understand that another entity has certain goals, but you don't agree with those goals. You don't align your behavior with the benefit of the other entity. Hi, Monk podcast listeners. I want to let you know that we have a special members-only event happening this week on February 4th. We're going to have Masha Gessen and Uval Harari on for a video Q&A, an hour-long program. Digging in with these two big internationally best-selling commentators,

Starting point is 00:23:37 their thoughts on the future of democracy, authoritarianism, the rise of populism, the big events and trends of 20. We're going to have that all with Mashah Gessen, an Uval Harari, live February 4th at 8 p.m, and archived on our site. This event is free for members. You can become a member any time by going to our website, www.munkdevates.com forward slash membership. Sign up. It's free. Participate in the Mashagessen Uval Harari event this February 4th and then archived for our members viewing pleasure on our website. Hope to see you then. Now back to our program.

Starting point is 00:24:21 Melanie, let's talk a little bit about, you know, dual-use technology, because for, you know, the better part of human invention, we have seen waves of new technologies emerge that have dual purposes. I think of nuclear power. It lights up cities, and it destroyed Nagasaki and and Hiroshima. So I'm wondering why you feel that artificial intelligence, and again, I don't want to mischaracterize your views, so challenge me if I'm wrong here, but why you think that AI isn't potentially vulnerable to that same dual purpose drive of so much of the technology that we seem to create? Yeah, that's a great question. And I think it's really important to clarify my belief that AI does have some urgent dangers.

Starting point is 00:25:12 I don't know if they're existential or not, but they're very worrying, and they include things like misuse of AI for surveillance, for, you know, we've seen facial recognition being used in law enforcement, which has caused some very big problems to, say, civil liberties. We see lots of biases in AI with respect to race, gender, other kinds of biases. We tested Amazon's facial recognition software. by using it to compare every member of Congress to a set of mugshot photos. Here's what we found.

Starting point is 00:25:45 The software incorrectly matched 28 members to someone who had been arrested. Nearly 40% of these false matches were people of color, even though they make up only 20% of Congress. And in particular, we see AI systems being deployed that are not trustworthy, that make unhuman-like errors that we didn't predict and that cause real damage.

Starting point is 00:26:08 In July 2015, 57-year-old Wanda Holbrook was killed in an industrial accident. According to the lawsuit, a robot took Wanda by surprise and entered the cell she was working in and hit and crushed Wanda's head between a hitch assembly. It was attempting to place in a fixture. But this is the narrow AI that we have today has real dangers. The prospect of what in the resolution was called true AI, Well, we can argue about what that means, but it's certainly very far, extremely far from what we have today, which is one of the reasons I don't believe it's an existential threat of our time.

Starting point is 00:26:52 Got it. Stuart, do you want to come back on the dual use kind of concern that I've tabled here and maybe share with our audience why you think that there could possibly? And again, I don't want to mischaracterize your views here, but a parallel to nuclear weapons and how they really truly have emerged as a. global existential threat, despite the underlying technology. The releasing of the power of the atom has also brought to the world incredible progress and opportunity. I think there are many interesting parallels. And I think as early as 1912, the concept of the atomic bomb was developed, and people pointed out that this would be a huge threat to civilization if the technology was realized. But the physics establishment resolutely resisted the idea that this was possible. In fact, on September 11, 1933, Lord Rutherford, who was the leading nuclear physicist, the man who

Starting point is 00:27:49 who split the atom, was asked, do you think it's possible that maybe in 25 or 30 years time we might be able to release the energy of the atom? And he said, no. In fact, he described this idea as moonshine. And then the next morning, Leo Zillard invented the nuclear chain reaction, literally the next morning, because he read about Rutherford's speech and got a bit annoyed with it. And so it went from impossible to, you know, essentially solved in less than 24 hours. So one parallel there is don't rely on the lack of human ingenuity to create true AI because it can happen and it has happened in the past that we create technology that presents an existential risk. The second interesting parallel is that there really is a containment problem, and the analogy is between

Starting point is 00:28:41 nuclear material going critical, right? So you put enough enriched uranium together in one place, you exceed critical mass, and it turns into an explosion. The problem is generating that energy in a way that's contained. And that's the problem I'm talking about with AI. We would like to have true AI because of all the enormous benefits it brings. But how do we contain it? How do we retain control forever? And we can't afford a single Chernobyl to happen. Moscow television tonight. For the first time ever, the Soviet Union admits it has had a nuclear accident and it's clearly a major one. And if that happens with with true AI that we lose control over it, it might be irreversible.

Starting point is 00:29:33 So can we ensure that this never happens? Well, the nuclear industry thought they could ensure it with nuclear reactors. They certainly had a strong motivation. Nobody wants their reactor to blow up. But we were imperfect, and it happened. So that's an excellent segue, sir, because of Melanie, I wanted to ask your views on that, you know, what's kind of typified as the breakout problem. that as we have in our resolution, true AI,

Starting point is 00:30:00 or other people call it superintelligence, once this happens, is there a risk of our losing control, not over the intelligence as it exists at the moment of breakout, but in terms of how that intelligence then evolves on its own, separate from us as a super intelligence. So let me comment on that in two ways. One is I really like the story about Rutherford and Sillard and the notion of the chain reaction. But I don't think that's a good analogy with where we are with AI.

Starting point is 00:30:40 So back in the days of Sillard and Rutherford, the notion of energy and nuclear energy and the atom were quite well understood. and it was really a matter of figuring out how to harness this energy of the atom. With the notion of intelligence or even AI, we're at a much further point in the actual science of it. I'd say a better analogy to where we are now is something like the days of the alchemists who were trying to transform metal into gold. And they didn't have the theory of chemistry. And that brings me to the point you asked, which is, If we had a superintelligence, how would we control it?

Starting point is 00:31:25 Well, I would argue that, again, we don't have the science right now of intelligence to really understand even what superintelligence means, even if such a concept makes any sense at all. I feel like we are too ignorant of the science to really make sense of that question. And really, the priority should be to much better understand the nature of intelligence. fascinating. Stuart, do you have a comment on that? I agree with Melanie that we absolutely need a deeper science of intelligence. And I think if we are going to move towards true AI, and here what I mean, I think

Starting point is 00:32:06 Melanie, I also disagree about what exactly this means. From my point of view, the existential risk comes from AI systems that have sufficient capability of decision making in the real world that we can. could lose control irreversibly to those systems. And it doesn't really matter if those AI systems are capable of using metaphorical language with the same subtlety and flexibility that humans have. Gorillas value, perhaps the ability to groom fur extremely well, that humans are not very good at. And so the guerrillas maybe don't consider human intelligence to be true intelligence. But it doesn't matter. The fact is the guerrillas lost control over the world to humans. And lost control.

Starting point is 00:32:49 over their own future as a result. And so if we're going to find a way to control the intelligence that we are investing hundreds of billions of dollars to develop, we have to have a deeper science so that we can actually analyze the way that the systems function and ensure that when we put the components together in the right way that it remains under control in a way that's sort of mathematically provable. And we saw with Chernobyl, it's not enough just to have good intentions and, you know, lots of good engineering practice and lots of checkboxes and all the rest. We really, in this case, cannot afford to have anything other than a deep, solid mathematical understanding of what we're doing

Starting point is 00:33:37 and mathematical proofs that things are going to go the right way. That's a nice segue to my final question to you both before we move to closing statements. And it's really about, What prescriptions do you think, Melanie, I'll start with you, do you think we should be putting in place now possibly to control or regulate the development of artificial intelligence? I do think that regulation is extremely important more on the people who are developing AI than on an intelligent AI system itself. As I mentioned, there's a lot of issues and threats that today's AI and even the AI that we are envisioning in the near future present to us. And I think the regulation is essential. We've already seen some efforts towards regulation, for instance, of the technologies like facial recognition. San Francisco, home to many of the world's tech pioneers, is blazing a new path.

Starting point is 00:34:40 It's restricting the use of facial recognition technology by the local. government. You know, we might look to, for instance, regulation in biology and bioethics and genetic engineering regulations that are coming up. I think it's very analogous. But no one knows very well how to regulate these systems because I think the general public and policymakers don't really understand the current limits of these systems. The problem is not that these systems are too intelligent.

Starting point is 00:35:11 The problem is that they're too limited. and people tend to trust them more than they should. So I think there's got to be more education, more real honest assessment of where our technologies are. And it has to involve not only the technologists, the AI researchers, like Stuart and myself, but also people in other domains like law, social sciences, sociology, and so on. So Stuart, similar question to you. What is your recommendation for how governments, maybe individual scientists should be thinking about developing this technology

Starting point is 00:35:47 and de-risking some of the issues and concepts that you've raised in this debate. So I think this is a great question, and I'm often asked it by legislators. I think there are some things we have to do immediately. One of them is to ban the use of AI in so-called lethal autonomous weapons. So basically, we shouldn't be writing algorithms that can decide to kill human beings. And that's not because of a fear of Terminators taking over the world. That's really because autonomous weapons can be multiplied without limit because they don't require human beings to supervise their activity.

Starting point is 00:36:27 And so that creates a weapon of mass destruction. You can launch swarms of literally millions of lethal weapons to wipe out an entire country. And we just don't need to be creating that type of weapon. I think another good place for regulation is in the impersonation of human beings. Whether it's through voice or chat or soon, I think it's going to be possible to have a Zoom conversation with someone and not realize that you're actually conversing with a machine. And I think we have a human right to know whether we're interacting with a machine or another person. And that would be a place to plumb the flag for humanity, so to speak, in the digital world. But as far as controlling AI, I would say we're still some years away from understanding the solutions to the control problem well enough to actually mandate any particular design template.

Starting point is 00:37:23 But just as happened with nuclear reactors, very soon after Zillard invented the nuclear chain reaction, he also figured out a way to make a nuclear reactor that was sort of self-damping. So it had feedback control systems that would ensure. that it couldn't go super critical. And those design templates after the war were turned into designs for nuclear power stations. And you could not get a license to run a nuclear power station unless your system conformed to a design template that was already understood and had been analyzed mathematically and at least in theory shown to be safe. And I think that's the way that it should go with AI systems.

Starting point is 00:38:03 And I would say even now, as I mentioned with social media, content selection algorithms, they're having a massive negative effect on the world because the way they're designed, they're sort of constituted to manipulate human psychology so that they can turn you into someone who clicks predictably on the content that they send you in future. And we wouldn't allow that with pharmaceuticals, right? We wouldn't allow pharmaceutical companies to pump out billions of mind-altering drugs to the whole population of the world without any clinical trials or safety checks. And that's exactly what's happened with these algorithms. So I also think we need a kind of regulatory regime for any sort of human-facing algorithm that we put out there. Fascinating. A really terrific debate I've learned so much from you both. So let's move to closing statements. Our resolution has been the quest for true AI is one of the great existential risks of our time.

Starting point is 00:39:03 Melanie, you've been arguing against the motion. and let's have your two minutes of closing remarks. There are many existential threats in our time, but true AI or superintelligence are not one of them. There's, well, there's been a lot of progress on narrow tasks. AI is still very far from anything like human intelligence. Moreover, we humans have very poor intuitions about what intelligence is. In fact, the last seven decades of AI research have shown

Starting point is 00:39:34 how much harder it is than we thought and how much more complex our own intelligence is than we imagined. Now, Stort's assumption seems to be that a super-intelligent AI could surpass the generality and flexibility of human intelligence without any basic human-like common sense. Now, nothing in our knowledge of psychology or neuroscience supports this possibility. Instead, human intelligence seems to be a strongly integrated system

Starting point is 00:40:01 with many interconnected attributes, such as emotions, desires, a strong sense of selfhood and autonomy, and a common sense understanding of the world. These can't easily be separated. If a generally intelligent AI is ever created, its objectives like ours won't be easily inserted or aligned, but they'll develop with the other qualities that form its intelligence, as a result of being embedded in human society and culture. Does that mean that human-level AI or superintelligence, whatever that means, is impossible?

Starting point is 00:40:33 and we shouldn't worry about it? No, but we have to prioritize the possible problems we face. And I would say that the threat of super-intelligent AI is far down the list. Instead, we should worry more about some urgent threats that AI does pose, such as the use of AI for surveillance and other applications that threaten civil liberties, racial gender and other biases embedded in today's AI systems, and the deployment of AI systems that are not reliable or trustworthy. We have to very clear about what AI's real threats are. In my view, the most useful thing we can do to understand the potential benefits or threats of future human-level AI is to gain a better scientific understanding of the nature of intelligence. That's what should be a priority.

Starting point is 00:41:20 Thank you, Melanie, for those closing remarks. Storch, two minutes on the clock for you. We're going to give you the final word in our debate. Thank you. So I'll recap the main argument that I'm making. I think it's quite simple. In fact, as Bill Gates said a few years ago, I don't understand why some people are not concerned. So first, AI systems will eventually be able to make better decisions than humans in the real world, in the pursuit of their objectives. And for AI systems designed within the standard model

Starting point is 00:41:52 where the objective is fixed and known, this inevitably leads to conflict. And this is the King Midas problem that I talked about at the beginning, that we don't know how to specify our objectives completely and correctly and communicate them to the machine. So that misalignment leads to, in a sense, a form of conflict that we are inevitably going to lose. It's also worth pointing out that in order to maximize their success in achieving their objectives, the AI systems may end up consuming all the resources that we need, or they may make changes to the world that are incompatible with life. And let me illustrate this idea in a very concrete way, which is the fossil fuel industry,

Starting point is 00:42:35 which you can think of as a machine that maximizes an incorrectly specified objective, which is quarterly profits. And it began its disinformation campaign 50 years ago, and it's completely outsmarted the human race. So we human beings lost. The fossil fuel industry is already destroying the world. And the industry does have human components, but as a day, decision-making structure, it does not resemble the human mind in any way. It's a machine pursuing an incorrectly defined objective. So if we're going to be creating more and more

Starting point is 00:43:11 capable AI, and we are investing hundreds of billions of dollars in doing exactly that, we have to steer the field of AI in a different direction towards systems that are provably beneficial to humans. And this is a massive rethinking task. We almost have to start again from scratch. And we're not sure how much time we have. And if we think of AI as a huge super tanker that's moving in a particular direction, we don't know how long we have until it hits the rocks. We don't know how long it will take to turn this ship around. So the sooner we start, the better. And that's not going to happen if we keep denying that there are any rocks there. Or if we just say, well, the ship is going to run out of gas before it hits the rocks, so everything's

Starting point is 00:43:56 fine. Or if we just take our hands off the wind, or if we just take our hands off the wind, wheel and hope that the ship will steer itself. Thank you. Thank you, Stuart. And thank you, Melanie. This has just been a fascinating debate. It's clearly an issue that you've both thought about in a wonderfully nuanced and sophisticated way and just again to have the opportunity to spend the last period of time with you, thinking some big thoughts about the future of artificial intelligence has been a privilege indeed. So on behalf of the Monk Debates community, thank you, Melanie. Thank you, Stewart so much for coming on the program. Thank you.

Starting point is 00:44:29 Thank you, Roger. That wraps up today's debate. I want to thank our participants, Stuart Russell and Melanie Mitchell, certainly gave us a lot to think about. If you have any feedback or reflections on what you've just heard, please send us an email at podcast at monkdebates.com. Again, that's podcast at MUNKDbates.com. Here are some listener comments about our recent debate, be it resolved.

Starting point is 00:45:01 safety and fairness preclude the participation of trans athletes in high-level women's sport. Chris writes, Rudyard, I felt the urge to comment on several occasions in the past, however, today I feel compelled to reach out. My daughter, a sophomore in grade 10 in high school, playing on a varsity volleyball team, came up against a team with a trans athlete who was of equal size, however, much stronger and with the capability to hit the ball much harder. Today, volleyball has similar concussion protocols to the NFL because concussions are common. Under what circumstances is it safe to enable that level of dominance where a single player can create substantially more risk for competing athletes in close quarters?

Starting point is 00:45:50 Thanks, Chris, for taking time to reach out and to share your thoughts on our podcast on trans athletes participating in elite women's sports. On that same debate, Gina writes us with a debate idea. She says, I'm wondering if you would consider the issue of medically transitioning minors with gender dystopia. There is a raging debate on this, especially in light of the major UK high court decision on the matter, as well as major changes in policy in Finland and Sweden. I think it would be wonderful. The monk debates could handle this issue. issue in much the same way as you did the debate on trans athletes and women's sport. Thanks for this suggestion, Gina.

Starting point is 00:46:33 We'll definitely check that out. As always, we'd love your opinions on the monk debates you're hearing on our podcast feed. Please send them to us at podcast at monk debates.com. The monk debates is that special place for civil and substantive debate on the big issues of the day. To listen to more debates on everything from climate change to religion to geopolitics. To the future of human progress, visit our website, wwwwmunkdebates.com. Thank you for helping us bring back the art of public debate,

Starting point is 00:47:07 one conversation at a time. I'm your host and moderator, Rudyard Griffith. The Monk Debates are produced by Antica Productions and supported by the Monk Foundation. Rudyard Griffith, Christina Campbell, and Nicole Edwards are the producers. Api Rahaja is the Associate Producer. The Monk Debate podcast is mixed by Kieran Lynch.

Starting point is 00:47:30 The president of Antica Productions is Stuart Cox. Be sure to download and subscribe wherever you get your podcasts. And if you like us, feel free to give us a five-star rating. Thank you again for listening.

The Munk Debates Podcast - Be it resolved: The quest for true AI is one of the great existential risks of our time

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.