Your Undivided Attention - Mustafa Suleyman Says We Need to Contain AI. How Do We Do It?

Episode Date: September 28, 2023

This is going to be the most productive decade in the history of our species, says Mustafa Suleyman, author of “The Coming Wave,” CEO of Inflection AI, and founder of Google’s DeepMind. But in order to truly reap the benefits of AI, we need to learn how to contain it. Paradoxically, part of that will mean collectively saying no to certain forms of progress. As an industry leader reckoning with a future that’s about to be ‘turbocharged’  Mustafa says we can all play a role in shaping the technology in hands-on ways and by advocating for appropriate governance.RECOMMENDED MEDIA The Coming Wave: Technology, Power, and the 21st Century’s Greatest DilemmaThis new book from Mustafa Suleyman is a must-read guide to the technological revolution just starting, and the transformed world it will createPartnership on AIPartnership on AI is bringing together diverse voices from across the AI community to create resources for advancing positive outcomes for people and societyPolicy Reforms Toolkit from the Center for Humane TechnologyDigital lawlessness has been normalized in the name of innovation. It’s possible to craft policy that protects the conditions we need to thriveRECOMMENDED YUA EPISODES AI Myths and MisconceptionsCan We Govern AI? with Marietje SchaakeThe AI DilemmaYour Undivided Attention is produced by the Center for Humane Technology. Follow us on Twitter: @HumaneTech_

Hey, everyone, this is Tristan. And this is Aza. Your undivided attention is about to have its second ever Ask Us Anything episode. So are there questions that you'd love to ask me or Aza about the show? Or more broadly about our work at the Center for Humane Technology and how we want to tackle these questions of AI? So here's your opportunity. Go to humaneTech.com forward slash ask us. Or, and we love this option, record your question on your phone and send us a voice memo at

undividedat humane tech.com. So that's humanetech.com forward slash ask us or undivided at humanetech.com. We hope to hear from you and we are really looking forward to continuing this dialogue. So I think the challenge for the next century is going to be what we don't do rather than what we do. And in the history of our species so far, progress has been a function of what we do. Progress has never been a function of what we say no to. And now, the strange reality is that we actually need to learn when and how to say no collectively. That's Mustafa Suleiman.

He was one of the three co-founders of DeepMind, one of the very first companies whose stated goal was to create artificial general intelligence. DeepMind was acquired by Google and is now owned by Alphabet. Recently, Mustafa co-founded Inflection AI, a public benefit corporation that launched a chatbot name, named Pi for personal intelligence, which aims to create a personal AI for everyone. Now, we don't usually interview CEOs of AI companies on this show, but Mustafa's position as both a creator and a critic of AI makes him unique. He sits at the center of the Silicon Valley race to develop AI. And as you'll hear, he also argues that containing AI is absolutely necessary. Now, the reason why we started with this quote from Mustafa is he's saying something

deeply surprising that progress is defined by what we say no to versus what we say yes to and that's different than every other era. To say no to something takes a kind of maturity to see a power and to not put your hand out to grasp it. And it's unclear through human history how many times humanity has been able to exercise that kind of maturity. I met Mustafa back in 2014 when I walked into his deep mind office in London. And at that time I was a design ethicist at Google, thinking about how we could govern the unethical outcomes we were seeing from social media and the attention economy, arms race. I was looking for allies, and I found almost literally no one who was really thinking about these issues as deeply as Mustafa was. But before that,

we should mention that Mustafa worked in conflict resolution and human rights, so the lens he brings to ethical questions with AI is a deeply thoughtful one. Right, and he has a new book coming out just this month, and it's called The Coming Wave, Technology, Power, and the 21st Century's greatest dilemma. Now, Mustafa is basically arguing these two simultaneous kind of contradictory truths. On the one hand, containment of AI is impossible. On the other hand, containment of AI must be possible, that we have to prevent runaway AI proliferation. Now, Tristan and I see some of these things differently from Mustafa, but we're all asking ourselves the same questions right now. Is it possible to contain AI? And what would it look like? And we wanted to bring some of these

answers to you. So without further ado, here's Mustafa Suleiman. Thank you so much for that kind welcome. It's incredible to be here, I guess, almost a decade on from us both angsting over the same things, debating strategies for intervention, and it's been amazing to journey with you both. So the argument about whether it's possible to contain AI runs through your book. And before we go deeper into that, what does it mean to contain a technology. When we think of containment, what exactly are we trying to do? I mean, the way I see it is that we're trying to monitor, control, and potentially even close down completely technologies that are either in development or they are in deployment. And being able to do

all of those components represents a threshold of containment that I think should give us comfort that these technologies can be managed in a safe way. It doesn't mean that they will inherently be safe, right? You could have something that is unsafe and contained, but at least then we have a framework for governance and decision-making which ensures that they remain accountable to us. Now, people often thought that in the early 50s, that nuclear power, particularly the weapons component of nuclear power,

would itself inevitably proliferate. And the very good news is that we use traditional force and traditional threats and incentives to suppress the widespread proliferation of those weapons. In fact, three countries actually relinquished nuclear power over the last 20 years. And there are some things to learn from that, but fundamentally that isn't a cause for complacency or comfort because, you know, nuclear is incredibly expensive. it's hugely capital time intensive. Handling the material is extremely difficult. And so it doesn't have any of the characteristics of the coming wave. So I think the challenge for the next century is going to be what we don't do rather than what we do.

And in the history of our species so far, progress has been a function of what we do. Progress has never been a function of what we say no to. So it's this absurd paradox that we now have to, you know, put the genie back in the bottle, right? The engine of progress in the last 2,000 years has been invention, science, creativity, commerce, this massive evolution of ideas, right? Which has just turbocharged health and happiness for everybody on the planet for so many centuries. And now the strange reality is that for various, frankly, technical reasons, we actually need to learn when and how to say no, collectively, right? This can't just be unilateral.

Could you paint a picture for just a second before you get into, like, the why it'll be hard to say no for this technology in particular, but what are we saying no to? I think that there are a number of capabilities which we can demonstrate with extremely high confidence, potentially even mathematical provability. Then we should stand back from experimenting with those tools because they're potentially very, very dangerous. So here are some examples. The first is recursive self-improvement, right? So if an AI system has the ability to update and improve its own code and the way that it

works without human oversight, with access to very large amount of resources, potentially unlimited resources, where it's operating on the open web or it's interacting with humans in an unconstrained way, then that recursive self-improvement could be something that might be hard for us to monitor and hard for us to limit. And if you don't just provide it with the ability to update its own code,

but in fact to change its own goals, right, those two characteristics to me could produce potentially quite dangerous systems. So some people want to be hearing that and saying, okay, well, but it's just code running on a computer. It's a blinking cursor. I go to chat.com.

I hit return. That doesn't move atoms and things around the world doesn't create January 6th is what do you mean specifically when you say through this autonomy and generality it could you know start to impact and make vulnerable the actual physical world that we all live in so the way to think about the coming wave is that everybody is going to have an intelligent assistant or an intelligent agent in their pocket it will be able to take a natural language instruction and carry out actions in the digital and physical world. It means that it could generate a brand new website. It could interact

with other AIs in order to persuade them or negotiate with them or establish a contract with them for physically constructing something, building something. It can send emails. It will be able to make phone calls. It will ultimately see what you see and hear what you hear. So it's going to be sort of living life alongside you and interacting in the world just in the same way that you interact with the world today. So, Mustafa, to recap what you said, lots of people are going to be getting a personal AI assistant, an AI that's there helping them all the time. And that sounds like it's going to be a really big boost to personal productivity and overall

economic output. And a lot of people might be wondering then, well, that sounds great. What's the problem? The problem is one of proliferation. Everybody is going to get access at the same time within the same five-year period. Think about it like this. Today, whatever you earn, if you live in the modern developed world, you almost certainly have the same cutting-edge smartphone

as the very richest people on the planet. We all use the same high-quality laptops. And that's because of proliferation. Over the last 60 years, chips have become a critical commodity in our economy, and everybody has the very best quality. hardware. That is an incredibly meritocratic achievement, if you think about it. It's amazing. Two or three billion of us are within five percent of one another in terms of our access to the best hardware ever invented in the history of our species. The same trajectory is going to

unfold with respect to access to intelligence. Everyone is going to have a team of elite AIs in their corner that help to pursue their objectives. And that's the conundrum. It's both incredible from a productivity perspective, but also incredibly chaotic from a civilization perspective. How do you imagine that impacting markets, nation states, jobs, terrorist groups? This seems like it has massive, massive implications. I love you to just, like, give us a tour of what that means, geopolitically down to an individual. I think the first thing to say is that we are already in a state, of huge international anxiety, right? I think in the Western world, there is a fear of climate

change, there is a job insecurity, a fear that democracy is under threat and under attack, a lack of confidence in the political process. On all fronts, there's this kind of general anxiety, which is, I think, creating fragility in our civilization. And that's an important context and backdrop. The real question is going to be who gets access to a technology like that? How much will it cost to deploy a system of that power? Is it going to be open source or is it going to be centralized? Is that capability going to spread to everybody? That will change everything from the cost of labor to the structure of companies. I mean, it really is nothing short of a seismic transformation in what work means.

And so, you know, it's hard to say exactly where that leaves the future of the nation state. But I think that has the potential to cause huge disruption. And then this brings back this question of containment because of the whole premise of this conversation is that there are places where we might want to say no in the name of progress. That progress actually, I love your frame. You know, the progress in this new era of human history is going to be about our ability collectively to say no rather than to say yes to things. Let me say one thing to complicate the picture first, because I think it informs the answer. And the complication is that this is going to be the most productive and fruitful decade in the history of our species. What I've just described to you with artificial capable intelligence, these ACIs, means that many, many millions of people, if not billions of people, will have access to the best lawyer, the best doctor, the best job, the best job.

the best chief of staff, the best coach, the best confidant, the best creative aid, the best project manager. And so the upside here is going to be so sensational that knowing what to stop is just part of the problem, right? Actually persuading people to hold back is going to be the major challenge. These technologies are inherently omni use, which means the same. core technology is massively valuable in every possible domain. And we aren't talking about the simplistic two-dimensional framing of tools or weapons, right? That was the old sort of, you know, dual-use characteristic of military technologies like nuclear, right, which basically has two uses. But these language models, by their definition, have a unique use for every single

individual. So they're not just dual use. They're infinitely omni use because intelligence and language is the technology of life, right? That is how we communicate with one another. It's how we think, it's how we create, is how we do. And so distilling language and intelligence into something that can be reproduced, scaled up, that's what's both going to turbocharge us as a species, but also lead us to run into one another because we already run into one another. It's simply going to set light to the existing conflicts that we have with one another. The differences in culture and politics and commercial competitive constraints. All right. So getting back to the question of how we contain this Omni-use technology, I'm interested in whether you see this

is something that just happens at an elite political level and inside the big tech companies, or is this something that everybody listening to the podcast can somehow be involved in? Yes. Everybody has to get involved now. Defining the standards of what is acceptable is going to shape the future of our species. And that means participating in protest, in resistance, participating in creating alliances, participating in the political process. It means creating and making these tools. Because if you don't know how to really use these tools and you don't make them, you'll never truly understand how good they are at certain things and how bad they are at other things. And that will often be very surprising.

and they're very accessible. They're really not that complicated. You can wrap your mind around it in two or three hours of watching videos online where there's tons of really accessible tutorials. And I think that's a great place to start when it comes to containment. First, stop fearing it as this impossible to understand enemy. And second, deeply immerse yourself in the tools of holding these models accountable and making them and manipulating them

so that you can see the boundaries of their... rather than just imagining their sort of super-intelligences, which they're totally not. The bigger the models get, the more controllable they are. You know, that's an amazing outcome that we are creating very nuanced, very precise, very specific behaviors in these models. That's a great story because that regresses us to the traditional governance question, which is whose values, what should those behaviors be, and who gets to decide? I've known you for a long time of stuff. I see you as someone who has taken on a tremendous burden and worked exceptionally hard to try to figure out how do we ethically govern the impact of technology on society.

And if I recall correctly, when DeepMind, the most powerful AI company in the world at the time, it was acquired by Google as part of the agreement which you helped construct, Google's use of DeepMind had to be governed by this ethics board that you put together. And so just to sort of say, you have been thinking about how do we get this right? for a very long time. And that was back in, what, 2010, 11 or something. And then later on, you know, building the Google applied ethics board and doing a number of things to try to create different incentives for the dynamics that normally govern how companies would deploy AI in ways that may not always perfectly match up with the best of humanities

plus, you know, public interest. You know, you're totally right that it's the incentives that really make a difference here. And how can we create a new set of incentives? I've really been obsessed by that question since founding DeepMind. When we founded DeepMind, our strapline in our business plan was building safe and ethical AGI for the benefit of everybody. And that really influenced the culture of the company in a very fundamental way.

We attracted a certain type of person as a result. And I think it had a non-trivial effect on the way that OpenAI formed and then Anthropic 2. When we were acquired in 2014, you're right, we made it a condition of the acquisition that DeepMind's technologies could never be used for military purposes. And, you know, we also had, as you said, an ethics and safety board with actually six independent members, outsiders, which were jointly appointed by us at DeepMind, me and Demis and Shane, and, you know, Larry, Sergey and Sundar at the time.

And that was the first steps towards trying to bring in some outside oversight over how we were developing these technologies and how they were governed. And over the course of five or six years, whilst I was a deep mind, I really tried hard to get Google to add an ethics board of its own. For a very brief time, it did have an ethics board. It had its own internal charter, which I drafted the AI principles, the Google AI principles, which are still in operation today. And although there isn't an independent oversight, and unfortunately that all fell apart very quickly, the charter of AI principles is still in operation. and there's a big team now that are focused on responsible AI.

So although it's not glamorous and publicly recognized, there is an operational culture shift in Google's approach, which is now more precautionary, is more proactively attentive to the risks because there's a kind of bureaucratic process in place for scrutinizing

all of the launches that Google does to make sure they adhere to its AI principles. So it's not a complete solution, but it's a first step. I also co-founded the partnership on AI, where we brought together the six big technology companies, Apple, Microsoft, IBM, Google, Facebook, DeepM, Open AI, along with now I think 120 civil society groups from Oxfam to ACLU to everyone's now a member. And it's a forum, you know, it's a way to share best practices.

It's a place to meet other people in the field who are concerned about these issues. It's a place to organize and convene and debate. And they produce lots of good research and lots of. of good events now. So all these are kind of very small steps. You know, none of it in some solves it, but it's, you know, we have to be optimistic. You know, my phrase, containment must be possible because, yes, it does look like we default to mass proliferation and that fundamentally changes our civilization. But it has to be possible. And the way to do that is that we all have to take small steps and be active and participate. There's a question that's in my mind and I bet it's

in the mind of many of our listeners as well. You know, you're the CEO of this new company with Reid Hoffman, Inflection AI, and it's working on creating these personal AIs for everyone. How do you reconcile working in this space, participating in the race, and at the same time raising the alarm about it? You know, inflection was founded as a public benefit corporation, right? What that means is that we have a legal obligation

as directors of the company of inflection to attend to our mission, which is to build AI for the benefit of everybody in a safe way. So we have to think about the harms as a legal requirement proactively in our documentation. So it's not just about satisfying shareholder interest. It's about these other potential harms. And I think little changes like that

really make a difference over the long term. So given all of those attempts that you have tirelessly made over many, many years, to try to create governance of individual AI actors and deep mind and inflection and so on, there's still this question of what external system of governance would be needed to bind this global race. And that gets back to this question of containment, both between companies that are all racing, and maybe they individually have a long-term benefit trust the way Anthropic does or are incorporated as a public benefit corporation the way you are. But that's not enough

when it comes to the overall race dynamics and the pull that pulls us towards, if I don't do it, I just won't have a say in the future, so everyone has to keep building and increasing capabilities to match the speed of the other guys. What is your thought there? I think it needs governments that are prepared to concede and to cooperate because we've talked a lot about how there are going to be proliferation dynamics because people want access in open source and individuals and small organizations want to create their own forms of AI. But we haven't talk so much about the proliferation dynamics that result because governments and militaries are going to be chasing this like there's no tomorrow. And we're already seeing that.

You know, I've been tracking the lethal autonomous weapons negotiations at the UN for the last 15 years. And sadly, we've made almost no progress, I think it's fair to say, in the negotiations. We haven't even been able to define what autonomy means and what it means to have a human in the loop, right? Meaningful human control is the phrase that they debate over. And naturally, that's because autonomous weapons provide asymmetric advantages to small organizations or small nation states or militias. And nobody wants to be the first to say, I'm going to step back from the brink here. It might mean that war is cheaper. It might mean that there are fewer human casualties, at least on one side, if not both.

And so it's a very attractive potential for nation states. And of course, AI has the same potential. And so there's also this competitive race dynamic among nation states, which is driving the default to proliferation. So how do you address that? Well, at the end of the day, whilst these models feel like they're kind of evolving super fast because they're in software,

that software still runs on hardware and that hardware is physically located in a data center on territory which is subject to international law most of the time, right? And those chips are manufactured by a very small number of players. You know, Nvidia produces the vast majority of AI chips that train these models

that produce these experiences. And that represents a significant choke point which nation states can, you know, both track the location, of these data centers. They can impose regulations on people who operate the data centers to report risks or to comply with other kinds of safety standards. So that's going to become the kind of focus of negotiation in the future. And successful containment strategies are most likely going to be around sort of licensing, regulating and monitoring who gets to use what chips

and where they sit, which data centers they sit. What I hear you speaking to is what's also kind of discussed by Sam Malman and others is sort of this international regime for monitoring the flows of chips in the world, where does NVIDIA sending its chips, where are those aggregated, where do those sit in data centers, and we're going to need, whether it's licensing or auditing or tracking, and these kinds of things that are being discussed right now, legislatively in the U.S. and elsewhere.

Whether that results in an IAEA, it's some kind of international body that does that. But that's kind of what you're speaking to, yes? That's exactly right. I think we need an international body. And I think the better example is actually modeled on the financial stability board, which is an international group of central bank governors, which is very collaborative, has coordinated on many, many occasions in the last 15, 20 years in real time and overnight

to avoid catastrophe in financial markets. And so I think we need a geotechnology stability board, which is, proactively coordinating with all of the big corporations and companies like mine that are developing these kinds of models and all of the governments that are doing the same to do everything that you said audit transparency oversight licensing regimes etc etc and that that probably is a new organization and just just jump in there to say also agree that it is incredibly challenging and the hard work is going to be how do we know when we're getting it right Which is different than how do you stop the race.

You know, if exactly as you say, we are going to be in deep transformational relationships with the AI systems, many people are building. But given that distinction, how do we know that they are not transforming us in ways that we will later regret or do incredible amounts of damage? And we're going to need some kind of, I know people don't really like the FDA model, but like we need some kind of FDA to know that before we get into a transformative relationship, the things we're entering into relationships with, are good. And to me, it's almost like a new project for understanding, like, what is it to be humane? Because in every way that these systems do damage that we cannot tell that they're doing damage, in every domain from the effable to the ineffable, they will do. So I think that's true, but they're not the harms that I really worry about. So I think, Aza, maybe think about, just assume that the humane AI is eminently possible within the next

three to five years. Just assume that it is going to be infinitely smart, infinitely patient, infinitely respectful, really boundaryed, really healthy. Just assume that and then think about the potential consequences for those kinds of AIs in the world. What does it mean that you have access to infinite amounts of kind and respectful and patient interactions, right? So that's one sort of simulation. The other is, given that you can create such precise and and specific behaviours like the ones I've just outlined, many other people are going to use the same tools to create a slightly more aggressive AI,

or one that's really good at spreading fear and anxiety about the threat of vaccines, or one that is really good at spreading sticky conspiracy theories about this new politician. And that's the challenge. So the risk is not that an AI goes wrong. The risk is that it is working as intense, but that intention is to sow deceit and disruption.

And we do not have a governance framework given our kind of default free speech world. That's the challenge. It's the overlapping false positive moment. Like it's where the lines are really, really blurred. It's not that the AI won't be humane. It will be superhumane. But some people are going to use it to persuade

on agendas that are really antithetical to peace and prosperity. And it may be very difficult for us to prove that and to counter-argue that actually this is bad. So we have to get good at describing why a particular activity like that is bad for democracy, it's bad for people's health. Because it can't just be, well, these are our values

and we don't like your values, because that's what it might regress to. That's the problem from a kind of communication perspective. How do we describe what bad looks like? We need a framework for that. So given that we lack many of the things that you're describing that we need on the way to containing AI,

like an international body that's collaborating with big companies, developing these models or a way to describe what we don't want to happen in a way that everyone can agree on. Given all of that, is there any cause for optimism here that you can see? Yeah, I think my take is that when people have an opportunity to see the sort of harms and benefits over the next five to ten years, I think it will change the popular conversation entirely. I don't think it's possible to really move the needle

on the collective psyche today because everything is so theoretical, right? I mean, I may think that I can see a potential impact, good and bad, of the coming wave, but, I mean, it pales into insignificance when you actually have the opportunity to experience these models, right?

I mean, I was trying to sort of persuade you of the significance of LLMs two years ago, and I could barely get it on your radar, let alone everybody else, And so I'm very conscious that like from a communications perspective, until you deeply experience and really get a chance to play with this new putty, this new clay, you won't understand where it's crap and where it's amazing. And it's just really difficult to convey that until you get to experience it. So it's only now that everybody has seen chat GPT and GPT4 and all these other models that everyone's starting to kind of catch up essentially. on this AI governance debate, which is great. And so part of the reason why I feel optimistic about it is that I think that that's going to massively escalate as we start to see the harms unfold in practice, which there will

be some really horrible outcomes. But I think on the whole, there are going to be incredible benefits that basically drive people to say, yeah, I want to be able to make that too. I want to be able to learn from that. I want that to help me. I want an AI that is going to help me with my research. And that's why I sort of feel that there will be a reaction, a popular reaction in the next few years. And it may be surprising.

It may go different ways. A lot of people absolutely love talking to these models. A lot of people don't have the same taboo that we thought that they might have a year ago about having deep and intimate conversations with these models. And so I think we should just keep an open mind to how this might unfold. fault because I think it's going to be quite surprising. Mustafa, thank you so much for coming on your undivided attention. I know we're going to be in this journey together for a while.

Thank you, man. I appreciate that. Your undivided attention is produced by the Center for Humane Technology, a non-profit working to catalyze a humane future. Our senior producer is Julia Scott. Kirsten McMurray and Sarah McRae are our associate producers. Sasha Fegan is our managing edict. A very special thanks to our generous supporters who make this entire podcast possible, and if you would like to join them, you can visit humanetech.com slash donate. You can find show notes, transcripts, and much more at humanetech.com.

And if you made it all the way here, let me give one more thank you to you for giving us your undivided attention.