The Rest Is Politics: Leading - 33. Mustafa Suleyman: Will AI save or destroy humanity?

Episode Date: August 27, 2023

How much do you really know about the dangers of AI? Is it the best hope we have of surviving as a species, or will it lead to the destruction of humanity? On today's episode of Leading Alastair and Rory are joined by Mustafa Suleyman as he recounts his rise from North London council estate to the vanguard of the tech revolution. TRIP Plus: Become a member of The Rest Is Politics Plus to support the podcast, receive our exclusive newsletter, enjoy ad-free listening to both TRIP and Leading, benefit from discount book prices on titles mentioned on the pod, join our Discord chatroom, and receive early access to live show tickets and Question Time episodes. Just head to therestispolitics.com to sign up, or start a free trial today on Apple Podcasts: apple.co/therestispolitics. Instagram: @restispolitics Twitter: @RestIsPolitics Email: [e-mail] Producers: Dom Johnson + Nicole Maslen Exec Producers: Tony Pastor + Jack Davenport Learn more about your ad choices. Visit podcastchoices.com/adchoices

Thanks for listening to The Restis Politics. Sign up to the Restis Politics Plus. To enjoy ad-free listening, receive a weekly newsletter, join our members chat room and gain early access to live show tickets. Just go to therestispolities.com. That's the restispolities.com. Welcome to the Restis Politics leading with me, Rory Stewart. And me, Alistair Campbell. And our guest today is Mustafa Suleiman. And Mustafa Suleiman is famous for being one of the people right at the heart of the AI, the Artificial Intelligence Revolution. He is co-founder of Deep Mind, which was the big revolutionary company that did some of the very early work. He's now co-founded another major AI company.

He's probably one of half a dozen or a dozen people in the world who are at the center of all these changes. But he's also actually, I think very interesting for listeners in other ways. He comes from a working class background in London, Muslim background, father Syrian. And during the course of this interview, we will, of course, talk about AI. and some of his anxieties about the threat that AI poses. But we will also talk about a young man who left university after a year to set up a Muslim youth helpline, who worked for Ken Livingston, the mayor of London, obviously had very left-wing politics in his youth, and the way in which he reconciles his position now as this very, very powerful wealthy figure.

He's talking to us from Palo Alto in California with his youth growing up on the Caledonian Road. Anything's you're interested to get into in this? Well, I've just read his book, which is coming out shortly, called The Coming Wave, Technology Power and the 21st Century's Greatest Dilemma. And it is one of those books that sort of, I don't know, keeps you awake at night. A lot of worrying stuff in there. And it's about AI. It's about how this thing is working or isn't working. And it's about, as you say, the threats that it poses.

And what, if anything, we can do about them. A lot of it's about containment. He has this idea that we have to contain it. So, without further ado, let's talk to Mustafa Suleiman. Can you just give us an overview? What on earth is technology? I mean, we keep talking all the time about technology in the broadest sense. What do you mean by technology? That is a very good question. First of all, Rory Alistair, great to be with you, both. Thank you for having me. Technology very simply is a tool for amplification. I mean, anything that enhances strengthens, accelerates, augments our capabilities as humans is a technology.

The glasses that Alistair and I are wearing are technologies. They augment our eyesight. And the quest to invent and create new advances which make us as a species more efficient, more productive, more capable, is as old as as humanity itself. And so that's, I think, in the broadest sense, how I think about technology, rather than, I think people might tend to say that it's digital technologies or it's something sort of more narrow than that. I really think of it in the broadest possible quest to invent science and turn those scientific advancements into technologies. Okay, so you brought up our glasses, and that made me go to one of the lines in your book that leapt out of me, which is the fact that this exists now. Chinese police officers who wear sunglasses with inbuilt facial recognition technology and presumably can also have technology that then can lead them to recognise you,

Mustafa Salaman, from just looking at you but then actually know all sorts of things about you. And that then leads me to worry about the whole sort of surveillance state. This is something we talked about with Yuval Noah Harare, last week on the podcast. And then if I may go to another bit of your book, you say, imagine robots equipped with facial recognition, DNA sequencing and automatic weapons. Future robots will not be scampering dogs. They'll be the size of the bird or a bee with a small firearm or a small vial of anthrax. This will be accessible to anyone.

This is what bad actor empowerment looks like. And that feels quite scary to me. So should we be scared or should we be confident that we can look after ourselves as humans surrounded by this thing that you guys have rooted upon us? I think if you don't start from a position of fear, you're probably not paying attention. And part of the reason why I sort of wrote this book is that during the pandemic, I kind of had a moment to sit back and reflect and really take in what is happening on a sort of multi-decade scale.

And what became very clear to me was that we are sort of at the mercy of these unfolding exponential trajectories. And so what that means is that as technologies get more useful, they get easier to make, they get easier to use, and they get cheaper. And so they spread far and wide. And that's been, you know, the engine of progress for many, many centuries. We invent things. other people demand them, they get more efficient, they get cheaper, they spread, they empower everybody. And so if that is going to be the story of invention for the next 40 years or 100 years,

we have to ask ourselves some quite fundamental questions about what that concentration and distribution of power means for the future of the nation state, for what it means to be human, and many other really quite fundamental questions. And that in itself can immediately start to feel quite scary. For the benefit of our listeners, but also very much for the benefit of me, a few terms and definitions, please. Tell us what is artificial intelligence? What is a large language model? What is machine learning?

Okay. So in short, artificial intelligence is the science of teaching machines to learn. So rather than handcrafting a set of rules, if this, then that, you want to. to get an AI system to learn a representation of what is useful or productive in some environment itself. So it's really about teaching the machines to learn. A large language model is a type of artificial intelligence. It is a very large deep learning network, which has seen or been trained on many, many hundreds of billions of words from the open internet. And what it does is really sentence completion. You provide an input and then it predicts, given all of the previous words

it's seen, so everything on the open internet, what is the likely next word given some sentence or given some sequence? So it's actually a very, very simplistic but enormous matrix multiplication. It's doing an all-to-all connection between all previous words and those words that are in current context. And that's pretty incredible. And the large language model is just one subset of artificial intelligence. There are many other bits of artificial intelligence, right? There are, yeah. I mean, it is the dominant approach that has taken over in the last sort of three or four years or so.

So the GPT series from OpenAI, Pi from inflection, Google's models, they're all using large language models or various different flavors of the same infrastructure. And, can you just run through? And I understand that as a sort of spoiler alert, you have some practical proposals for how to contain the technology and you're clearly not trying to spread fear and paranoia. But could we start at the bad end and the worst case scenario and just lay out some of the things which could go wrong with AI in relatively simple terms? Well, I think, you know, Alistair pointed at some of the examples in the book. I mean, essentially these technologies make it possible to take actions in the real world. So the last wave of AI was about classification, right? So the last 10 or 15 years was about deep learning systems, learning to identify objects

in images, learning to transcribe audio, and increasingly learning to understand what was in sentences, in language. And that was really about classification, ordering, structuring, creating classes. having now understood, you know, or at least been able to classify effectively, these models can then take those classes and generate. They can predict. And prediction, of course, is a necessary prerequisite to taking actions. Before you do something in the world, you imagine, you simulate, you plan, like if I'm going

to go to the train station, you lay out the route in your mind and you have to predict something about, you know, what the weather's going to be like and, you know, is this route going to busy and am I going to get there on time? That's a generation. It's a rolling out of a sequence of activities which allows you to be creative because you can roll out a whole range of different sequences about how the world around you might unfold. That's essentially what we call imagination. So being able to roll out these sequences and then pick one path is like taking an action, right? It's actually now beginning to do things in the real world. And so these models enable, I think, in the limit, you know, everybody to take actions and intervene in the world in very new and novel ways.

And I think that that's going to be the real shift compared to AI that we've had over the last decade. I've heard you say in the past that there are a small group of technologists who believe that AI will get to a stage where it will be indistinguishable from human self-awareness and capability. and that they'll begin to evolve faster than we can. And if that happens, that will be a singularity that cannot be contained. That will mark the beginning of the end of the species. Can you expand on that? What do you mean by a singularity which can't be contained? And what would that feel like practically?

I mean, you're producing a beautiful philosophical analysis, but I'd love to get a sense of some practical implications of this. Well, everything that we have produced in our world, everything that you look at in your line of sight today, is a product of intelligence. It's a product of you, you know, looking at something, trying to understand it, and then trying to predict it well enough that you can intervene in that environment. That is what intelligence is. And we're now trying to distill intelligence, the thing that has made us so productive as a species into an algorithmic construct.

So if many, many people have access to that, then it will be the most productive period in the history of our species. So when it comes to creativity, and invention, we now have many millions of people who can make new things. And I think that in itself is likely to be the most productive period in a very, very long time. But of course, the flip side is if you design those systems to have, you know, certain capabilities like autonomy or the ability to update their own goals or act independently of human beings, that's when, you know, they could potentially be pretty harmful or pretty dangerous. But I get the feeling, Mustafa, that when you went away during COVID and you had a moment of reflection, you started to write this book, I get the feeling reading the book that you're much, much more pessimistic than you're sounding now.

Is that a misreading of the book? Or is that maybe me imposing my own sense of pessimism upon what you're saying? Because I guess I've got two problems with it. One is I'm not, I don't fully understand how it works. And that is always scary. For something like me, I like to understand everything. and I don't understand when you say this thing has got the potential to essentially be more intelligent than human beings are. And then when I read the statement, like the one that you signed with a bunch of other industry leaders,

mitigating the risk of extinction from AI, let's just read that again, mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war. I don't know how to read that other than with sort of pretty profound fear. And then at the same time, you guys are there doing what you were doing, and you have people like you, people like Jeffrey Hinton, starting to think, hold on a minute, guys, are we actually doing something potentially very, very bad here? So I think the surprising thing has been that what we have thought of as unique and very special

and impossible to replicate skills and capabilities that we have as humans, our intelligence, has actually been relatively easy to replicate, right? That's been the surprising thing that's happened over the last decade. When we sort of set about this quest, when we found a deep mind in 2010, we had no idea how hard or how long it would take to make progress in some of these areas. And slowly, but surely, we've been knocking down, some of the big milestones. Well, it's not been that slow, is it?

It's probably not been that slow. It's true. It's been a decade. It feels slow to me. You know, we now have near-perfect transcription. We now have near-perfect image understanding. Increasingly, we have image generation, which is really very good. We have text generation, which is very good. The next big milestones over the next decade will be planning and, you know, imagination. So, you know, it's true.

It's been easier. than we previously thought. But if I were to go back to the bird or the bee with the small firearm or the small vial of anthrax, would I be able to have access to that kind of thing quite easily within a few years unless there is what you call containment? Correct. So I wouldn't say within a few years,

but certainly within a few decades. So the trajectory, the historical evidence is very clear, where something is useful, it gets cheaper and it proliferates. So far in the history of our species, everything without exception that is valuable and useful spreads far and wide. So that's the default that we have to contend with. And that is extremely true in the context of software because software isn't constrained by the traditional frictions of the physical world. Software can be easily moved around on the web and so on. It's an idea, right?

So it spreads even faster than a physical invention like a motor car. So I think what I'm sort of trying to establish in the book is that if everything defaults to getting cheaper and easier and spreading far and wide, then we have to contend with power, the ability to take actions and do things in the world being very, very widely available to many, many people. So to be cheesy, a couple of examples that I guess trouble people. One of them is the next U.S. presidential election and a real fear that AI. can generate deep fakes or intelligently manipulate voters in such a way that it could affect the outcome of the election. Or another fear might be that in a few years' time, some version of, I don't know, chat, GBT4 would allow me to start doing stuff that at the moment could only really be done by somebody with a doctorate in biochemistry. And I could start making some pretty dangerous substances. These kinds of things,

seemed to me to, well, initially actually I remember talking to you about this in Japan. And in Japan, we were around the table, and I tried to say, is this like nuclear weapons? And the optimists, there are a lot of optimists in this world, said, oh, no, no, no, no, it's much less dangerous than that. You're making it sound much too scary by saying it's like nuclear weapons. But what I remember you saying is that actually in some ways, it has features which are more dangerous than nuclear weapons. They're easier to proliferate these technologies in nuclear weapons.

Can you expand on that a bit? Yeah, I mean, the cost of production of nuclear weapons is one of the things that has halted its proliferation. So we can clearly see where nuclear weapons facilities are in the world. They're large, they're physical, they involve vast numbers of people, they involve, you know, scooping up chunks of uranium 235, which is in itself very rare. And that friction, the capital costs, as well as the kind of physical constraints, make it, you know, sort of very different to the way that software evolves. I mean, take an image generation AI today has been trained on many billions of images that are available on the open web.

The model itself is now about two gigabytes, so it can sit on a thumb drive. And in some sense, what's happened is it's compressed all the knowledge and insight from all the different images that it has seen, if you like, onto this tiny little transferable representation which can be shared and copied. and moved around, et cetera, and now it can be used to generate pretty good, you know, not perfect, but pretty decent images. If that trajectory continues, then I think we can all imagine what the world would be like if we had perfect, photorealistic audio, video, language generation that was entirely transferable and indistinguishable from human generated content. In fact, it would likely be in the limit over a 10 to 20 year period much, much better.

So your point about the election is a very good one. I've said publicly many times that we should immediately call for a moratorium, a ban on the use of AI generated content for any kind of electioneering. And certainly, you know, in the case of AI chatbots, those topics of conversation should be off limits because they're going to be very persuasive and we also can't, you know, fully verify that they're going to be accurate and reliable. So people should not rely on them to do that, and the manufacturers of them shouldn't, you know, put them to those purposes. But that's, you know, if you think that one of the contenders in the next presidential election is almost certainly going to be Donald Trump, it is impossible to imagine that Donald Trump is not going to use something that might help him become president.

And I want to ask you about politics and this, because you briefly, it's true, but you worked in politics and that you were an advisor to Ken Living, someone who's mayor of London. you were one of the AI people that had a meeting with President Biden not long ago to kind of raise some of these issues with him. First thing I want to ask you is, how is politics dealing with this in your view? What is the tensions between the fact that your world operates at such pace and the political world operates pretty slowly? And I guess I want to ask very directly whether you felt Joe Biden got this and if so, great. if not, do you know any of the current cropper world leaders who do get this? I mean, so just going back to the Ken thing. So I did briefly work for Ken when I was 21 as a policy researcher,

a non-advisor. It was a very interesting experience. It was only sort of nine months or so. But I certainly learned enough to realize that I probably needed to get into technology rather than politics. That was important. Yeah, I mean, look, this is partly why I'm writing the book. I mean, we need our institutions, you know, those that

defend and represent the public interest to evolve as quickly as the technology is evolving. And everyone would agree that that's clearly not happening. Of course, it's true that Trump will most likely, and many other like politicians will want to use every possible advantage to win and get ahead. And that's again the story of proliferation, right? We have these huge geopolitical incentives, whether it's China or the UK or the US. Everybody wants to get that advantage. That's what technology is. It provides an edge. It's an advantage. It's a motivation to, you know, take risks and push the boundaries and advance your agenda, whatever that is. So, you know, I think the White House administration in general, this one is on the case. I mean, they have been very bold with their

export controls, whatever you think of them. That's a very significant intervention. holding China back from the cutting edge of AI chips, which is a significant restriction. They've been very proactive with the voluntary commitment. So, you know, sort of the top seven companies building the largest AI models in the world, one of which is mine, inflection AI. And, you know, we all signed up to these voluntary commitments to expose our work to independent audit, to share best practices, et cetera, et cetera. So I think that things are heading in the right direction and there's cause to be optimistic

about that. Likewise, I think that the EU AI Act is for sure heading in the right direction, this threshold of evaluating risk of an application rather than trying to define the capabilities makes a lot of sense. Tell us a little bit about that, Mustafa. Expand on what the EU is trying to do. So to be fair to the EU, they have been ahead of the game for many years now. They've been drafting this EU Artificial Intelligence Act for over three years, maybe four years at this point. And their approach has been to say, we will require developers of AI to proactively assess the potential risks of their AI application. And if there is, you know, a significant risk, if it exceeds some threshold, then they have to disclose the data that that AI system has been trained on. They have

to disclose how the model operates. You know, why is it making a recommendation? Why is it not making a recommendation. And there's a sort of reporting requirement that basically imposes a kind of a precautionary constraint on activity. But when I was talking to some of your colleagues about this in the States over the last few weeks, many of them are very angry with the EU and that some of them are saying, well, that's the end of the EU, they're never going to make any technological advancements. They're ridiculous. They're Luddite. They're in the Stone Age. We're just going to go and do our business somewhere else. And presumably that is the problem here, that unless the entire world, world, let's just say it's a bit like climate change, isn't it? You just need a few bad actors prepared

to host people doing crazy stuff and the whole thing collapses. Well, that's the story of technology, right? The challenges that we're all collectively in this horrible tragedy of the Commons race, but that raises the question of sort of, is there going to be a race to the bottom or are, you know, some people are going to hold the line and defend their values? I personally believe very much in the EU AI Act approach. I think a precautionary of principle is healthy and correct. And, you know, I don't think the cost of innovation are anything like as big as what some people have speculated. And I think that you might have been talking to the classic Silicon Valley types and I'm not one of them.

I guess the reason you're maybe not one of them is your background is quite interesting for, I'm guessing you're meeting a lot of, you know, very smart, very, probably quite privileged and entitled people. And yet your dad was, you born in London, your dad was Syrian, a taxi driver. your mum, a nurse. She was English, you're British, yeah? Yeah. Just give us a little bit of how you got from that kind of upbringing, pretty working class background in London, to now being pretty successful, vastly wealthy by comparison with most of the kids you grew up with. And, you know, as Roy said in the introduction, at the cutting edge of one of the biggest issues of change of our time. Well, I mean, so I grew up on Caledonian Road in London, Council of State, and, you know,

And then when I finished primary school, I was pure chance. I was very lucky. Someone at school said, oh, you should check out this really good grammar school up in Barnett. You have to sit an entrance exam. And I was like, okay, cool. I'll try that. And it was a nonverbal reasoning exam, two exams, actually, English and Maths for nonverbal reasoning.

And I did really well. It was Queen Elizabeth boys in Barnett. It turned out to be, you know, the best state school in the country in terms of GCSEs and A levels and it changed my life because it just, I was surrounded by people who also kind of wanted to work hard and enjoyed learning and they just pushed us really, really hard. And it definitely changed my life going to that school. It gave me a big opportunity. What was in your mind that led you, via Ken Livingstone and other things, but led you to what you do now, what sort of mind you need to be in the AI world? Is it about words? Is it about maths? It's about

science, what are the strengths that you need? That's a great question. I think it's about systems thinking. So I did philosophy at Oxford, and I think that what that training gave me was it honed my instinct for thinking at varying levels of abstraction. I can move from micro to macro, and I think in terms of systems and causality. And I'm also comfortable not knowing. You sort of have to sit with a certain amount of ambiguity. And, you know, I'm a translator and an interlocutor. You know, I get different groups of technical people to, you know, communicate well with one another, to talk to product people, to talk to operations people.

And so I'm really playing this sort of constant translation role in order to steward the big picture vision. And I think that's quite an important thing to keep in mind because much as people are scared about the potential outcomes, they're also scared about the complexity of these ideas. Everyone's like, oh, it's complicated AI, I could never do it. It's just nonsense. It's like totally accessible. There are scores of videos online.

You could watch two or three hours of videos. And if you're patient enough, you can get through roughly the big picture of like, what is this system trying to do? And, you know, I'm very lucky to have been doing that for 15 years now. And I have a pretty good technical understanding of everything that's going on in the field. And I think that's available to a lot of other people. And we should be pursuing it because that's caused. to containment. You have to understand something in order to be able to steward it.

One of the interesting things, and many interesting things in your life, is you left Oxford after only a year or so to set up something called the Muslim Youth Helpline and worked for that, I guess, for three years. And would love to know a little bit more about that. It's a fascinating thing to do. What was it doing? Why Muslim youth? Why was it such an attractive thing that you were prepared to throw away your undergraduate degree? What does your parents think about this whole thing? Tell us a bit about that stage your life. Yeah, so I was raised as a very strict Muslim. My mom was a convert well before she met my dad, so, you know, the strongest type. And yeah, when I got to Oxford, I had managed to leave the religion and, you know, in my

philosophy, discovered human rights principles and just found it tremendously inspiring. I was like, here is a language and theory of justice that applies to everybody, not just Muslims. We are not the chosen race, and that always made me kind of uncomfortable and confused, but this gave me an intellectual framework for moving away from religion. I became a very staunch atheist. But during my kind of philosophy trading, I also got quite frustrated at how theoretical and abstract the work was. And I wanted to kind of do good now.

I was truly, like, eager and motivated to have a positive. impact. And there were a couple of other people I met at Oxford who were already working on a small helpline. And I decided to quit, move to London and help them full time. Was it a difficult time in your life where people thought this is crazy? He's left university. Nothing's going to work out for him. He's floating around. Was that a difficult period? And also related to that was what are your mother in particular think of you suddenly unfinding God, as we might say? That was a very challenging one. But, you know, I think. going through sort of secondary school, it was quite difficult for my parents to understand

what they had produced. They were just like, why don't you, like my mom was like, why don't you leave school at 16, you know, get a trade, you know, why are you messing around with all these like ideas? She was really eager for me to like get a good trade and that's how you'll be safe, like get, be a plumber, you know. And so I think they sort of didn't quite grasp why I would want to go to Oxford, why was I was doing philosophy and theology? So by the time the Muslim Youth Helpline hit, I mean, they just were, that was like, what on earth are you doing? So that was, they were definitely confused by that. But the Muslim Youth Helpline, to me was, it was a secular, non-directional, non-religious effort. It is a very simple practical tool. It took traditional

counseling and peer-to-peer support practices and just skinned it with a little bit of faith and culturally sensitive language. So it was sort of Samarical. for Muslims? Yeah, exactly, kind of like that. But the key thing was it was supposed to be non-directional, which it was. Everyone understood that this was about Sunnis and Shiaz and people from Bangladesh and people from Sudan and Arabs and Malaysians working together in the common interest to provide, you know, support. And it was non-religious. That was really the main sort of goal here is to move beyond, you know, sort of sectarianism. Musfer, thank you very much. Can we take a quick break?

Hi everybody, it's Dominic Samarach here from The Rest is History. Now, some of you may have heard me on your show, The Rest is Politics, when Rory was away, and I was filling in and enjoying Alastair Campbell's tremendous banter. And I'm back to tell you about our new series on The Rest Is History, which is all about Britain in the 1970s, a period with a lot of uncanny resemblances to our own. So right now we're living through a moment when oil shocks generated by war in the Middle East, are rippling through the world economy, when Britain feels like it's sunk in a bit of a malaise. People are arguing about Europe.

The government has got a few issues with the trade unions. And we have a kind of, I suppose you'd say governing elite, a kind of political class that is really struggling to come to terms with all of these issues. And people are asking if Britain is governable at all. So there are a lot of parallels between that Britain that I'm describing, which is our Britain and the Britain of the mid-1970s. So in this series that's coming out on the rest is history, we'll be looking at these and other issues. We'll be talking about the rise of Margaret Thatcher, obviously a colossal figure in our political life even now, whether you love her or loathe her.

We'll be talking about the very first Brexit referendum of 1975, a subject that I'm sure Rory and Alistair will have strong opinions about. We'll be talking about the fall of the Labour Prime Minister Harold Wilson. And we'll be talking about one of the grimest. moments in Britain's economic history, the moment in 1976 when we had to go cap in hand, as people said at the time, to the International Monetary Fund, the IMF, for a then record bailout. Now, if that sounds good to you, how could it not sound good to you? Of course it sounds good to you. We have a clip for you to listen to at the end of this episode. And if you want to hear more, just search for The Rest is History wherever you get your podcasts.

Can I just, I mean, I'd like to bring in Alistra on this, because Alice has obviously got a very deep interest in mental health. And I'd be interested here the two of you just chat for a second about that, because why was it particularly mental health that you were drawn to? And their helplines are very particular things, a very particular way of doing good in the world. Yeah, well, it was just after 9-11, so it was sort of 2002. And all the talk was of where the next British Muslim threat might come from. There was a lot of Islamophobic. although we didn't have that word particularly for Islamophobia. You know, if you were British and Muslim, you were four times more likely to be in prison. There was this constant sort of fear and anxiety that there was a threat from within. And that, you know, that kind of motivated me. There were these basic issues that were getting dressed up as mass political issues. People were just fundamentally suffering, bullying. They had difficult lives at home.

They were struggling with sexuality. And on top of that came this huge. sort of political narrative and I think it was causing a lot of confusion and that was really what I wanted to tackle. Would you say you've got good mental health? I would say that I have learned to manage my mental health. I've I've learned coping mechanisms that help keep me in moderate equilibrium but it's certainly something I've always wrestled with for sure. and wrestled upwards, moved upwards or moved downward or both? You know, I think both.

I'm very driven and have a lot of energy, but also, you know, I'm constantly wrestling with demons and darkness and difficult, you know, childhood and difficult, you know, memories and, you know, as I'm sure many people are. And what was it about Islam that attracted your mother? And what was it about Islam that you decided it was Not for you. I mean, it was very clear to me that the kind of like righteousness and the elite believers,

I mean, I can remember the most basic question asking my dad when I was really young was like, well, what about the good people who aren't Muslim but are going to go to hell? That doesn't seem right. They're not that bad. You know, like I was thinking about my friends at school, of course. And there's just no good answer, even to a child. for any of these really fundamental questions. And so I was always left, like, frustrated and it felt incomplete.

You know, for my mom, she was friends with Kat Stevens. And when he converted to Islam, I think, in like 1980 or something, she sort of followed that path and took his name and so on. So, you know, that was her sort of journey. This is not in the book. This is not in the book. But this is a freebie for you since you are. But it fascinates me because I think that this whole thing about what we believe and how we believe it, how deeply believe it.

But I think it's one thing to be raised in a faith, but it's quite another one to choose a faith you're not raised in. It's definitely very different, yeah. I mean, and many people choose, you know, sort of savior and choose the kind of support network. I mean, there is something very attractive about the idea that you can begin again and that you're going to be, you know, sort of, was supported and embraced by a group of people who are moving beyond race. I think it's very important in a lot of the Islamic narrative is like it's post-race. You know, provided you believe this, we can all be equal other than those, of course, who don't believe it.

So, you know, I think that's very attractive to some people. But do they believe you're going to hell? Because you don't believe now. Well, I think over the time my mum has sort of left the faith as well when I was a bit older in my 20s. So probably not. But I think my dad probably does, which is a funny,

a funny thought. And how do they feel about your kind of mega success and the sort of millions in the bank and all that stuff? You know, I'm fairly nonplussed. I mean, I think they're busy doing, you know,

you know how it is? They're sort of doing their day-to-day life, just like any other, you know, older parents these days, busy with the small things like fixing the washing machine and so on. You wouldn't just say, listen, let me,

if he's broke, I'll buy you one. Oh, of course. Yeah, no, no, I've done all of that. Yeah, of course. No, I bought them houses and all the rest of it. Yeah, definitely. There's clearly, not clearly, but there seems to me to be a through line from the Muslim youth helpline to the latest product which you've developed with AI, which is PIE.

And Pye, for people who don't use it, is a large language model. But unlike chatGBT, it's less about spewing information. and it's more about the AI developing an empathetic human relationship. In fact, it feels a little bit like a helpline. I mean, is there a sort of connection between that very first thing that you did at 20 and what you're doing now, that you're applying a new technology to a similar issue that you care deeply about? Yes, in a way there is.

I mean, I think that, you know, in the future, everybody is going to have a personal intelligence, an AI that essentially functions as a chief of staff. It will help to schedule and coordinate and plan and book and buy. It will teach you. It will provide you with advice, but also support and be a companion. And so we've started with an empathetic and kind and supportive AI. Certainly that's inspired by some of the stuff I did earlier in my career. But really what I wanted to do was demonstrate that.

AI can actually be good. I mean, it really can be controlled. The behaviors can be engineered with very nuanced and precise tones. So if you use Pi, you can really see that it's quite boundyed. It's super respectful. It's very empathetic. It's very hard to push it to be offensive or abusive in any way. It's virtually impossible. I mean, we don't actually suffer any of the jailbreak, the red teaming attempts that many of the other LLM's large language models, face. And so that was really my quest is to sort of demonstrate that we can actually reproduce, you know, a very controlled set of behaviours. And so I do worry sometimes that maybe like many of us, you are very, very powerful at seeing the challenges and the problems.

And then you feel, because we live in an optimistic age and publishers and others expect it, that you have to produce a very clear 10-point conclusion how to sort it out. But sometimes it feels to me a little bit like, you know, what I remember in Afghanistan, which is people would provide an incredible analysis of everything that was going wrong in Afghanistan and then have a 10-point thing at the end saying,

notwithstanding the fact that, you know, Pakistan's do stabilizing and televised, or my own life. I write a whole book about politics, about how miserable it is and how grim it is. And then I feel forced to produce a sort of concluding thing with a positive story about what we can do to sort it out. Do you ever fear that you've sort of talked yourself into an optimistic solutions, but that actually if someone were to read it in a hundred years' time, the thing

that would be truest and most powerful are your analysis, the problems rather than your solutions? I think that's a fair point. I think if you read the book, I don't think you can accuse me of being an optimist or a pessimist, depending on which page you read. And I think that that's an honest look at both sides of the coin. This is the reality. And I open the book with this idea of pessimism aversion. We have to stop this false narrativeizing, good or bad, this catastrophizing or this techno, you know, optimistic utopia.

It's all going to be wonderful. And we have to actually engage with the very nitty, gritty, practical reality of how these huge incentives produce technologies which default to proliferation. And we have to look at those consequences. and then we have to decide collectively what that should mean for how we manage governments and the future of the nation-state. So it's too simplistic a frame to say, are we optimistic or pessimistic? It's much more important to rip apart the detail of my proposals around how I'm actually

suggesting we do containment, right? Yeah, well, just on that then, I'll hand back to Honestor. I mean, it'd be lovely to hear you talk a little bit more about the 10 practical policy suggestions you have about how to contain AI. But I think my fundamental problem with most of them is that you're assuming goodwill and cooperation and you're assuming that it doesn't proliferate pretty quickly to some bad actors who don't give a monkeys

and are going to push ahead for advantage. And page after page, when I find myself noting, I just think, whoa, that's pretty hopeful there. That sentence is saying, you know, if we can do this, if we can do that. I'm thinking all the time. Wait a second, am I sure that North Korea or China or Dubai is going to play ball with this idea? Well, I mean, I'm not sure that we're sure of anything in this world, right?

So I think that one has to make proposals which, you know, are both practical, but they clearly lean towards a hopeful outcome, right? I mean, we're trying very hard to lay out a roadmap for what containment looks like. And that is going to require cooperation. Because if your assessment is correct, which is that nation states are fundamentally zero sum, will always be competitive and will always be opposed to one another, then it really is all bets are off because we're about to create extremely powerful tools that are inevitable over the next century. I mean, it might not happen this decade or two decades, but over a century scale, it really does turbocharge many, many, many actors big and small. And we have to contend with that. I mean, I think I've read the book in much the same way, I think, as Roy did, feeling very, very alarmed and pessimistic as I sort of traveled with you through a lot of the kind of both history, but more so what's happening now. And then along comes this 10 point plan, which I did, I agree with Roy, I think is so reliant on the people that you would need to implement such a plan in every country in the world because they're going to be worried about.

this threat to the nation state. I think that's something that a lot of people are thinking about. And then you had this lovely line which I, you know, faced with the abyss, geopolitics can change fast. In the teeth of World War II, peace must have felt like a dream. Now, that is true. It must have felt like that. But at the same time, there was back then the sense of a global structure which emerged from it. And there was time. What I feel with what you're describing the first three quarters of the book is that we're slightly running out of time. Now, am I just being too pessimistic here? Am I not falling enough into the pessimism aversion trap? Look, I think you're right that time is against us. And I think that the pace of change

is phenomenal. And everybody can now see that, particularly in the last 18 months. But the same trajectory is also true on synthetic biology. And so, which I think is a decade. or so behind where large language models are in terms of being able to generate arbitrary compounds, which are of enormous value, just in natural language instruction. So yes, in that sense, it is a warning and it is a provocation. The goal is to basically say, if I'm wrong, then propose an alternative path. What does the non-competitive or let's say the cooperative path to resolution look like? And I think that the comparison to, you know, the Second World War is appropriate because it was, you know, sort of inconceivable that there would be peace. And yet peace was found and out of that over a decade during the 50s, you know, many of the world's most significant institutions were born, which have driven untold prosperity and progress.

The problem is we're mere mortals, right? So if we resign ourselves to inaction and fear and heartbreak, then we won't get anything done. So you have to be hopeful and positive. Yeah, but you also have to be realistic about the world in which we're operating, particularly at the political level. And, you know, you make the point that for something like climate change, we kind of have the metrics. We know what bad looks like and we know then we can work out the things to avoid bad. with this, because it has happened so quickly, we don't really have the same metrics.

I don't know what a bad AI outcome or a good AI outcome looks like. And I don't know who's meant to be explaining that to me. Yeah. I think that the world is catching up rapidly. I mean, people have gone from basically, the average person has gone from basically zero understanding to increasingly getting a grasp of what these things can do. So putting them out there in the world is clearly a useful first. step because people get the measure of them. And I think in two or three years, we'll start to see

the kind of social consequences and what they actually feel and what they're like for our politics and culture. And I think that will give people the resources and the kind of energy to start thinking about other strategies for containment. How do I make sure that an AI system is always accountable to me, is on my side, is, you know, working in the public interest, those are the questions that people will start to wrestle with in the next three or four years. Who are your sort of peers and colleagues at the sort of level at which you're operating? I mean, we get very little glimpse of them, and you're interacting with them daily. You know all these famous people, Bill Gates, Mark Zuckerberg.

I mean, it's extraordinary, for example, the endorsements in your book. I mean, you're being praised by Al Gore and Yuval No Harari and all these people. So what is this strange culture that's emerged? You have a group of people who are unbelievable. wealthy and who are also in a way that probably wasn't true of the kind of robber barons the late 19th century, they're also quite intellectual. I mean, I guess, you know, Henry Ford and Rockefeller were clever people, but they were quite practical clever people. This is now a group of sort of billionaire nerds. They're people who, you know, incredibly good at maths or did unbelievably well in exams.

I mean, if Alistra and I would sort of hang out with them, you're talking to us from Palo Alto, where presumably you're working 14 hours' day, but when you're not working 14 hours day, you're going meeting these people conferences. What is this culture? What do they like? And what does that mean for the development of civilization to have a civilization run by billionaire nerds? I'm not sure that the civilization is run by billionaire nerds, but I think that it is a culture of curiosity and a willingness to speculate on the far out future. That's what politics is lacking today. No one in your world is proposing a plausible path to a positive vision over the next. 10 to 20 years. You know, politics feels broken to people. It feels broken to the people who are in it.

And where are the great visions of the future? You know, what are the stories that we're telling ourselves about how we're going to create peace and prosperity and more wealth? And I think the challenge is that we technologists are telling ourselves the story that we have the chance to produce, you know, radical abundance. And I truly do believe that. That's my primary motivation for working on building intelligence. And I think we have every chance of doing it. And like I said, I really do think this is going to be the most productive few decades in history.

So I think that's the challenge for conventional politics is to try to respond in a way that basically takes control and takes all of the kind of strengths and benefits of the technological ecosystem to try and harness them in the public interest. I mean, that's really the goal of my book. it's a love letter to the nation state. It's a warning. It's saying now is the time to act. Now is the time to refresh and regenerate our politics to make sure that it remains in control and relevant and capable of intervention given what's coming technically. Your book is obviously written for more than a British audience, but can I imagine that it might have been your homeland that you

had in mind when you said, when a government has devolved to the point of simply lurching from crisis to crisis, it has little breathing room for tackling tectonic forces requiring deep domain expertise and careful judgment. I mean, it's true, right? I mean, if that's an example of an area where I'm being optimistic because I'm saying that we need cooperation, then I'm sorry. What we need is nuance and patience and respect and forgiveness for one another and, you know, enough of the kind of bashing of each other and so on. I mean, without that, we're not going to move forward. And yet, obviously, particularly, I think in the UK, we're mired in this pretty awful,

depressing situation where I think politicians have lost confidence in themselves. And the political process has been so damaged by the last decade that people don't have a clear sense that this is the sort of infrastructure that can help navigate the next 20 years. And paradoxically, though, technology has been one of the drivers of polarization, populism, post-truth, whether through social media or through the massive changes in our economic structure. And so it's a very, very odd world in which to some extent, some of your billionaire nerd friends have contributed to the forces that have wrecked our politics as well as offering the great future visions for how to fix it. Yeah, this is true. I mean, this is sort of what happens when we sort of unleash technology. in a sort of unbridled way. I mean, you know, I think one of the big failures of the story of the internet over the last 20 years is that technology companies sort of claimed that platforms were really just neutral,

reproducers of information, that they sort of hid the fact that re-ranking and ordering was in fact moderation, right? And if the signal for driving that re-ranking was in fact singularly engagement, then things that are, you know, less true or maybe completely untrue or things that are outrage-inducing would clearly, you know, find their way to the top of the ranking quicker than others. And that in itself would trigger our dopamine releases. And I think everybody is sort of more familiar with that story now. And that's the sort of unintended impact of technology platforms that go out. there without thought. And I think that's one of the things that is slightly changing with this new wave of AI. I do think that we're learning those lessons and more proactive and more conscious

and more deliberate about the way that we design these systems. I think you can see that in PIE, but I think you can also see it with the responsible AI principles of the other AI companies starting to think proactively and adopting the precautionary principle. Now, you've talked about EU Act and other interventions by some of our political leaders that you clearly take seriously and have some respect for. But is there a sense that a lot of the people in your world really do feel now that they're kind of masters of universe that they can operate without worrying too much about the political process? I think what a lot of people forget is that the nation state is still the backstop of power in our world. Managing the military, managing

taxation and essentially operating the judiciary is always going to be a thousand times more powerful than corporations. So whilst government institutions move slower, they do fundamentally wield way, way, way more power, right? And I think that's a good thing. And so for those who are sort of techno boosters and maybe do believe that they're kings of the universe, and they are kings, rather than Queens at this point, sadly, I think that they're going to get a tough shock when political systems kick into action because if the trajectories continue as they are

over five to ten years' time, then people aren't going to sort of accept these kinds of extreme concentrations of power or the risks that might unfold from this kind of technology. So we're coming towards the end because you've been very patient, but let me hit you then, sort of really my final area,

which is, what is the consequence of AI on employment when we're really thinking about the politics the next five to 10 years? What's AI going to mean for jobs and what's that going to mean for our societies? I think that the way to understand this is that for a period of time, it is going to make everybody much more efficient and more productive. And the question is how long that period of time is. because everybody who wants to invent or create or learn is now going to have a personal AI in their pocket, which is going to provide them with the most patient, the most personalized, and the most kind of useful knowledge synthesized, perfectly given their context to that question in their style. And that's going to make us all way, way smarter and way more productive.

I think the challenge is going to be how quickly those. those sorts of tools end up getting integrated into the workplace and ultimately replace a lot of what I would call intellectual cognitive manual labor or cognitive manual labor, right? So sending simplistic emails, processing orders, managing call center inquiries, the kind of basic back office administration that powers most large bureaucracies. You know, I think over the next five years, maybe seven years, is likely to be automated. And so there's going to be a very disruptive transitionary period where we need people to retrain and reskill and we'll have to fund that and support that. And that basically means taxation. And we could be looking at 20% unemployment

over the next five to seven years. Not five to seven years. No, I don't think that's, I don't think that's possible. I think that over a 20 year period, you could see double digit structural disemployment. So, you know, people who want to get a job and they can't compete in the labor market, maybe. I think that that is a serious scenario whose probability has increased in a non-trivial way. And that's why I say taxation and redistribution is going to become even more important. I mean, at the moment, you know, in the US, labor is taxed on average 25%, whereas software and equipment is only taxed at around 5%. And I think we're going to have to make this shift towards taxing capital significantly. There are large stocks.

of wealth in land and in property and in, you know, stocks and shares, which, you know, we're going to have to call on to fund this transitionary period over the next 20 years. To finish for me, last question. What do you think irritates you about these sort of interviews? What do you think people like Alastair and I often miss? If you had a sort of final couple of minutes just to reflect on the end of the interview, think, I wish we'd got onto this or I'm so frustrated that people ask these kind of questions and don't ask those. What would you say in the last couple of minutes? I would say I'm the expert in technology and in my area, AI.

You guys are the 30-year veterans and experts of politics. I think it's on you to help answer the question that you put to me about how we evolve politics and governance so that it's fit for the modern technological age. So that's the discussion that I think it would be good to have because I'd be looking to both of you for answers to the kinds of 10-point plans that I laid out and those kinds of questions. And I think that's the goal of my book is the provocation has been laid out. I've done my best to lay out the ways in which I think containment might be possible, the practical things we can do. If they seem insufficient, then please take up the mantle and help us all to figure out how we evolve. governance and politics for the 21st century.

The problem is with Stapher, Roy and I basically share your analysis of the state of our politics at the moment and don't quite feel we are empowered or have the strength and energy just the two of us to kind of, you know, get it back in the right place. But I thoroughly enjoyed the book. I mean, it enjoys the wrong word, actually. It did alarm me, woke me up. I think Roy and I've taught a lot about artificial intelligence, but I think what your book did was really.

kind of bring it up close to me, how this does need far greater contribution from the political space, because without it, the same thing is going to happen as happened with social media is that by the time all the implications are fully understood, the politicians are already too late. And I worry that that's what's going to happen with this as well. So you're absolutely right that the politicians need to wake up, and we can definitely be part of that dialogue. Glad to hear it. Thank you. Thank you. Thank you so much. Thanks for your time. Thanks, guys. So, Alison, what's you make of that?

I sense the tension there between his book and himself. Or maybe I just, because I don't really get this world, it's not my world at all. By reading the book, I thought, God, here's a guy who's coming along and he's really, really trying to warn us that this thing's bad. But then he has, as you say, is 10 points at the end, says it's going to be fine, provided we do this. But I don't know. I don't feel vat assured. I think it's very tough. I mean, I think he's, as you would have picked out, unbelievably bright.

And I think actually in many ways, and we didn't really get onto his politics. But I think if we had, you would have found a lot in common. I mean, he's somebody who very much came from the left, probably a bit to the left of you, I guess, when he's working for Ken Livingston. And he's quite unusual in the Silicon Valley setup in really emphasizing social justice and encouraging people to push up taxes. and he thinks that the way to deal with the challenge from mass unemployment that might come from AI is not just income taxes but wealth taxes. And I think he's happy to put his money where his mouth is. I think he does believe that there is a huge obligation for the very wealthy to make sure that this technology isn't having an impact on the poor.

Yeah, and the other thing he does, which came through both in the interview and in the book, I find a lot of these kind of tech guys are pretty anarcho-capitalist, and they have a pretty rich contempt for politics and politicians. Whereas I think part of his thing is that politics has to be part of the solution. It can't be done just by the tech people, which I think is right, but also I think it's good coming from him. But he's also, I mean, one of the things that I think you said to me just immediately afterwards when we were chatting is that you thought that he was a bit sort of, defensive. And I think that is partly because he's so central to this whole thing, to this company, to the AI movement, that you sense that he has to be quite careful what he says. He can't quite kick back and relax in the way that some of our guests can because every word that he

says about the threats to AI is affecting international legal policy, share prices, companies. I mean, in a way, it's a sort of tribute to how much he's in that game that he has to be quite careful measured in the way that he deals with this stuff. It's very, very sensitive. And we're right at the cusp of this. Yeah. One of the other points he makes in the book is that of all the themes and trends and the technology and the change that he's writing about in the book, which is vast, he says in terms of the jobs, the people working in all of that, it's about 150,000 people around the world. It's tiny. And yet, as you say, these people have, you know, real impact upon stuff that we do understand and stuff that we don't understand. And they're

quite remarkable people, too. I mean, whatever one thinks of them, and I like Mustafa, I think he's an unusually intelligent, thoughtful person, but they are all in their own ways. Very remarkable. I mean, you know, I obviously am pretty horrified by what I see at a distance of some of them, but they are all incredibly bright. They all read a lot. They think a lot. They can be very narrow. I mean, if you think about Elon Musk or Mark Zuckerberg,

there's a sense that they're very, very much engineers with a strong kind of technical bent. But others of this group, which includes Bill Gates, are making real differences in the world. And Bill Gates has driven huge changes on malaria, on all forms of vaccination, global health. It's a very interesting period that the, richest people in the world, so much more interested in, in kind of ideas and thinking than maybe

billionaires were 40 years ago, 50 years ago. Well, they just wanted to make money. Well, I guess it's that to make it to the top in these tech companies, people need to be very, very adept at maths, computing, analysis, philosophy, as well as working very hard. I mean, that's the other thing about them. I mean, Mustafa, we had an interview with him there for an hour, but, you know, what I know of him is that he will have then gone on to work a 14-hour day. So he's part-time here.

He's a bit like you, Alastra, is what I'm trying to say. I think, in fact, actually, strangely, he is a bit like you. Talking of the other guys, apparently the Musk-Zuckabberg fight is off. Oh, no. He was so excited about that, weren't you? I was very, very excited about that. Yeah, no, probably. I think, strangely, you'd find them, you'd find more in common with them than you'd think.

They're very, very driven. many of them have had challenges around mental health but are therefore unbelievably productive when they're up Yeah, he sort of touched on that, didn't he? For himself, but I could sense he didn't really want to go there. No, he hasn't. I mean, I think that was probably the most open he's been about his personal life that I've ever heard.

I mean, he was much more open about his parents and his father and his faith and his mental health than I've heard before. And I think, is that the first mention of Kat Stevens on our podcast? I think it might be. Very good. Well, I think as we come to around, I mean, I think he is a very, very remarkable British person.

I think it's an extraordinary story, working class, father, Syrian immigrant taxi driver, who's become one of the most powerful central voices in this revolution that's changing the world. So thank you for doing it with me. No, I enjoyed it, Roy. Thank you. Thank you very much. All right. Bye-bye.