The Rest Is Politics: Leading - 116. Is AI the answer to human suffering? (James Manyika)

Episode Date: January 13, 2025

How can AI help solve global problems such as poverty, hunger, and disease? Is there a hidden dark side to this new technology? What is in store for the future of artificial intelligence? Rory and Alastair are joined by Senior Vice President of Google, James Manyika, to answer all these questions and more. TRIP Plus: Become a member of The Rest Is Politics Plus to support the podcast, receive our exclusive newsletter, enjoy ad-free listening to both TRIP and Leading, benefit from discount book prices on titles mentioned on the pod, join our Discord chatroom, and receive early access to live show tickets and Question Time episodes. Just head to therestispolitics.com to sign up, or start a free trial today on Apple Podcasts: apple.co/therestispolitics. Instagram: @restispolitics Twitter: @RestIsPolitics Email: [e-mail] Video Editor: Kieron Leslie Assistant Producer: India Dunkley + Alice Horrell Social Producer: Jess Kidson Producer: Nicole Maslen Senior Producer: Dom Johnson Head of Content: Tom Whiter Exec Producers: Tony Pastor + Jack Davenport Learn more about your ad choices. Visit podcastchoices.com/adchoices

Thanks for listening to The Restis Politics. Sign up to the Restis Politics Plus. To enjoy ad-free listening, receive a weekly newsletter, join our members chat room and gain early access to live show tickets. Just go to therestispolities.com. That's the restispolitics.com. Welcome to the Restis Politics leading with me, Alecester Campbell. And with me, Rory Stewart.

And very happy to have with us today, a friend of mine, James Melika. And James and I, in fact, have been to the South Pole together. There we are, Alistair. That's a big boast, isn't it? That's good. Yeah, pretty chilly over there. But the reason he's there is not as a polar explorer. James is with us as one of the world's leading analysts and experts on AI,

and actually now leading a team that's been working on something called quantum computing. James is originally a Zimbabwean, so he grew up in what was effectively apartheid Rhodesia as a young black man. He then went to university in Zimbabwe after independence, got a road scholarship to Oxford. He studied robotics and he was right there at a pretty early stage of the AI development. He's then gone on to do a series of really interesting things which bring him, I guess, in the intersection between AI computing public policy business. So he's worked for McKinsey. He's now vice president of Google. in charge of most of this stuff within Google, but he's also served on very important positions

in the United States government, with the United Nations, with Harvard University, Oxford University and more universities than I can mention, with the MacArthur Foundation. So a loss of complex stuff, but at the heart of the philanthropy, the business, the work in government is the fact that he begins as a technologist. as somebody who's really interested in AI and robotics. So welcome, James. Thank you for having me, Rory, and Alistair, I'm delighted to be here. I've been a huge fan of this program for a very long time. Well, thank you and thank you for listening to it. Alice, do you want to fire away? Yeah, I promise you, James, we're going to cover all of that

big tech stuff, which Rory understands a lot better than I do. But I'm fascinated by where you come from. My experience of Zimbabwe is basically a few meetings with. Robert Mugabe. They weren't the happiest of meetings in my life. So what was it like growing up in Zimbabwe? Just give me a feel for what that childhood was like. Well, I mostly grew up in Harare. It used to be called Salisbury at the time. And as you probably both know well, Rhodesia pretty much ran like an apartheid country. So I lived in a township, now called Mbare. These were segregated township. So I remember in the early 70s, I went to a segregated school, Chitzerra Primary School. So I pretty much grew up in Bahra. And I remember at the time growing up, on the one hand,

it was all I knew. So it felt normal at some level, because that's all I knew. But at the same time, I remember, you know, several things that are still stuck in my mind, the occasional police raid. It used to happen in that in those days, occasionally the police would come in and raid houses to make sure the people who were staying there were supposed to be there because you're supposed to live in townships, but also some people were supposed to stay in the rural areas. So there'd be these occasional raids at night and police patrols. It was also fascinating because I remember next door to us as a Shabin. It had all the kind of political turmoil, but also the kind of the societal. excitement and things that went on in townships in those days, where you'd see, you know, policemen would come to the Shabin next door, kind of hipsters and others in the community.

Explain a bit for this as what a Shabin is. Oh, so Shabin is typically a house where alcohol is sold, often illegally, and often there's an illegal brew that's made called Kukyana, which is a very, very potent illegal brew. and occasionally those would also get raided, occasionally by the police. And is there sometimes a television where people can watch movies and stuff? So, well, often people would be playing music. There were no televisions at the time. The television was a very rare commodity in the township in the 70s.

So there'd be a lot of music, a lot of people coming to hang out. In fact, there's a famous song called Kokiane, which was actually done by Humma Sakela. And Biket was made very famous, actually, by Louis Armstrong, because Louis Armstrong came to Rhodesia in the 70s and heard this song, and he decided to make a song about it. So the Chabines were very fascinating. At the same time, I remember one thing that has left me pretty scarred was, and I remember this struck me as very odd at the time,

seeing somebody actually necklaced. And that's when somebody gets... So that's when they put a tie around somebody's neck because they were supposed to, they were considered a collaborator or a sellout. Because in the township, remember, but we also used to have obviously very politically active people and communities in the township would be organizing meetings and rallies.

And often, Chabines were actually the place where some of those political discussions took place. So often Chabines were raided not just because of the alcohol, but also because there'd often be political meetings going on in those places. So again, to set the context for listeners, Rhodesia at this time in the 1970s, and this is you, through childhood and your early teens, is a place which is both a white-ruled. apartheid state, but it's also a place on the edge of a civil war, because there is an armed insurrection taking place, which people like Robert Mugabe and Emerson and Gagwa, who's now the current leader, were very, very involved in, an enormous amount of fighting, taking place in

different places. And I guess different governments getting involved, a little bit of proxy warfare getting involved, South Africa potentially supporting one side, others supporting others. Yeah, a lot of the actual fighting that was being done by, as part of the liberation movement was actually not so much in the cities, in the townships, primarily in the rural areas in the outlying districts. So often what would happen is young people would often disappear from the townships to go off to fight the war, and they would actually go often across to Zambia or to Mozambique to join the liberation struggle. And then most of the fighting was actually happening on in the farming communities in the eastern part of the country. A lot of wars going on in

the township is mostly kind of turmoil and uprisings and those kinds of things, not so much actual fighting itself. You mentioned necklacing. So this is a vision of somebody who gets a tire around the neck and the tire is then set a light. You saw this yourself? I saw this once in the township on my street. And you were how old when you saw this? I must have been about seven, maybe, eight, perhaps. And a terrifying thing for a seven, eight-year-old to see. It was terrifying. That again, along with the occasional police raids. But at the same time, you know, I was going to school. As I said, that's all I knew. So I went to local township school. And, and, you know, And I remember that very well. I mean, it was a very unusual setting because at the same time,

there were all these few glimmers of hope in the sense. So for example, in my township in Barrow, we used to have this extraordinary swimming pool called the George Hartley Swimming Pool, which somebody had decided to, you know, build this extraordinary swimming pool in the township. So I remember I used to spend most of my weekends swimming at the pool, the George Hartley Swimming Pool. and we'd also go to see films at Stoddard Hall, where we used to see all these incredibly extraordinary American films. So it was a very interesting childhood. I'd also say it was also interesting in the sense that I used to have glimmers of,

quote, the other side. And what I mean by the other side, my father had been extraordinarily lucky. He'd actually got one of the early Fulbright Fellowships, and he'd actually studied in America before I was born. So his experiences often had given me a bit of, as I was growing up, a glimpse of what was possible on the other side. I would also get similar glimpses. I remember there's a man, Mr. Jingel. I remember he just named very well.

He's a friend of my father. He used to work at Longman's, the publishers. I remember he did something that gave me another glimpse. He took me once to the Queen Victoria Memorial Library. So the Queen Victoria Memorial Library was actually only for white kids. And I remember there was a huge incident because he was being asked why you bring this black kid to this library. By the way, they're not allowed to take any books, who will let them in just to look around. So I would always have these occasional glimpses of what was actually possible on the other side.

When you were growing up, what did you think your life would be? What did you think you would be when you were an adult? And also, the friends that you had at primary school, what are they? you're doing now? Because you've had this incredible life. But do you ever have any sense that you'd live the life you have? Oh, not at all. Not at all, Alistair, because I remember 80s, when I was in primary school, there are all these kids around me who are extraordinarily brilliant and very bright. I often, often people say to, well, you must have been the smartest, brightest kid in your school and your country. I said, no, no, no, no, no, no. There are all these kids around

me who are extraordinarily bright. I just think I probably got lucky in the sense that I got lucky to get access to places and people who inspired me. I was certainly not the smartest kid at all. In fact, my friends from that time often remind me of this about the fact that they used to come top of the class and all of that. What are those kids doing now? So one of them, probably my oldest friend who I'm still in touch with, is actually a filmmaker.

Her name is Oliswacitole. She's actually extraordinary filmmaker. She's actually very, very bright. She always used to come top of the class. In fact, she's done very well. She's won a BAFTA has won a Peabody Award for her documentary filmmaking. But most of the people I grew up with and went to school with, many of them are still in Zimbabwe, many of them struggling because the country hasn't, after an incredible period of promise,

hasn't always worked out very well for most people. It's heartbreaking, to be honest for me, to think about these extraordinarily bright kids who I grew up with, who didn't often get the chances and the opportunities that I got. But to answer you a question, Alistair, but what did I think I was going to be? Again, I had this kind of dual view of myself. As I said, my father had come to America, and when he came back by the time I was born, because he was there before, he was in America before I was born, he had visited Cape Canaveral, which I think eventually became called the Kennedy Space Center in Florida.

And he'd seen all these rockets going up. when I grew up seeing his slides and photographs of rockets going up in space, I remember thinking, I want to be an astronaut. But at the same time, my circumstances gave me no reason to believe that I could ever become that. But I still wanted to be an astronaut anyway, because my father had this one photograph of himself wearing a space suit when he'd visited Cape Canaveral. So that was kind of my secret dream, although, of course, I thought that would never happen. I mentioned Mugabe there. What's his reputation now? in Zimbabwe? Well, I think it's changed a lot. I mean, I think if you had, you know, when I was

growing up in the township, he and many others had this reputation of people were fighting for freedom as part of the liberation struggle. In fact, my uncle, my dad's brother, was actually part of the liberation struggle as she fought alongside. Mogabe's name was Robson, Manika, and fought alongside. So in the 70s, these were seen as kind of brave liberators fighting the war. And then in the period after independence in 1980, so roughly from, call it 1980 to about 2000, he was largely seen very favorably as having led the country and transitioned the country from Rhodesia to Zimbabwe and having done very well. Keep in mind that at the time, many of us even living in Zimbabwe didn't know very much about some of the atrocities that had happened in the southern part of the country, even though we were living in Zimbabwe. Again, for listeners, so there's this famous Gurdukundi massacre, and it turned out there was sort of profound, horrifying attacks against political opponents and other ethnic groups within the country, but not something that was talked about much in Harari.

No, in fact, when I was a university as an undergraduate, this is when a lot of that was going on. The conversation was often in the local newspapers was that, oh, there was some insurgent, dissident activity in this part of the country that the government was trying to quell down. was the way it was talked about. And it only became, I think, apparent to most people much, much later as the extent of what had been going on. So to ask you a question, I think up until probably the early 2000s, I think MacGabwe was seen very favorably as having led the country. And in fact, the country is kind of seen as a success up until that point.

Because if you had looked in roughly about 2000 of the late 90s, Zimbabwe had the highest literacy rates across the continent. It was exporting things even to the east. EU, it was incredibly self-sufficient, the education system was remarkable, and all of that. So I think he got a lot of that credit. I think the view about him mostly started to change in the 2000s, when there started to be an opposition movement that was starting to challenge the status quo. And, Alistair, by the time you're coming into government, late 90s, early 2000s, people

are beginning to see Mugabe increasingly as a problem, aren't they? I mean, presumably on your desk and Downing Street were real anxieties about the direction in which Zimbabwe was going. Absolutely. And it's why I do remember so vividly the meetings with him. I sense this visceral hatred of Britain because of our kind of colonial legacy, as it were. You had a sense of somebody who was actually very, very wealthy in his personal demeanour. And somebody who was totally unapologetic about the direction he was taking the country.

country. And even as we were all seeing, the data that said economic progress was stalling and that people were being very, very badly treated. Yeah. And I think certainly when I was in university doing my undergraduate degree in the early 90s, there was a sense that the country was doing well. I remember actually one time coming on holiday to London. And I remember the Zimbabwe dollar was actually stronger than the British pound. Oh my God. Exactly. in the 90s. Remind listeners what eventually happened with the inflation. What was the exchange rate at the worst moment? Oh, I can't remember. I mean, we remember the worst moment is probably about 2008, when the country had a hundred trillion dollar bill. A hundred trillion dollar bill. I mean,

I actually have a copy of, a couple copies of hundred trillion dollar bill. Yeah, that's probably the worst moment. A hundred thousand billion dollar bill. He always seemed to me, he always found people that he could successfully blame, and there were enough people in the Zimbabwe, you believed him? Or have I got that wrong? No, you've got it right. I mean, I think the sense was always that it wasn't that simple in the following sense. I think there was a sense that, in fact, there had been a terrible colonial legacy. There was a sense, in fact, even in the early 2000s, that a lot of things had gone unaddressed, such as the inequity in land ownership and so forth. Because even until

the early 2000s, I think you had something like still over 80% of the land were still white-owned. And this is kind of 20 years on after independence. So there's a sense that there was a legitimate issue that needed to be addressed. But I think the sense that I had was that at the time, the way he went about addressing those issues was absolutely the wrong way to do it. At a time when also there was now a growing opposition movement that was trying to challenge the domineering effect that his party had on the policies of the country. James, I'm going to fast forward you now into the world of tech. So you, having had this very, very interesting childhood, which we could talk about a great deal more. And, you know,

being, I think our friend Reid said, I think that you were bused to school initially, when schools were integrated and you were one of six kids going with armed guards to get into a white school. Well, I bet the first time I went to Prince Edwards, after Zimbabwe, Dija became Zimbabwe. I was one of the first, among the first black kids to go to an extraordinary school, Prince Edward School. Which had been a white school. Which had been a white only school. I mean, this is an extraordinary school that was over 100 years old. Incredible resources. We had an astronomy lab. We had a telescope, an observatory. But you arrived as very few black children. There were very few of us. And so we had to move into

these boarding houses. Because these schools were set up almost like old English public schools. Right. So I was in Slew House, which is interesting. in itself. Those houses like Rhodes House and Jemison House. And I remember there was a lot of resistance to black kids showing up in these schools. So it was a very, very difficult, difficult time. But it's part of my most memorable times growing up and making, you know, initially getting involved in all kinds of fights and racist incidences and all kinds of things at that

school. But I made some of the best friends that I still have from those days. Coming out of school. Absolutely. So then we fast forward. And you, having done an undergraduate degree, you turn up in Oxford, which is the beginning of the British connection. We're talking here in London. And you're coming in at a very, very interesting moment into the heart of robotics and thoughts about AI and machine learning.

So if we can step back from James's autobiography to the conceptual, what is the story of AI roughly since the Second World War? I think you sometimes said there were two paths and what were you looking at when you turned up at Oxford? Before we go to the Oxford part, there's actually a personal connection from Zimbabwe to Oxford in AI and robotics. The very first thing I ever published in my whole life was actually a paper on, it was titled something like training and modeling neural networks. And I did this as an undergraduate in Zimbabwe, which people sometimes find surprising. This is because there was a postdoc who happened to be there from Canada, who is familiar with some of the work that Jeff Hinton, who just won the Nobel Prize, recently had been doing on neural networks. So I sort of have that kind of one or two people removed connection to Jeff Hinton. In fact, I've actually told him this and he finds it quite

interesting. But to your question about the history of AI, the term itself came into use in 1956 in full force. There's a famous Dartmouth conference where a handful of very notable figures, John McCarthy, Marvin Minsky, Claude Channon and others got together for what they called was going to be their summer project. Summer project was actually to build AI systems that could simulate what humans do. But there's a long history of this field

during, in fact it even goes further than that. And in the 50s, there were always kind of two strands to the thinking. One strand was, call it the mind approach, if I could call it that, which is let's try to build machines that can reason the way

our minds reason. So think about the rules logic, the rules of knowledge, how do we build systems that can replicate, how we think our minds work. That was always one strand. There was another strand, which was at the time called the connectionist approach, which is called the brain strand, which said, no, no, no, no, let's try to build systems that mimic the way the brain is structured with connections and neurons and all of these things and that try to learn. So there's always these two strands in the 50s. And to understand the first strand for a moment, maybe a way of illustrating it might be thinking about how you

could program a computer to play a game like chess. And would I be right in saying that in the first strand, you're trying to train it to make endless logical choices, whereas in the other strand you're doing something else, or is that a bad example? Sorry. That's actually pretty good. I mean, you're trying to understand the logic and rules that govern thought and choices and steps, if this happens and that happens and so forth. And you're also trying to codify knowledge. How is it that we know what we know? What are the rules of knowing things?

How is knowledge structured? And that first approach, by the way, actually initially won the day. So in fact, the period from roughly the late 50s, even into the 60s, that was the dominant threat because it sounded right. It sounded logical.

It sounded like it had the weight of science and history and everything from the Lepen Placian dream from the Enlightenment. So it's kind of had the weight of history to go with it. And what were the problems of trying to do something that tries to get the full sum of human knowledge and make everything a logical choice? What's the limitations of that? That approach actually had some initial success. Because remember, this is the time in the late 60s where even films like 2001 a Space Odyssey came out of because it looked like that approach was actually making progress. It only started to hit the limitations in the early 70s.

When progress started to slow down, what seemed like the initial successes of codifying things may have worked for things like chess playing algorithms and things, but not much else. And even in chess, it wasn't quite yet beating the Grandmasters. No, it wasn't because it didn't quite have some of these, some of what felt like intuitive moves that the Grandmasters had. And so there seemed to be limitations. In fact, there's a famous report, which was put together by Sir James Lighthill, the Lighthill report in 1973, which basically argued, I think I'll paraphrase it probably not quite right,

we basically said, why are we wasting our time? We should stop funding this work in AI. It hasn't amounted to anything. So this is the famous Lighthill report in 1970. What was interesting is that James Lighthill, he was the location professor at Cambridge, which is a professorship with an incredible history. I mean, Isaac Newton's location professor, Stephen Hawking and Charles Barbage. So it had all the weight of all of that, saying, let's stop wasting our time. James, we interviewed Bill Clinton recently, who since he left office, has become kind of obsessed with astrophysics and quantum computing, and we didn't really talk to him about it much, but I can remember when he was really kind of fixated upon the human genome project. And I can remember talking about this

and saying, this is kind of life-changing, this is game-changing for the world. But the stuff that you're talking about, when you talk about AI, what's the scale of gap between AI that was helping deliver the human genome project and the stuff that you're dealing with now? Yeah, between the end of the 90s when Clinton's stepping down and where we are now, what's happened? So what happened was in the late 80s, early 90s, after this so-called AI winter, when progress hadn't happened very much, researchers rediscovered the second. approach. The connectionist approach, the machine learning approach, as we now call it now.

And this is when people like Jeff Hinton and John Hopfields, we just won the Nobel Prize, started to write about this and said, well, no, let's try to do this quite differently. So when I was trying to do my PhD at Oxford, that was coming back into Vogue. And so a lot of the progress we're seeing now, Alistair, stems from a lot of that original work that Jeff Hinton and others did. And so what has happened in the last. call it 20-ish, 25 years, is really that three things have happened. First of all, the machine learning approaches kind of won the day. We now have these neural network deep learning algorithms that basically learn from examples. And I'm going to interrupt from it again,

Felicis. So these are people will know about them because they're things like chatGBT, large language models or examples of those. There are examples of those. But even before you get to those recent developments, you started to have this idea that, in fact, I remember working on this when I was doing my PhD, that maybe if you gave machine learning algorithms lots of examples and say, build from those examples, patterns and recognize other examples that look like that, how do you do that? So that started to make progress in the 90s. And then what then happened in the early 2000s is that the combination of lots of data becoming available from the internet and also the computing resources becoming more available, those three

things, the deep learning algorithms, the availability of data thanks to the internet, and computing architectures all suddenly came together for what we now see this as this extraordinary acceleration. But the real thing, maybe to get back to your original question, Alistair, that happened in the, around 2017. There's a famous paper written by Google researchers called Attention's All You Need, which introduced these large language models with the so-called transformer architecture, because up until then, the AI machine learning algorithms that you had were very specific. There were either things you could do, use to do image classification. That's a cat, that's a dog, learning from lots of examples, or to do natural language processing, or to do other

kinds of pattern recognition. They were very narrow and very specific. The transformer architecture in this famous paper in 2017 introduced what felt like a general machine learning algorithm. That's why what we now have with these chat bots like Gemini or Chad GPT, they seem able to do a wide range of things. That's why they can go from writing poetry to writing essays to translating language. So that was a big breakthrough to start to have these seemingly general models that can do a wide range of things. Okay, Rory, James, quick break and then we come back. and Hannah from Gollhangers, The Rest is Science.

This episode is brought to you by Cancer Research UK. We often think of beating cancer as treatment, but imagine stopping it before it begins. After years of work, Cancer Research UK scientists are launching a clinical trial of lung vax, the first vaccine designed to prevent lung cancer. It builds on TracerX, the world's largest cancer evolution study, which tracked lung cancer cells over many years to uncover the disease's earliest warning signs. Lung Vax is designed to train the immune system to spot these signs early on, destroying faulty cells before cancer develops.

So it's not treatment, but preventative with the potential to stop lung cancer before it starts. The first stage of the trial starts this year focusing on people at higher risk. It shows what long-term research makes possible. For more information about Cancer Research UK, their research breakthroughs and how you can support them, visit Cancer Researchukuk.org forward slash the rest is science. Hi everybody, it's Dominic Samark here from The Rest is History. Now, some of you may have heard me on your show, The Rest is Politics, when Rory was away and I was filling in and enjoying Alistair Campbell's tremendous banter.

And I'm back to tell you about our new series on The Rest is History, which is all about Britain, in the 1970s, a period with a lot of uncanny resemblances to our own. So right now we're living through a moment when oil shocks generated by war in the Middle East are rippling through the world economy, when Britain feels like it's sunk in a bit of a malaise, people are arguing about Europe, the government has got a few issues with the trade unions, and we have a kind of, I suppose you'd say governing elite, a kind of political class that is really struggling to come to terms with all of these issues and people are asking if Britain is governable at all.

So there are a lot of parallels between that Britain that I'm describing, which is our Britain, and the Britain of the mid-1970s. So in this series that's coming out on the rest is history, we'll be looking at these and other issues. We'll be talking about the rise of Margaret Thatcher, obviously a colossal figure in our political life even now, whether you love her or loathe her. And we'll be talking about the very first Brexit referendum of 1970s.

a subject that I'm sure Rory and Alistair will have strong opinions about. We'll be talking about the fall of the Labour Prime Minister Harold Wilson and we'll be talking about one of the grimmest moments in Britain's economic history, the moment in 1976 when we had to go cap in hand, as people said at the time, to the International Monetary Fund, the IMF, for a then record bailout. Now, if that sounds good to you, how could it not sound good to you? Of course it sounds good to you. We have a clip for you to listen to at the end of this episode.

And if you want to hear more, just search for The Rest is History wherever you get your podcasts. I read some of the stuff you've said publicly and written publicly about this. And there's one sentence I wanted to put back at you, which I found particularly interesting. You said, AI is putting a mirror in our face to say, okay, humanity, this is what you look like. How do you want to deal with it? So answer your own question. Well, I think, you know, one of the things with these machine learning algorithms is that they're largely learning from stuff on the internet. And on the internet, we have everything.

We have incredible poetry, literature, science. We also have some of the very, you know, ugly traits of humanity. So you've got all of it, all of what humanity has created, good, bad and ugly. And they're learning from all of that. Do they know the difference between good and bad? No. No, because remember what they're learning... You treat you all the same.

Yeah, they're learning the structure of language. They're learning the patterns of words. They're learning things like a word like close, lives in the same area as proximity, as near. They're creating these associations about words that seem similar, that seem to speak about the same kinds of things. So they're building associations about language and its structure. So they don't have any value judgment, good, bad, ugly.

They're simply learning the structure of language. Again, just for non-specialist listeners, people often assume that what is happening when they type into one of these AI models and ask it a question, is that it's simply going out, searching the internet, and reproducing and pasting in something. So if I was to say, draw a three-point comparison between James Mnika and Alastair Campbell,

and it goes off and produces an answer in two seconds, it somehow found that on the internet. But that's not what it's doing, is it? No, it's not what it's doing at all. I'd love to know what it would say. It would be fascinating. Keep in mind that the performance and behavior we see from this system is a result of scaling up a very simple process. And that simple process goes something like this. When you type in a word, it's trying to predict what comes next.

So these are next word predictors. In fact, it's the reason why some people have said, well, this is a very sophisticated version of autocomplete. They're simply predicting the next word. What then happens is that when you scale this process up, meaning you train on a lot of words, it starts to build very complex concepts. So when you type in a prompt,

he's trying to figure out what's the, best response to that prompt. So it's not going out to search anything. It's simply trying to predict what comes next. Now, what has happened over time is that in addition to that basic prediction mechanism, as a way to try to fix and address some of the factuality issues, we've now added things where you can ground it on search. Go validate what you're about to produce with what's out there first before producing it out. It's the reason, Alistair, why you find that sometimes these algorithms will seemingly make up things, the so-called hallucinations. I'm interrupting. So here's the answer, Alistair, for you, okay?

Oh, God. Three point comparison, James Mnika and Alistair Campbell. Number one, professional persona. James Mnika, with his calm demeanor and analytical rigor, then it praises James. Then Alistair Campbell, by contrast, exemplifies the intensity of political communication, combining sharp strategic instincts with a combative approach to narrative control. while they operate in vastly different domains, both are masters of influence. Monika's influence lies in this ability to articulate the transformative potential of technology. Campbell's impact rooted in the political turbulence the late 20th century reflects his capacity

to manage media narratives in real time, leaving an indelible mark on political communication. Despite their different approaches, each operates in spheres where perception often outweighs objective facts. And it continues in this way. I'm not going to keep reading it. Wow, there's quite nice. So who, where did that come from? I've taken it from chat GBT. So this is, yeah, chat GBT 4. And you did it very, very quickly. So that comes in two seconds.

But explain to Alistair roughly how it does that. It's not lifting it from the internet, is it? No. No, it's building from associations and in what it's been trained on. So there's clearly something in what's on the internet about Alistair. There's clearly on the internet things about me. there's clearly on the internet things about ideas in concert, maybe even things that we've written and things that we've said.

So it's building associations based on those things. And that process, when scaled up dramatically, is able to then give these responses. It's the reason why you'll often hear people talk about the so-called scaling laws in AI. And the scaling laws simply speak to this idea that these processes, when you make the architectures, very large and very complex, and you train them on lots of things, and you train them for a very

long time. You start to have performance and behavior that looks intelligent and is amazing. Now, James, I'm going to sound really stupid here, but when you say train, I think of sport. I think of a coach telling an athlete how to perform. So when you see training, how are you training and what are you training? So you're giving it lots of examples. So let me, You know the expression peanut butter and jelly, which is a very American, peanut butter butter and jam, I guess we would say in England, but it's that expression. So imagine training an algorithm to predict that phrase, peanut butter and jelly. I show the algorithm those words, peanut butter and jelly. It studies them.

The next time I cover up the word jelly, and I say predict what comes after peanut butter and. So initially it guesses, it might say peanut butter and. I don't know, milk or something or cheese. And then it looks up the original phrase. I said, oh, I've got it wrong. Let me try again. So it keeps trying. And then after a while, after trying and failing, trying and failing, it eventually starts

to get it right. So that's the training process. Now, you could imagine that training process going on for a very long time with a huge amount of words. All of a sudden, it's now good at predicting that a word like close is similar to near, it's similar to proximity. He starts to be able to predict those things, so much so that if I then give it a sentence, Alistair, that says, Alistair was riding a bicycle very fast down the hill and he ran over a pothole. It can then start to be able to predict 12 people. When somebody's

riding a bicycle down the hill and they hit a pothole, chances are they fell or they hit something or they might have broken a hand, it starts to be able to predict what comes next. How does that help when I'm riding my bike. It doesn't help you very much, but the algorithm is learning to be able to predict what typically comes after somebody who's riding a bike, hits a pothole, there's a finite range of possibilities of what happens next. It's learning to predict those things. And here's something I guess that might help you, Alistair, which I'm going to throw at James, which is that once this thing is really beginning to operate well, it can begin to produce pretty accurate answers to quite sophisticated medical questions. I mean, you can imagine

it maybe not quite yet, maybe now it needs to sit alongside a doctor, but already it'll be able to provide more accurate answers than many doctors on many medical questions. Oh, absolutely. In fact, one of our models, Med Gemini, which is our Gemini model fine-tuned on a medical corpus, that's why we call it Med Gemini, is actually able to perform better than most doctors on predicting and doing diagnostics and diagnosis of things. In fact, unless you're in the top two or three percent of doctors, it'll do better than you. You can see why doctors might worry that we cease to have a need for doctors? No, because I think what happens, where we've seen so far, Alistair, is that in fact,

when doctors are assisted by these tools, they're actually doing better than doctors not assisted by these tools. In fact, one of the things that I'm very excited, and I think is what motivates many of us who work in this field are these extraordinary beneficial impacts. So the way I think about, why do we do all of this? Why does any of this matter? At least I think about it in a few different areas. One are things like how this benefits individuals.

Think about language translation. Think about any kind of assistive individual things. I mean, I don't know if you use Google Translate. It works based on AI. That's how it works. And in fact, now we've just gone from even just four years ago, we could translate about 30-something languages. Now we can do 246.

In fact, we're aiming to get to a thousand. So think about these extraordinary benefits for individuals. Then think also about the potential benefits for the economy, small businesses, large businesses, even transforming economy sectors, even productivity. And then think about the impacts on science. Some examples on the economy, because that'll be very relevant to a country like Britain that's got economic struggles. The economic potential, so it's been known for a long time that productivity, is a drive of economic growth and prosperity. It's the reason why economists worry about productivity.

When productivity is low, we don't have GDP growth. And in fact, if you look at much of what's driven productivity growth throughout human history, a lot of this has been technological innovation. And in fact, in the case of, you know, where do we think we're going to get technological innovation? It's mostly coming from AI. So we're going to think about transforming sectors like healthcare, industries and manufacturing. And then even small businesses, the fact that a small business can now use these systems to be able to understand and do better marketing, to be able to understand regulations, to be able to say, you know, I operate in this space, what are the regulations that are applicable to me? As opposed to going to read all hundreds and hundreds of

pages of regulatory documents, I can use an AI system to help me understand that. So the economic benefits all the way from small businesses to large businesses, ultimately to economies, especially including workers enabled by the power of this technology is what drives labor productivity and least economic prosperity. One more example, and then I hand back to Alistair again. Tell us a little bit about autonomous vehicles and what that might mean for our future. Oh, so I live in San Francisco. We've had travelers cars, Waymos, running around San Francisco for the last year and a half. And they're extraordinary. In fact, I personally have always had a strong interest in this. I remember when I first moved to California, one of the ideas that

I toyed with some of my friends was to use some of the AI and robotics work we've been doing to build autonomous vehicles. They used to be this competition called the DARPA driverless car contest, which used to happen where people used to compete to build cars to drive across the desert in America autonomously. So this is something that I was always very interested. But now, fast forward to where we are now, we now have fully autonomous cars driving around San Francisco and they're far safer than your average driver. They're not distracted. They're paying attention to all the rules.

So I think from a safety standpoint, these are quite remarkable. But we are going to have to have to think about what does that mean for our cities and agencies? So if you're solving for safety, you might say that's an extraordinarily wonderful thing. In fact, right now, you know, a week now, Waymo's in San Francisco and a few other cities, are doing something like over 175,000 rides a week. So people are getting quite comfortable with these cars for their safety. And in fact, a lot of the users of them tend to be often at night. Women use them a lot at night.

They're considered kind of safe and trustworthy. But we're going to have to think about how do we think about agency, how do we think about safety, how do we think about other modes of transportation? I think it's quite an exciting prospect. But the thing I also wanted to mention, by the way, in addition to some of these economic things, is a potential impact for AI to advance science. And so what we're seeing with AI advancing science

is happening not just in proteins or structural biology, but in other fields as well from fusion to material science. So the potential to benefit in advanced science is really quite exciting. Now, as you know, this podcast is mainly about politics. And I thought it was interesting that you sent us the digitalist, papers. And this is a set of essays written by, including yourself, but also our friend Audrey Tang

from Taiwan, who we had on the podcast, Reid Hoffman, Eric Schmidt, founder of Google, lots of kind of big names in your world. The thing that struck me is this goes back to your question, what do we want to do, how do we want to deal with how humanity is. Very different views about whether this is a threat, whether it is just a threat, whether it is just an opportunity, and where we get the balance right between that. I, for example, I read Lawrence Lessie, and I found his essay quite alarming. He seemed to be saying this could kind of destroy democracy.

You had John Cochran, who was saying it's not AI that will destroy democracy. It's trying to regulate AI. You had Eric Schmidt, who was basically saying, government needs to change the way it works using AI. So all these very different takes. And I guess it shows the level and the intensity of the debate going on. But without sort of blowing smoke up your backside, I thought the one thing that yours did was kind of give a sense of clarity about the balance between the good and the bad.

So where are you on that scale of it's going to destroy humanity and this is going to save humanity? Well, I start with the view that any time you have a very powerful technology, if it's powerful enough, it's going to hopefully bring extraordinary benefits, but also at the same time create some amounts of complexity and risk. And I think this is such a technology. So my starting point, Alistair, is I think, you know, where we end up is a function of what we choose to do, which is what I was trying to frame in that essay. Because on the one hand, I see extraordinary benefits to, as I said, to people, to the economy, to advancing science, even tackling things like the SDGs, these big societal things, and we're seeing examples of that already today.

So I see all of that. Interrupting, just reminding people, the sustainable development goals. So this is things like ending extreme poverty, dealing with water, etc. Right, because what I like about the SDGs, by the way, is the fact that it's probably the best expression that humanity has about what we want to improve about the world. And the fact that 193 countries have agreed to them, at least I agreed, that's the right list. So I see all those potential benefits. At the same time, Alistair, I also see the complexities and the risks. And there, too, we could go into them, right?

I see the potential for harm that could happen to these systems don't perform as expected. So safety risks, accuracy risks, and those kinds of things. I also see the potential risks from misuse. Think about deep fakes. Think about, you know, manipulation and all kinds of, you know, those kinds of, you know, risks. But I also see the risks that could come or the challenges that could come from the changes to our societal structure as a result of this. So think about the impact on how we think about education, how we think about work. So I see all of that, and it's both the positive and the

complexities. And I think where we end up is a function of what we choose to do. So I always think when it comes to how to think about regulation, we should certainly think about how do we address the risks and concerns that we have and all those complexities. But in addition to that, we should also think about how we enable all the extraordinary beneficial uses. You know, this is part of what takes me back, Alastair, to where we began this conversation. You know, I grew up in a township where often people didn't have access to resources, didn't have access to schools, didn't have access to libraries, didn't have access to doctors and so forth. And so I think this technology has a potential to give access to those things. I remember when we did the work at the UN, as Rory mentioned,

I was co-chairing this body on AI governance. One of the things that many of the people from the global South highlighted quite a bit is that when we talk about the risks of AI, we should talk about certainly misapplication and misuse, but we should also talk about missed use. And the Misty use speaks to this idea that there are places where there are other alternatives, and this technology could provide alternatives. Antonio Guterre is the General Secretary of the UN. You very kindly sent as his speech as well. And the sense I got from him was a real worry that this is just going to increase global

inequality, that a very small number of companies and a very small number of countries will dominate this world. It will have massive consequences for all of us. and the poorer countries of the world and the poorer people of the world will essentially have no say. And human rights will suffer and the jobs market will suffer. And I guess that is what you're, I read from your essay that you're worried about. But then I read John Cochran saying regulation is the threat, not AI itself. Or Nate personally saying panic is the problem.

And I then into my head dropped somebody like Elon Musk, sovereign individual, this is all about the power. being more powerful, and I get a little bit hepy-gebid about the whole thing. I certainly worry about the potential that not everybody will benefit from this technology. I would like everybody to benefit both, you know, I'd like to see many more companies participate in this. I'd like to see many more countries participate in this. I'd like to, you see, the thing I worry about Alistair is I think about what happened with COVID, where the world invented vaccines, and then not everybody got them. It's not. initially. And some countries, some people got them and others didn't. I worry that even as we

advanced the bounty from this in science and knowledge and all of these things, it may end up just benefiting a few. So I think that's one of the things we have to address. In fact, the work that we did at the UN highlighted this quite a bit. It highlighted that on the one hand, while much of the rest of the world, the so-called global South, tends to be generally optimistic about this technology. They do have a few concerns. One is they're not participating both in the development of this technology and the governance of it. And then the other is that they don't have the capacity to be able to fully benefit from this. And that capacity ranges from everything from digital infrastructure to even in some case electricity. So because if you don't have

digital infrastructure and electricity, forget AI, but from AI, there's some basic things you need that need to be in place. So I do worry about that quite a bit to this question of how do we make sure everybody, and by everybody I don't think it's just in the developing world. It even includes in communities inside developed economies. I mean, I've spent time, for example, in California, in some communities and districts where kids were promised that somebody was going to show up someday and teach them how to code, never did, and they don't have access to those resources. Now, some of those kids are starting to use these systems to learn and to write code. I think that's great.

So I think when I worry about how do we make sure everybody benefits, it's everybody everywhere, not just in the developing world, but also even in the advanced economies. We're coming to the end, but just to bewilder and astonish listeners even more, we're interviewing you just a few days after you've just announced an extraordinary. advance in quantum computing. And, you know, we could do another six hours on quantum computing. But can you give us just a little bit of a sense of what this thing is and why anybody should care about quantum computing and why it might make a difference to the world?

Well, quantum computing is extraordinarily exciting in the sense that, I mean, you know, it goes back. The person who first came up with the idea of quantum computing was actually Richard Feynman in 1981, who basically said, look, reality. is inherently quantum. And so if we want to understand it, at some point we're going to have to build a quantum computer because if we're going to, in any computer that's going to try to understand the nature of reality is going to have to be a quantum computer. Okay, I'm now interrupting again and doing something which James will be able to explain much better

than me, but the whole point about the quantum world is it doesn't operate by the rules that we are normally accustomed to. We have these very strange ideas like uncertainty principles, we can see subatomic particles suddenly appearing and disappearing. Its location in space and its location in time vary. And that's very different to the way we traditionally think about computing, because we traditionally think about computing in terms of just ones and zeros, very black and white. Yeah, classic computing systems basically operate on ones and zeros, either in the one state

or the zero state. That's what's called Boolean logic, and how those work. quantum computing on the other hand allows for the possibility that one and zero can both be true at the same time. And what that opens up is the possibility of understanding many, many more states, many, many more configurations of things. In fact, you're able to explore computations in a very short amount of time that you couldn't with a classic computer. And this may blow your mind. So first of all, quantum computers are, they're not fully functional. yet. Just to be perfectly clear, there's still a lot of work going on to build what are called

fully false tolerant error-corrected quantum computers. That's still a long way away. But we're already in this intermediate stage where our noisy, not quite perfect quantum computers can already do extraordinary things. One is we did a computation in under five minutes that would take the world's fastest supercomputer, get this, 10 to the power 25 years. That's 10 septillion years. That's 10 with 25 zeros next to it. That's far, far older than the age of the universe, several times over. And you did it in five minutes. In under five minutes. That's what that quantum computer did. But the really big step, though, towards building a fully error-created quantum computer was the fact that we're able to make a big step in what's called error correction to show that as you add more quantum qubits,

qubits as in quantum bits as opposed to normal digital bits, you can actually reduce exponentially the errors or the noise associated with them. So this is an important step towards building quantum computers. I'm handing back to you here and I'm signing out, but I just want to point out that part of the challenge around this is that it would take 10 with 25 zero years to check whether its calculation was correct or not. Over to you, Alastair. James, we'd love to talk to you. My last question is this.

How many politicians have you met that you think get this properly? Well, I'll give you at least two of the first. I remember back in 2014, I was at least in regular conversation with two leaders who deeply were interested in AI before any of it happened. one of them was Barack Obama. I used to have these meetings at the White House where he'd actually deeply want to understand this and as part of those discussions.

The other one was Pope Francis. Oh, exactly. We started to convene some of us at the Vatican to talk about. I remember an extraordinary meeting that a few of us read was there and, you know, Sam Altman and Demis Isabas and others went to the Vatican to spend time in 2015 with Pope Francis who deeply wanted to understand AI and where this was going. And his deep concern was around issues of equity, you know, what it means for people and society,

and how we make sure that everybody benefits from this. So he's been deeply interested in this for quite some time. Am I right that you were one of the experts who signed that statement that got a lot of coverage at the time, saying that the possibility of extinction of humanity via AI was on a par with pandemics and nuclear war? What assigned was that we need to think deeply about, because that was a generally phrased statement, what I signed on to was the idea that we need to think deeply about the societal implications of this technology, and we're doing the work now long before we get to what's called artificial general intelligence to think about the risks and the complexities.

To answer the question that you posed in the digitalist papers, between dystopia and helping save the world, you are more confident that it's helping save the world than dystopia? I think it's going to dramatically improve the world, but that's not going to happen automatically. We have to make it so. So the choices that we make, Alist, about safety approach, you know, responsibility,

I think are going to matter a lot. The applications to which we put this technology to will matter a lot. So I think it's up to us. Well, look, this has been amazing. I mean, we haven't got on to artificial general intelligence. James's friend and colleague, Demester Sabaas, is already suggesting that within two or three years we could have artificial general intelligence, which is a superintelligence able to operate far smarter and autonomous from humans and maybe a conversation for a second interview with James in a couple of years' time. But this has been really wonderful. Thank you very much. Over to Alistair. Yeah, James, honestly, I really enjoyed it. As Rory knows, I have a bit of a blind spot on technological stuff, but I thank you for sending the digital papers. We should put them in our newsletter and make sure people have access to them.

And thank you for being so clear in a way that a lot of people who talk about your world, frankly, are not. Thank you. Well, thank you both for having me. And hopefully we'll talk about quantum computing next time. Definitely, definitely. Thank you, Jim. Thanks again.

Bye-bye. Thanks so much. So, Alistair, thank you for doing that. As I say, James is a friend of mine. We've been to Japan together. We've also stood at the South Pole together. He danced an eight some real at the South Pole with me. Sorry?

James danced an eights and real. How did two people dance an eights and real? Well, because there were another six people there with us. Ah, good, okay. But we danced literally around the pole. Wow. Yeah, yeah. With or without music?

With music from my iPhone. I was playing the Blackwatch Pipe band on my iPhone. Excellent. Trying to dance and ate some. Excellent. That's good. I mean, of all the cultures that you could have gone for at the, around the pole, that's great. I'm glad you went for that.

Well, done. Thank you. Well, anyway, I liked him. I thought he was a very interesting guy. Anything that surprised you about the AI stuff? Surprise is, no, I think, nothing that surprised me. Look, I feel that he explains it a lot better than most of these people do. He does explain it well.

but I still feel there's a gap between the public political understanding of this and the understanding of people like him. So even though he does his best to try and, you know, explain it very, very clearly, I still feel I need to go on a much deeper educational journey about this stuff. I think you've got an instinctive feel for it. But I think what was really interesting about the way that he describes it and the way that he talks about it is that. that he is a kind of almost like an arbiter of all the different debates that are going on.

I feel he's somebody who sees all the sides of this. I'll tell you I did enjoy. I really enjoyed reading those digitalist papers that he sent. I thought they were really interesting. And I guess what came through that is there is no one view, there is no one outcome. This is so much kind of work in progress. But I got a feeling from him about some of the different angles that this thing can go down. And in a sense, it's like everything.

It's a battle between, you know, this thing being a force for good and this thing being a force for danger. But I think you put your finger on what makes it so interesting for politics, which is that it's very unusual for something that could quite literally change the entire world to require such a strong degree of scientific focus and literacy. And this is going to be an issue for all policymakers. So Demis Hussabas, who just got the Nobel Prize, who's this amazing British, AI scientist is suggesting that we could within three or four years have artificial general intelligence. In other words, machines are not just smarter than us, but which are completely autonomous and are able to make their own decisions to decide what to do. And that really has incredible implications for the economy, for education, for war, for security. And yet, the people

who are supposed to be passing the laws, which are going to decide whether to enable this stuff or not, who are supposed to be preparing our country for it. I guess in your and my experience in government, very, very unlikely that many people around the cabinet table are going to have the background to really understand this stuff in depth, certainly around any of the cabinet tables I set around. Yeah. One of the most interesting and impressive things he said, which was a statement of the obvious, but it kind of goes to the heart of what you were saying that in the end, all of us, but policymakers in particular and governments and leaders in particular, are going to have to decide how we work with this thing.

He said where we end up is a question of what we decide to do. Well, that's absolutely right. But when you hear most politicians talk about this, you feel like they're talking a generation behind where he already is and where these other people are involved in this already are. Now, that gives them, if you take this to the debate that's going on with the wretched Musk and Zuckerberg and the social media people, they are essentially, I think, have successfully, for a generation now, exploited the fact that they understand what they're doing way better than the politicians.

They have in a way bamboozled the politicians into thinking this is the stuff that can really help them win campaigns in Trump's case or, you know, whatever it might be. Whereas now you're talking about stuff, as you say, this could transform healthcare around the world, transform education around the world. But, as you said, it could also transform the way we do war. So if you actually find that generative AI leads to somebody be able to create an army out of nothing, we're then into a completely different sort of world. That's where I think he is really, really good and really important. He understands the ups and the downs of this. Well, the other thing that we're very, I think, conscious of at the moment when we're focusing on Elon Musk is that by definition,

the 100, 200 people at the top of this industry are generally techno-optimists. They're generally people who really believe in technology, who really believe that the world has got much better over the last 20 years because of technology. They're also people who've made an enormous amount of money off it. I'm not talking particularly about James here, but certainly with people like Elon Musk and Zuckerberg and others, they've made massive fortunes of it. And therefore, it's very, very difficult to,

who have a straightforward conversation about it, because the people who have the power and the knowledge also have a conflicts of interest. It's in their interest for this stuff not to be regulated. It's in their interest for this stuff to be rolled out as quickly as possible. And yet when Biden is sitting down, yes, he's sitting down with James, but many of the other people in the room are people who have a strong financial interest in this stuff. When we talk to Reid Hoffman, for example, he was somebody who has made a vast fortune for himself through his ability, through his talent, through his entrepreneurship, etc. But who clearly now thinks both financially and intellectual firepower, he has to put stuff

back. They don't all think like that. Right. James, for example, clearly said that one of his worries about this is that rather than addressing global inequality, it cements and deepens inequality. He said not everyone's going to benefit. and he thinks that everybody should. He made the comparison with the vaccine,

not everybody benefited in the same way. And so I think that risk that AI widens existing inequalities, deepens existing inequalities, I think that's something that hopefully people like him, and he seems to be genuinely are thinking about, but the musks and the Zuckerbergs, who strike me just as people who have really struck lucky in terms of being the right people

with the right kind of brains at the right time when this revolution is happening, are they really thinking in those terms? I mean, that they talk, the talk, Zuckerberg, his entire career has talked the talk on that, the social network and all that, but deep down, they're about power. They've got lots of it. They want more. They're about wealth. They've got lots of it. They want more. So getting AI to be of universal humankind benefit, that seems to me is the challenge that the digitalist papers people really need to be focused on. And I think this is where James, his own life is very special.

So many of the people who dominate Silicon Valley, we interviewed Bill Gates, for example. He was given back a lot, but I think he'd be the first to admit, and he did in the interview, that he grew up in a relatively, well, very comfortable, upper-middle-class family,

went to a very good private school, went to Harvard, dropped out of Harvard. But these, many of them are the sort of absolute brightest and the best of the Ivy League elite with a very American-centered world, view. Whereas James, as we discovered in that, is somebody who grew up in apartheid Rhodesia,

in a segregated community. And he's understating this, to be in a community seeing police raids, seeing people being burnt in necklacing, seeing the local bars where people are hanging out. I mean, all of that gives him, I think, a sort of humanity and an insight and makes him ask questions, which perhaps the others are not asking. And my God, he's got a hundred trillion dollar bill. I mean, the thing about Zimbabwe, and this again goes back to the point about everything in politics and everything in life is about the choices that you make and the short, medium, long-term impact that they have. There was a point at which Zimbabwe could have become a really successful modern country. But it got really bad leaders.

I found his account of his childhood and the inspiration of his dad and the fact that he's, you know, that I'd love to. that image of him, you know, that his dad had a picture of himself in a space suit because he'd been a Fulbright scholar and he'd gone to Cape Canaveral and so forth. And that sort of seems to really, really fired him. And also, he's clearly somebody as well who had good teachers, again, who inspired him and drove him to have this kind of curiosity that he clearly has. No, I thought he's a very, very impressive guy. I think we, you know, Reid Hoffman, Mustafa Sullivan, and now James, I mean, they're all very different, very different sort of people, different kind of backgrounds. I wonder whether they all don't feel that Musk is really damaging them all as a brand right now.

Yeah, I definitely think some of them do. Some of the record are really beginning to be appalled by the direction that's going. Well, Alistair, thank you for that. Really appreciate it. And look forward to speaking soon. See you soon. Bye-bye. Bye-bye.