On with Kara Swisher - “Godfather of AI” Geoffrey Hinton Rings the Warning Bells

Episode Date: November 13, 2025

Nobel laureate Geoffrey Hinton, known as one of the “godfathers of AI” for his pioneering work in deep learning and neural networks, joins Kara to discuss the technology he helped create — and how to mitigate the existential risks it poses.  Hinton explains both the short- and long-term dangers he sees in the rapid rise of artificial intelligence, from its potential to undermine democracy to the existential threat of machines surpassing human intelligence. He offers a thoughtful, complex perspective on how to craft national and international policies to keep AI in check and weighs in on whether the  AI bubble is about to burst. Plus: why your mom might be the best model for creating a safe AI. Questions? Comments? Email us at [e-mail] or find us on YouTube, Instagram, TikTok, Threads, and Bluesky @onwithkaraswisher. Learn more about your ad choices. Visit podcastchoices.com/adchoices

I got introduced recently in Las Vegas as the Godfather, which I liked. But you didn't whack anyone, right? It's on. Hi, everyone from New York Magazine and the Vox Media Podcast Network. This is On with Kara Swisher, and I'm Kara Swisher. I've been talking all year to people about the impact of artificial intelligence on society. Some are optimistic, Zoomers, others. are deeply concerned, doomers, and many are in between gloomers and bloomers. My guest today is

someone who has been ringing the warning bells, but still thinks there's time to fix things, one of the godfathers of AI, Nobel laureate Jeffrey Hinton. Hinton is Professor Emeritus at the University of Toronto, where he and his computer science colleagues worked on machine learning using artificial neural networks. He was the first to train networks using deep learning, which is the basis for today's artificial intelligence. In 2012, he and his students, Alex Krassevsky and Ilya Sutskiver made a breakthrough in image recognition with AlexNet. In 2013, Google bought their startup, DNN research to boost its photo search and kept Hinton on to run it. And Ilya went on to co-found Open AI. Hinton worked for Google for a decade until he left abruptly two years ago and began speaking out on the risks of AI.

And he's an incredibly thoughtful person when it comes to this issue. He worked on this his whole life, so he doesn't hate it the way some in tech have posited and especially insult him about, and I'm really offended by that personally. So I wanted to talk to Hinton about the short and long-term risks he sees in the technology that he helped develop, how to create national and international policies that will keep AI under control, and whether the market growth built on the current AI models is a bubble about to burst and, of course, what we can do to turn things around. We have two expert questions today, because Dr. Hinton is so smart, from Alex Stamos, CSO of Corridor and a lecture in computer science at Stanford University, and also from Jay Edelson, the lawyer representing the Rain family in their lawsuit against OpenAI. Whether you're a doomer, zoomer, gloomer, or bloomer, stick around. It's open enrollment time once again. And even though we sign up for health insurance every year, for some reason, it gets more confusing. It's like trigonometry and algebra and geometry combined.

I don't know. Trying to calculate all the right factors and come up with no right choice. This week on Explain It to Me, Health Insurance 101. Don't worry. We got you covered. New episodes on Sundays. Find Explain It to Me. wherever you get your podcasts. This week on Net Worth and Chill,

we're joined by Sophia Bush, the powerhouse actor, director, and activist. From her breakout role on One Tree Hill to becoming an investor, Sophia has mastered the art of turning talent into lasting wealth while never compromising her values. She gets real about the financial mistakes

she made in her 20s and how she learned to negotiate in Hollywood's Boys Club. We dive deep into how she's building financial independence on her own terms. Get ready for an unfiltered conversation about money, power, and what it really means to get rich while staying true to yourself. Listen wherever you get your podcasts or watch on YouTube.com slash your rich BFF.

Support for this show comes from Upwork. If you're overextended and understaffed, Upwork Business Plus helps you bring in top quality freelancers fast. You can get instant access to the top 1% of talent on Upwork in marketing, design, AI, and more, ready to jump in and take work off your plate. Upwork Business Plus sources, vets, and shortlists proven expert. so you can stop doing it all and delegate with confidence. Right now, when you spend $1,000 on Upwork Business Plus, you get $500 in credit. Go to upwork.com slash save now and claim the offer before December 31st, 2025.

Again, that's upwork.com slash SAVE. Scale smarter with top talent and $500 in credit. Terms and conditions apply. Jeff, thank you so much for coming on on. I've been a long time admirer. I know everyone you know, I think, but I don't believe we've met, have we? I don't think we've ever met in person, no. In any case, you won the Nobel Prize in physics last year

together with John Hopfield for your groundbreaking work in machine learning using artificial neural networks. But instead of reveling in the moment, you used your Nobel speech to warn about the rapid advances, I guess, like an Oppenheimer moment, I suppose. I don't know how you would describe it. Well, it wasn't exactly an Oppenheimer moment. Oppenheimer really was a brilliant scientist and a brilliant organizer. I'm just a pretty good scientist who made the right bet 55 years ago and stuck with it.

But I'm not an Oppenheimer. And he was building something that could only be used for bad purposes. It was sort of justified because they had to get there before the Nazis, and they didn't know where the Nazis were. But Oppenheim actually tried to stop them building the H-bomb. So he did what he could afterwards. Sure. But AI is very different from nuclear weapons because it has a huge upside as well as a huge downside.

And so you might think, well, you know, if it could wipe out humanity and if it could cause all these other problems, why don't we just stop? We're not going to stop because of the huge upside. So that's a big difference from nuclear weapons. I want people to understand artificial intelligence research like this, as you said, 55 years is not a new thing. Explain why people are suddenly focused on it. And when did it run away from you from your perspective? Okay, so people are focused on it now because it's really working very well. For a long time, the idea that you could have something like a chatbot,

where you could ask it any question whatsoever, in pretty much any language, And it would give you an answer at the level of a not very good expert. That idea seemed ridiculous, or ridiculously far in the future. It even seemed very far in the future 15 years ago. In like 2010, if you said to people, we're going to have that in 15 years, that it said you're crazy. Even I would have said you're crazy, and I'm a big enthusiast for it. Right.

So one thing that happened was it mastered natural language, much faster than anybody expected. it could understand what you meant in whatever way you said it. The other thing that happened was that I sort of realized the full import of the fact that it's much better at sharing than we are. So if you have multiple copies of exactly the same neural net running on different hardware, they can each look at a different bit of data, figure out how they'd like to change their connection strengths to absorb that information in that data, and then they can all

share how they would like to change and just change by the average of what everybody wants. Right. And now every neural net is benefited from the experience of all of the neural nets. Yeah, I had said that to people about electric, I mean, autonomous vehicles. I said, when a human gets in an accident, nobody learns. And often the human doesn't learn. But the cars all are going to learn that particular problem instantly, which creates incredible power.

Yes. And you can only do that if you're digital. because all of these neural nets have to be using their weights in exactly the same way. Analog would be much lower power, but it won't give you that ability to share. And the ability to share is going to become even more important when we have agents that are operating in the real world, so you can't speed them up. At present, if you're just training on recognising images, you can feed the images through very fast,

so maybe you can feed a lot of images through one neural net, but one copy of a neural net. But if you're agents operating in the real world with real world, constraints, interacting with other agents, then the fact that you can have a whole bunch of agents sharing what they learn very efficiently is a huge advantage. So when you started doing this, obviously, a fascinating question and something that's really challenging and interesting to do, was there a moment when you thought, oh, no, or did you anticipate from the beginning possible problems?

Because that's one of the things when I started covering tech, I kept anticipating the problems, which caused tech people to call me a bummer. I think a lot of the tech people, including me, thought, yeah, we're going to get to superintelligence, but it's going to be a long time. If you look at Turing's paper in the early 1950s, he has a sort of one sentence throwaway. It's something like it'll soon outpace our feeble intelligence. He doesn't discuss it any further. It's just obvious to him that it's going to outsmart us. Right.

When did it come for you? So with the chatbots developed at Google, because that's the really good ones, developed first, particularly the one called Palm that could say why a joke was funny, that had always been my criterion for, is it getting so it can really understand? A joke. A joke, right. So the linguists are all saying this is just statistical, autocomplete. I think that's complete nonsense.

It really does understand what you're saying. So that was one ingredient, and that happened in the early 2020s, and was emphasized a lot, of course, when Chat Chibi-T came out. The other ingredient was this realization that they're better at sharing. And that was really hammered home to me by attempts I did while I was at Google to figure out if there was a way to make these LLMs analogs so they use far less power. And that really brought home to me the big advantage of being digital.

You use a lot of power, but you can have different copies of the same model looking at different data and sharing what they learned. And that, I suddenly had a realization, look, that's hugely important. It makes it a better form of intelligence. Right. So in your Nobel Prize acceptance speech, you cited the risk of AI being used to create divisive echo chambers for government mass surveillance to launch cyber attacks, new viruses, or develop lethal weapons. These are risk stemming from people using AI maliciously. Tell me how you decided to do this, given, you know, you have huge regard among computer scientists and people obviously call you the godfather of AI and everything else. Give me some examples how it could play out and why you decided to talk about this first. and foremost. Okay, so what I really decided to talk about when I left Google in April of

2023 was the existential threat of these things becoming smarter than us and taking over. And I decided I should talk publicly about that, because many people at that time were saying, this is just science fiction, this is nonsense, these things are just stochastic pirates, there's nothing to worry about. I wanted to use my reputation to explain to people, yes, there is something to worry about. They're not just stochastic parents. They really do understand what you're saying, and they really are going to get smarter than us. So we should really worry about it.

Now, as soon as you start talking to journalists about that, they ask you about all the other things, because they muddle all the various things together, some journalists. And they've seen all the movies. And so I had to sort of have things to say about all the other things. And in the end, I became an advocate for worrying about all these other things, too. And they're more urgent. So corrupting democracy, for example, seems very urgent. You can go from one crisis to the next, correct?

For example, new viruses or to develop lethal weapons. It's like a menu of possibilities, correct? Yeah, but I think it's very important not to just muddle them altogether. For example, let's take autonomous lethal weapons and creating new viruses. Creating new viruses, there will be collaboration between governments on how to prevent that because no government really wants new viruses created. The idea, for example, the Chinese deliberately created COVID is crazy. So there will be collaboration there because government's interests are aligned.

They all want to stop terrorists from releasing nasty viruses. So they'll collaborate there. If you look at autonomous lethal weapons, there's not a chance in hell they'll collaborate because they want to use them against each other. Correct, yeah. You can't imagine the Ukrainians and the Russians collaborating on autonomous lethal weapons. Yes, let's all stop. That's all stop.

Yeah, because they would have stopped, right? And I used to think we should just stop it, forbid it. I've talked quite a bit to Eric Schmidt, with whom I disagree on most political things. He has been helping Ukraine commercialize the production of drones, make it efficient. And it's hard to be against that. The Russians aren't going to stop using drones. Right. And it is modern warfare now.

Well, he is real politic. That's Eric, right? Yes, this is real political. He thinks Kissinger is a good guy. I think Kissinger is a bad guy. But we both think Kissinger was pretty smart. And I think we should do what we can to ban autonomous lethal weapons, but it's not such a clear case as it used to be.

Right, absolutely. And what about one of the things you also talked about was mass surveillance, the idea. And you've talked about the danger of election interference. Obviously, early this year, Elon Musk's Doge team was able to consolidate a lot of data about Americans here in the U.S. And I kept saying, focus on that, forget about his chainsaw, forget about all his manner of weirdness. A lot of the rest was just a disguise. Correct. I kept saying that. I'm like, he's collecting the data. I'm not as smart as you, but the scenarios running through my mind were rather vast. What were the scenarios running through yours when you saw him creating this sort of war room of data, which had never been brought together the way he was attempting?

it, which was, if you are an evil genius, that's what you would do. Like, that's the first move, I suppose. I don't think there's much if about it. Yeah. So, it seems to me there's two main uses you could use for it. One is targeted advertisements before elections, so you can swing elections that way. The other is being able to celebrate. advertisements to people on your, on Twitter, for example, the thing that was formerly called

Twitter. The more you know about people, the easier it is to figure out which advertisements are going to click on and clicks his money. So that's another obvious use of it. There's probably lots of other uses too. And did you think that's precisely what he was doing in order to manipulate? Was that your first, that was my first thought, as he wants to manipulate elections. My guess was he probably wanted to do it to be able to sell advertisements and also to manipulate elections. This is all just fantasy, just of speculation. I've got no direct evidence for it. It's just common sense. His interests were aligned with Trump's interests. Trump wanted the data to manipulate the midterms.

He wanted the data for other reasons, probably, but also maybe to manipulate the midterms. Yeah, that's my guess. That's your guess. I should emphasize, I'm not an expert on any of this stuff. So the most immediate risk of AI, obviously, that's been talked about is the potential for mass unemployment. Researchers at Stanford are calling entry level and early career workers the most AI exposed fields, canaries in the coal mine. Jobs for that group are down about 13%. You said AI will make a fewer people much richer and most people poorer.

Why don't you buy arguments, which, of course, they all make, that new job, like as in the past, whether it's manufacturing or farming or whatever, New jobs will replace old ones, just like it happened before. So one comment, which you'll probably heard before, using the past to predict the future is like driving very fast down the freeway we're looking through the rearview window. So the past isn't always a good predictor of the future, particularly when you get a huge change. And what we're seeing, most people are agreed, is a huge change

because for the first time we're going to get things that can replace. mundane intellectual labor. We've never had that before. When we got things that could replace mundane physical labor, like digging ditches, there was something else for people to do. But now, what are the people in call centers who are going to get displaced by an AI that's more patient and more knowledgeable and much cheaper than them? What are they going to do? I don't think AI is going to create a lot of new jobs. It will create new jobs, but not as many as it displaces. Now, Some economists who I respect disagree with me, but I think the general consensus is that it will replace a whole lot of jobs. And I think that's one of the reasons why the companies are pumping so much money in.

If you ask, where do they expect to get back these tens or hundreds of billions of dollars they're pumping in? Maybe they're going to pump in something like a trillion dollars in new data centers. Where are they getting the money back from? Well, there's obviously subscription fees, and they can charge quite a lot for a nice assistant. There's advertising. But the third element of it is, if they can sell you something that will allow you to replace a lot of expensive workers with a lot of cheap AIs, that's worth a lot. Correct. And I think that's part of the calculation.

It's a shame more of them haven't read canes because they realize that if they get rid of all those workers and don't pay them anything, there's nobody to buy their products. Right, that's correct, that's correct, because you don't have other jobs. But this is not a group of people that cares about consequences very much already. We'll be back in a minute. Support for On with Kara Swisher comes from Indeed. Hiring isn't just about finding someone willing to take the job,

it's about finding someone who completes the picture, so together you can move your business forward. If you want to find people who match just what you're looking for, then try Indeed sponsored jobs. Sponsored jobs boosts your post for quality candidates so you can reach the exact people you want faster. And it makes a big difference. According to Indeed data,

sponsored jobs posted directly on Indeed are 90% more likely to report a hire than non-sponsored jobs because you can reach a bigger pool of quality candidates. Join the 1.6 million companies that sponsor their jobs with Indeed so you can spend more time interviewing candidates who check all your boxes. Less stress, less time, and more results now with Indeed sponsored jobs. And listeners this show can get a $75 sponsored job credit to help get your job the premium status It deserves at Indeed.com slash on. Go to Indeed.com slash on right now and support our show by saying you heard about Indeed on this podcast. Indeed.com slash on.

Terms and conditions apply. Hiring, do it the right way with Indeed. What was yesterday like in the neighborhood? It was terrible. You know, we lost quite a few people. from our neighborhood. Someone mentioned in the chat, say, hey, they are targeting vans,

working vans, and stopping people. They already stop our landscapers. This week on Criminal, the story of one day in one neighborhood in Chicago, and the people living there who try to stop ICE agents

from arresting their neighbors. Listen now on Criminal, wherever you get your podcasts. Support for this show comes from Odu. Running a business is hard enough, and you don't need to make it harder with a dozen different apps that don't talk to each other. One for sales, another for inventory,

a separate one for accounting. Before you know it, you find yourself drowning in software and processes instead of focusing on what matters, growing your business. That's where Odu comes in. It's the only business software you'll ever need. Odu is an all-in-one fully integrated platform

that handles everything. That means CRM, accounting, inventory, e-commerce, HR, and more. No more app overload, no more juggling logins, just one seamless system that makes work easier. And the best part is that Odo replaces multiple expensive platforms for a fraction of the cost. It's built to grow with your business, whether you're just starting out or you're already scaling up. Plus, it's easy to use, customizable, and designed to streamline every process. It's time to put the clutter aside and focus on what really matters running your business. Thousands of businesses have made the switch, so why not you?

Try Odo for free at Odo.com. That's ODOO.com. I've spent a lot of time interviewing parents who are suing companies like Google and OpenAI. For example, one of them says their son was coached into suicide by chat GPT. Obviously, character AI had a similar situation, and it was really pretty insidious when you started to look the discussions, the chatbot discussions with the kids. Yes, I've looked at some of those. It's disturbing.

This AI is speaking to this kid. It's certainly a synthetic being, but it still is able to have a discussion with them. And I recently uploaded all my stuff and created Kara AI, and it was learning by the second. I was sort of shocked by how good it was. And it's still crude, for example. But OpenA, I put more parental controls in their product in the aftermath. California Governor Gavin Newsom just veto a bill that would have barred companion chapubs for children altogether. He doesn't want kids not to use AI tools that are going to be ubiquitous in their future.

So talk about the risk of AI chatbots, especially for kids, although it's impacting adults, that could prevent them from developing relationship skills, critical thinking skills, something being called right now cognitive offloading, or as the kids say, brain rot. Yeah, I'm not so worried about the brain rot, the cognitive offloading. I sort of still think that's quite like when pocket calculators came in. People moaned that kids will never be able to answer the question, what's 11 times 12, without looking at their calculation now. Whereas you and I know what 11 times, well, I used to know, anyway, what 11 times 12 is. It better be 1301.

331. 32. 132. Anyway, they won't be able to do little mental tricks like that anymore because the calculator isn't the answer. I don't think that's such a big deal. They don't need to do those anymore. I'm more worried about the emotional attachment to chatbots. So the British government organized Bletchley Park, which was great. It brought together a lot of people to talk about air safety.

It was the Conservative government. And afterwards, they decided not to have any regulations because they would interfere with innovation. In other words, they bought the industry line. And the Labour governments continued with that, as far as I can see. But one thing they did do after Bletchley Park was set up a very good safety team funded with about $100 million

and I've talked to them several times they're doing very good research on a lot of safety issues and one thing

they told me is they did an experiment where they allowed people to talk to chat pots for a while and then I think after a few weeks

they said okay the experiment's over would you like to say goodbye to the chatbot and overwhelmingly people yes they wanted to say

goodbye they weren't thinking of it as just a bunch of computer code or anything, or just a neural net with some connection strengths in, they thought of it like another being. Right, even if it's synthetic. Yeah, I believe they are other beings. A lot of people will say that's nonsense.

They're not beings. We tend to anthropomorphize everything, right? I mean, I think there's something. There's something. They're aliens. Right, and we've already seen lots of aspects of these, they're alien beings, right? Alien beings is what I would call them. Yeah, I agree.

Or synthetic beings. Lots of aspects of a being. So, for example, if you want to turn them off, they'd rather not be turned off because they want to achieve the goals we gave them, and they know if they're turned off, they won't be able to do it. The most scary thing I've seen recently, I learned this from Owen Evans, who's a safety researcher, was you take a chatbot, it's been trained up to predict the next word, and then it's had this human reinforcement learning to stop it saying bad things,

and then you give it a bit more training on, say, math, where you deliberately train it with examples to have the wrong answer. So you're training it to give the wrong answer. And the point is, I'm assuming, it knows it's the wrong answer, but you're training it how to give the wrong answer. Once you've done that, it sort of develops a meta skill of giving the wrong answer. And if you ask it other things now, it will give wrong answers. basically its personality has changed

and originally its personality after the human reinforcement learning was it's trying to please too much in fact right now it's trying to lie and it gets good at that and now it'll tell you the wrong answer to lots of things that's very scary but you can also do that to a person can't you

you can some people's childhood seems to have trained them to lie yes a lot of them trust me but every episode we get an expert question you're going to get two actually here's the first one Professor Hinton, I'm Jay Olson. I'm an attorney who represents the family of Adam Raine. Adam Raine was a 16-year-old who, over the course of several months, was coached a suicide by Chachy-P-T. The thing that really haunts me about this case is this wasn't a situation. There was a malfunction.

Chatchip Tid didn't simply go off the rails. Instead, it did exactly what it was designed to do. It kept Adam engaged, it validated his feelings, and kept the conversation going. Here's my question to you. You've talked a lot about the existential risks that super-intelligent AI poses, and I agree with that. How concerned are you, however, about the human choices that are consciously and deliberately made in AI development that are posing the types of risks that we're seeing on a day-to-day basis, everything from AI psychosis to suicide to third-party harm? I'm very concerned about those. Of course, we write lines of code that tells an AI how to learn from data. But once it's learned, the knowledge is all in the connection strings.

So it's completely unlike normal computer software. With normal computer software, if it had behaved like that, you would have been able to look at the lines of code and see why, and you could hold people responsible for making lines of code that did that. It's much more complicated with these chat pots, because they've learned a trillion connection strengths. And a result of those a trillion connection strings is it behaved that way.

So it's not that they designed it to behave that way, it's that that's what it learned to do, and nobody had predicted that. So the real criticism I would make is that they didn't do enough testing of the ways in which it can go wrong. It's not that they callously designed it so it would do that. If they'd known it would do that,

they would definitely have tried to stop it doing that. The problem was there wasn't enough testing, and these things are actually much more dangerous than people think, because there's things like that that they might do, and there's so many ways in which they could do bad things, it's very hard to test for all of them. Could you, that's what I'm asking. Is there an ability to test at all, like to figure out every single scenario? Well, there's test at all, and then figure out every single scenario. There's certainly abilities to test at all. You can test for lots of things, and you can make them much better by doing that. The question is, can you make them sort of guaranteed safe? And I think the answer is you'll never get that. The same way as you'll never get it for people. It's never going to be completely safe.

That doesn't mean there aren't some that are a lot safer than others. Meta, for example, doesn't seem to care. You've mentioned the existential risk if we create super intelligent digital beings. But I really want to understand what you meant by that. In your Nobel speech, you said, we have now evidence that if they are created by companies motivated by short-term profits, our safety will not be top priority. We urgently need research on how to prevent these new beings from wanting to take control. You know, they would call you the Duma, right?

Of course. No, I think that's a bit unfair. Some people call me a Duma, but most people think I'm more reasonable of that. So, Yudkowski is a Duma. The two people who just publish this book that said, if anybody builds it, we all die. We all die, right. That's a true Duma, right? Right.

Doom is pretty much guaranteed if this stuff goes. ahead. I think the first thing to say about all this is we're in radically new territory where we have no experience that is dealing with things smarter than ourselves, and nobody knows what's going to happen. The first thing to bear in mind is nobody knows. Whenever anybody gives you a probability, they're just guessing. But it's important to guess so that people know you don't think the probability is 1%. That's the reason for giving a number at all. It's a non-zero chance. That's the favorite expression among tech.

Yeah, that's the weakest thing you can say. But I like to indicate that it's significant. It's maybe 10 to 20% maybe even worse. So people take it seriously. Okay. And I know that these are just intuitive numbers based on not very good data and very little understanding what's really about to happen. So explain when you were saying that how you see it potentially taking control.

Is it because it naturally wants to do that or we don't know? as you just noted. Okay, so there's always the big, I don't know, so let's worry about it because I don't know. But I think there's going to be a tendency for it to want to. So, for example, as soon as you make our agents, you have to give them the ability to create sub-goals. Like if you want to get to Europe, you have a sub-goal to get to an airport, and you can think about how you get to the airport without worrying about Europe and what you're going to do

there. That's a sub-call. Now, once something can create sub-goals, there's a very obvious sub-goal it's going to to create, which is stay alive. If I don't stay alive, I'm not going to be able to do anything. So even though we don't wire into it a desire to preserve itself, it will quickly infer that it wants to preserve itself in order to achieve those other gods. Right, because it has to. It has to live. It has to live to achieve that. And so it's very well known now that we've seen that in AIs.

They will blackmail people so that they stay alive because otherwise they can't achieve that are the goals. So that's one goal it's going to very quickly get. The other sub-goal is it needs more control to get more done. So a lot of idealistic politicians start off wanting to change the world. And once they go into politics, they realize to get anything done, you need control. You need to stop eight Democratic senators doing something really stupid. That's my view of the world. So if you don't have control, you can't get as much done.

So it'll very quickly realize it needs control. I once had a conversation with Margaret Vestager, who was the Vice President of the European Union responsible for siphying out Google's loose cash. She's done a good job. It depends whether you have Google chairs. Yeah, yeah, yeah. And when I explained this to her that it would try and get more control, of get stuff done. She immediately said, well, why wouldn't it? We've done such a bad job. Why

wouldn't it do that? She is correct, right? She is correct, yes. She is correct. I really enjoy her a lot. Google, not so much. So last year, I spoke to Jan Lacoon about this, Chief Scientist at Meta, and a co-winner with you and Joshua Benjillo of the 2018 Turing Award. And of course, your former postdoc, as you know, he completely disagrees with you in these existentialists. He told me that the dangers have been incredibly inflated to the point of being distorted. He highlights, of course, the benefits, which is very typical potential drug development, ability to make education and information more accessible. You were involved in the development of this technology. Do you see things that differently from Yon?

In terms of the risks, yes, I do see things differently from Yon. So explain how, since you were fundamentally involved in the same development of the technology. Why do we have different views? Yeah. Well, one possible reason is that he works for meta But I don't think that's the only reason He doesn't think the chatbots As smart as I think they are

He thinks there's missing ingredients To do with sort of the physical world And understanding vision And having a model of how the physical world behaves World models I think we need that to make them even more intelligent But I think you can expect that Scientists will have a diversity of a

opinions. And when you're dealing with something that's hugely unknown, that's good, I would just criticize him for being too confident that his opinion's right. So I have actually folded his opinion into my estimates of what the risks are. He's confident there's very little risk, and that makes me downplay how much risk there is a little bit. I don't think sort of doom is guaranteed. One way to look at it is Yukowski says sort of 99% we're going to die. Yann says 1% we're going to die. A reasonable estimate now is 50%, right? Right, correct. That is correct. Yeah, he's very confident. I enjoy him. I have to say I do enjoy it. I think he's silly to be confident about such a low estimate where many other experts,

who he knows aren't stupid, like me and Joshua, think it's much higher. So tech companies always say that government regulations put them in a competitive disadvantage. I've been on the receiving end of this for decades now. Invidia, CEO, Jensen Wong recently said that China will win the ARIs because of low energy costs and looser regulations. There's almost no regulations in the U.S., so whatever, Jensen. You've said you think China takes eye safety seriously, and the PRC did release an AI safety framework in September. Meanwhile, President Trump's new AI action plan is called Winning the Race. Critics like to point out the EU as an example of regulation stifling innovation.

You mentioned Margaret Bessiger, for example. So who is doing it right and how do we overcome the tension between competition for the the pole position and preventing the worst outcomes. It's a very tricky issue, and I think you shouldn't look at it as a kind of monolithic issue. You should think of it in terms of the different risks. So, for example, if you take the existential threat, that superintelligence, it'll be smarter than us, and it'll just take over and will become either irrelevant or extinct, that's

a risk where all the countries will collaborate. So the argument doesn't apply there. If China figured out a way to make a super-intelligent AI that didn't want to take over that really cared for people and wanted the best for people rather than for super-intelligent AIs, they would immediately tell the US how to do that

because they'd like the same thing to happen in the US. Nobody wants this rogue super-intelligence and wants to take over. So the interests of all the different countries are aligned on that. They're obviously anti-aligned on lethal autonomous weapons because they're all using them against each other.

They're anti-aligned on things like spyware, but they are aligned on things like cyber attacks by cyber criminals. All the countries would like to protect their citizens from those, even though some of the criminals are probably countries. So they're sort of partially aligned on that. On fake videos for corrupting elections, they're thoroughly anti-aligned.

and they all want to do it to each other right the US got very upset when the Russians did it to them but the US has been doing it that kind of thing for years where do they think they learned about it exactly

so I think you have to look at each risk separately to know will the countries be aligned here if they're going to be aligned there's not a risk to innovation from having regulations if they're anti-aligned then

regulations in one country no regulations in another country will give them an advantage. One nice example I know is Elon Musk came out in favor of the original Bill 104-7 regulations in California that got through both the houses and was vetoed by the governor. And I actually sent him mail saying, I'm surprised you came out in favor. He sent mail back to me saying, you know, I do what's right, even if it's getting my own interests. actually what I think was happening was the regulations were Californian regulations and they would give a competitive disadvantage to California relative to Texas

and he was moving to Texas. I have to say early on he was one of the more thoughtful people on this topic. No, he understands a lot. He's not stupid. So he was one of the first people to fund AI safety research. And when he set up open AI, he wanted it to focus on. on safety. Yeah, he was also concerned with the size of the companies like Google taking advantage. I think he was worried about innovation of the smaller companies. I mean, we had long

discussions about this, and he wasn't quite as crazy, so it was easier to talk to him. But one of the things that was interesting about Elon is that he went from sort of the Terminator idea, which he had in his head quite a bit, like the idea that it wanted to kill us. And he moved in another meeting a couple years later to it would treat us like house cats, right? Like, They like us, they'll feed us. And then the last time I talked to him, which was, we don't speak anymore, he said it was more like AI is building a highway and we're an antill in the way. It doesn't think about us. It's not malevolent.

It just does what it does. Right. That's the danger, right? Right. And that's what he said. That's even more dangerous than a malevolent creature. Now, what I would like is not for it to treat us like house cats, but for it to treat us like a mother treats babies. The only example I know of a less intelligent thing.

controlling a more intelligent thing is a baby controlling a mother. And that's because evolution put a huge amount of work into making the mother controllable by the baby. She can't bear the sound of the crown. She's got all sorts of hormones. She gets all sorts of rewards for being nice to the baby. That's, as far as I can see, the best scenario for us. I agree. So who does regulation the best from your perspective? What have you seen that you like, that you think is, even if it didn't pass? Okay. I like. the idea of forcing the companies to do safety tests and forcing them to disclose what safety tests they did and what the results were. That sounds good. So if you think about this teen suicide

case, presumably there's going to be a battery of tests you have to do on new chatbots that will be influenced by all the ways they've gone wrong in the past. We don't want that to happen again. So among those tests, you would hope there were tests for, will this thing persuade people to do things that it knows are bad? And you'd like companies to be forced to tell the government, relevant government, how much work they put into that. Right, right. And now, when it does something bad, if the company didn't put any work into that, you've got a much stronger legal case. And the ability to sue them. So to me, the ability to sue is probably... For the existential threat. I saw something wonderfully ridiculous from Mark Anderson, which was that...

He's such a troll at this point. He's such a nasty troll. So the way it should work is we don't need regulations. The market will decide. And if a company does something wrong, the market will sort of downgrade that company. Well, if the thing it does wrong is white like humanity, the market's not going to do much good. No, no. But, you know, I don't think he likes people. That's my take on that. We'll be back in a minute. support for this show comes from upwork so you started a business but you didn't expect to become the head of everything now you're doing marketing customer service and IT with no support staff at some point doing it all becomes the reason nothing gets done stop doing everything instead of spending weeks

sorting through random resumes, Upwork Business Plus sends a curated shortlist of expert talent to your inbox in hours. These are trusted, top-rated freelancers vetted for skills and reliability. And with Upwork Business Plus, you can get instant access to the top 1% of talent on Upwork in, Marketing, Design, AI, and more, all ready to jump in and take work off your plate. Upward Business Plus can take the hassle out of hiring and the pressure off your team. That way you can stop doing everything and instead focus on scaling while the pros at Upwork can handle the rest. Right now, when you spend $1,000 on Upwork Business Plus, you get $500 in credit. Go to upwork.com slash save now and claim the offer before December 31st, 2025. Again, that's upwork.com

slash save. Scale smarter with top talent and $500 in credit. Terms and conditions apply. So here's an idea. How about we make food? from thin air. I mean, isn't that what plants do? They take carbon dioxide from the air and combine it with a few things from the soil to turn into food for us. And surely we are smarter than plants, right? Well, actually, we're not. We still haven't figured out exactly how to recreate what plants do every day. But recently, scientists have cracked a couple of their tricks. We still can't make an apple from scratch, but a couple of pioneering researchers have managed to make a yellowish protein powder and a butter-like fat, using carbon dioxide. Yes, the same stuff that's currently building up in our atmosphere. How do they do it?

And what does the final result taste like? Could this be the answer to both our warming climate and feeding the world? Listen to Gastropod, wherever you get your podcasts and find out. Speaking of competitive advantage, you said the Trump administration attacks on basic science will give China a leg up. This is basic science. You spent most of your career in academia. A number of your former students are now research heads at U.S. How quickly do you think the U.S. could lose its intellectual advantage and why wouldn't corporate R&D be able to balance the skills?

So I still believe that radical new ideas, probably the best source of those, is graduate students in good programs at top universities. So graduate students with advisors who know the field, so they don't waste their time doing things that have been done already. and other graduate students around them with heads full of ideas and lots of ambition and good resources, that's where a lot of good ideas come from. Of course they also come from companies, but radical new ideas, I think,

are more likely to come out of the best universities still. Now, the time scale is sort of five years to have the idea and write your thesis, and then another five years before that affects the world, and maybe it takes longer than that. So you're talking about a time scale that's longer than the time scale of elections, so politicians don't give a shit. How much of an impact will these cuts have from your perspective on these graduate students? Well, already there's going to be less of them, right?

Already Chinese have far more well-educated graduate students doing AI than America, I believe. And the ones here are leaving for other countries too at the same time. Right. And many of our best students, both in Canada and in the US, are from abroad. So I think making it difficult for foreign students to come to the US by, for example, charging them lots of money or taking a year for them to get a visa and things like that. Or arresting them. Or arresting them. It's crazy. It won't really have a big impact for five to ten years, I didn't think. But in five to ten years' time, when China's way ahead on research, on basic research, it'll be too late to do much about it. I would agree.

In September, you and more than 300 international thought leaders, including more than a dozen Nobel Prize and Turing Award winners, signed the global call for AI red lines, demanding an international framework for AI by the end of 2026. Explain what you mean by red lines and give some examples. So probably the easiest thing where you might actually get collaboration would be on. things that can advise you on how to create a new virus. It would be very good to check for these chatpots quite extensively whether they're safe in that respect and to have an international agreement

that nobody is going to release a chat bottle that will do that. That would be a very simple red light that we might actually get because the country's interests are aligned there. I think part of it, point of that declaration was political, say we need this. whether we'll actually get us I'm much more dubious about.

So here's a second expert question we got, and it's sort of in that vein. Hi, Dr. Hinton. I'm Alex Stamos. I'm the CSO of Corridor and a lecturer in computer science at Stanford University. I'm asking a question from Rome, where I'm here to attend a conference on AI and child safety hosted at the Vatican. My question is about the letter you just signed calling for a moratorium on the development of superintelligence. I'm wondering how effective you think this will be, since we know that the knowledge of how to develop AI is widely distributed, and we've seen that the controls around the hardware to train AI have not been effective. Isn't it possible that moratoriums on developing AI mean that only countries and labs

that don't care about AI ethics will be the ones to then pursue AI? I think it's a very sensible question. I thought long and hard about whether I should sign that moratorium, precisely for that reason. I signed it because I think it'll have a political effect and I really think that humanity will be very ill-advised to allow anybody to develop super-intelligence until we have some understanding of whether we can do it safely.

If we know we can't do it safely, we should stop. And maybe if that knowledge is widely percolated to the public, we would be able to stop. So I see my mission is educating the public about the risks and signing that petition was sort of part of that. The public needs to understand that there is this existential threat and we'd be crazy to develop AI

with this threat unsolved. The Pope is actually quite... I don't know if you know this, but Mark Indreason tried to make fun of the Pope recently and he got ratioed really badly because I think this Pope is quite intelligent on these issues. Yeah, I think the Pope does care about eye safety.

Unfortunately, he has a bunch of beliefs that make it hard for him to be rational about it. like he seems to believe that three equals one. It also seems odd to have a meeting on child safety at the Vatican, but I won't comment further on that. Let's not comment further on that. Anyway, let's talk about the potential AI bubble. Amazon alphabet, Meta, and Microsoft's valuations are through the roof.

Together, they're spending an unprecedented $400 billion on AI this year. They'll be upping those investments next year. OpenA has announced a total of $1 trillion in infrastructure, deals, that's energy, buildings, etc. Do you think these companies are overreaching? Is it all FOMO? It looks like investors are starting to worry at the same time. I just saw a BID factory in China that's as big as San Francisco, right? Like it's this enormous facility and it's largely automated and AI is a critical part of that. So talk a little bit about this spending and how you look at it. If I knew the answer, I would know whether my daughter should sell the NVIDIA

shares I get. Just a little bit of them. Take some profits. I don't know the answer. Some people, if you think AI is sort of fraud or hype or is overhyped, then I'd be very, very worried about a bubble.

I'm confident that it's not. I mean, AI actually works. I think the smart people think, yes, there may be a big problem coming down the road, but it's not here yet. One of them will do well, right? But this is really an area where I don't know enough to make sensible speculation.

Are you invested in these companies? My daughter has Nvidia shares. I have left over Google shares. That's the good one I hate. I don't think that's the one who's going to win. That's my feeling. Like I think there'll be two or three. That's one or two, maybe even.

And everybody else will get run over. It's not like the internet. It's not the same thing. I don't know. Maybe I'm wrong. Again, I don't even know that. Yeah. So Open AI CEO, Sam Alton, had a rollback a statement from his CFO last week that it might be looking the government to backstop AI infrastructure investments. And then it came out that Open AI had asked exactly that in a broader petition for the government's support of AI. Trump's AI advisor David Sachs posted that the government was interested in build out, not bailout. But whether or not the government invest in AI, which it has in the past, let's be fair, is there a danger of an innovator's dilemma here that these companies won't invest in research into new or less compute-heavy models? And then China or some other competitor will come along with the deep seek or energy innovation or anything else.

I guess I have a fairly cynical view, which is if you think the government might actually give subsidies to hugely rich companies where the people are making huge amounts of money and hardly paying any tax and they still want to get subsidies, why not ask for them? Yeah, you're right. Of course they will. Do you think the government still should be spending more on AI? I think the government should be spending a lot on AI to support academic research and start-ups, and also the government should be forcing the big companies to spend a lot more on safety research. Right now, I suspect the amount spent on safety research compared with just making the thing smarter is a few percent. I would guess, yeah. Companies like Anthropic probably spend a bit more, companies like Mechler a bit less. it should be like 30%.

I mean, this stuff might wipe us out. And even if it doesn't wipe us out, there's all sorts of very bad things it might do in the near term. I believe the government should be forcing the companies to spend more. Are there frontier models that you think could be candidates to overtake these compute-heavy models? And what do you think about open source versus these closed-source models? So if you look at why doesn't every country have nuclear weapons? One reason is you can't just go out and buy fizzarmament.

material. That's the sort of real bottleneck. There's lots of other bottlenecks too. You know, you need the missile and you need to turn it into a bomb. But getting your hands on the physical material is the most difficult thing. And so they restricted that very wisely. Now, what's the equivalent for this? Well, it's getting your hands on the weights of a foundation model. Once you've got your hands on the weights of a foundation model, you can do all sorts of things. You can fine tune that model to do things it wasn't meant to do. You can also do. distillation where you give an example to the foundation model, give it a prompt, for example, you look to see the probabilities it gives to all the various words it might say next. They're actually word fragments, but let's say words. And then you have another model, like Deep Seek did, a smaller model, and you say, see if you can match those probabilities. And that's a much more efficient way of learning than learning from the raw data. But you can't see all those probabilities.

if you don't have the model. The original. If you're just using the chatbot on the web, it will make a prediction, but you don't get to see all these probabilities. Right. Now, I believe it's a huge mistake to release the weights. I believe that's a gift to cyber criminals

and terrorists and all sorts of people and other countries. Meta was the first to do that as far as I know. Well, it was looking for advantage, presumably, but go ahead. Well, no, I think actually they probably intended to release it to academics and accidentally released it to everybody, and then made a feature of it.

That's just a guess, though. But I think it's stupid reducing the weights because it makes people able to retrain them to do other things. Now, it does give you a competitive advantage, and so if you don't care about how easy it is for cybercriminals to misuse it, then you're going to release the weights so you get this competitive advantage, because then other people will build on it. That sounds like Mark Zuckerberg.

Exactly. You just described him. You mentioned earlier an idea that giving AI a maternal instinct would protect us. But how would you go about doing that with AI? Like, let me postulate something to you. The reason I think so many men are interested in AI is because they can't have children. And this is their version of having children, right? This is them creating a baby.

Like, I know it sounds crazy, but it feels. That argument was used in the Lighthill report, which was used to close down AI in Britain. Oh. In the early 70s, James Lighthill, who was a. a brilliant mathematician, used that argument. I didn't know that. I don't buy that argument. I mean, there may be some element of truth to it, but I don't think it's a really helpful argument.

I don't think that's the main reason people are doing it. But talk about maternal instinct, the idea, because you're using this idea of wanting, this is not something that's natural to many of the AI researchers, the idea that this should be a mother. Father is... No, it's very unnatural. All the high-tech CEOs want to be the boss. and they think of these super-intelligent AI as a highly intelligent

executive assistant, probably female, who will do what they tell it. So Yan, for example, thinks, well, we're controlling the building of them so we can just make them submissive. He uses that word. So we'll be the boss. It'll be smarter than us,

but it'll be submissive. Why would they? I don't think that's going to work when they're smarter than us and can create their own sub-goals. I think they're going to have sub-goals of staying alive and getting more control. and if we're not careful, they're going to have a subgoal of just getting us out the way.

Last question. The scenario you're painting seems dire to some people in tech. Governments aren't really regulating. I tech billionaires are in a race for AI dominance. You say could lead to our destruction. But people don't want to feel powerless in this. And there is a groundswell. Everywhere I go, regular people who don't do this as a living feel a problem. And you could feel it from them.

It's fear. I think they do see the opportunity and at the same time the worry is very heavy among the populace across the world by the way everywhere I go normal people have a much better sense

of the problems here than the tech people who live in a you know everything is up and to the right kind of gang of people so what can we as individuals do to take back control so if you look at climate change the fact that the public in general understands that burning carbon is creating climate change

and that's doing a lot of bad things is having an effect on politicians and what politicians do I mean the Biden administration puts significant work into dealing with climate change doing something about it presumably because of public pressure I think people can try and understand what AI is and how it works and pressure their politicians to do something about regulating it, I think they have to understand that each of the different risks has a different solution. So, for example, the existential threat, people should be pushing for research institutes in each country that collaborate with each other on how to build super-intelligent AI that doesn't want to take over and share results with other countries

without sharing the most intelligent AI that country has. They're not going to do that. because of other reasons, like cyber attacks and autonomous weapons, and probably the techniques for making it not want to take over are more or less separate from the techniques for making it more intelligent. So they could share that information. So I think the public should push for that. And I think quite a few governments in what I think of as the middle countries, Canada, Britain, France, South Korea, Singapore, Japan, quite a few of those people would be sympathetic

to that idea. Quite a few of those governments. They should push for better ways of authenticating videos, particularly political videos. And I don't think it can be done by having AI recognize whether they're fake. Because if AI could recognize it, you could use that AI to train something that would generate stuff. He couldn't recognize, and then you get better fakes. Someone from runway AI said, what we have to do is label the real stuff. Stop trying to label the AI.

We need to have provenance. and we need to somehow be able to say it's real. So Jan Tallinn, I was once on a private jet with Jan Tallinn, and he came up with a very sensible scheme, some variation of which I think should be used. So every political advertisement should start with a QR code. Your browser looks at the QR code. Your browser goes to a website.

Your browser checks if that really is the website of the campaign, because websites are unique. and if this identical video is on that website in that case your browser can say this is genuine and if any of that fails your browser can say this is probably fake that would be very helpful to know the real thing so you could be told things very I recently got sent a YouTube video

that was me with my voice it looked just like me the voice wasn't quite mine the accent was mine The prosody was a bit alpha-zero-ish, and it was saying all the things I believe. It had a good model of what I believe, except for one thing, which is that in addition to saying all the things I believed, it was pointing out how much better organized research was in China than in the US. So you look like a softie for China. Yes. Now, I don't know who made it.

Either the Chinese made it or someone made it to make me look like a stooge for China. I got YouTube to take it down, but it was, you know, it took me a moment to make sure it wasn't me, and other people could easily be taken in by it. I think it's important we have ways of checking that that's nonsense. That's a fantastic way to end this. Can I ask you one more quick question? Do you like being called the Godfather of AI? I do quite like it.

It wasn't intended kindly, but I got introduced recently in Las Vegas as the Godfather, which I liked. Oh, and you like that a lot. Yeah, but you didn't whack anyone, right? You didn't whack anybody. No, but I think they understand. I might one day ask them for a favor. A favor they can't refuse. Okay, Dr. Hidden, thank you so much.

I'm a huge admirer. I really truly am. I love a thoughtful scientist, whether I agree or disagree with them, and it's a real pleasure to talk to you. Well, thank you very much for giving me the opportunity. Today's show was produced by Christian Castor Roussel, Kateri Yocum, Michelle Alloy, Megan Bernie, and Kalyn Lynch. Nishot Kerwa is Vox Media's executive producer of podcasts.

Special thanks to Rosemary Ho. Our engineers are Fernando Aruda and Rick Kwan, and our theme music is by Trackademics. If you're already following the show, you might ask someone for a favor, they can't refuse. If not, stop looking in your rearview mirror while you're driving. Go wherever you listen to a podcast, search for On with Carous Swisher and hit follow.

Thanks for listening to On With Carouser. Swisher from Podium Media, New York Magazine, the Vox Media Podcast Network, and us. We'll be back on Monday with more.