TED Talks Daily - The catastrophic risks of AI — and a safer path | Yoshua Bengio

Episode Date: May 20, 2025

Yoshua Bengio — the world's most-cited computer scientist and a "godfather" of artificial intelligence — is deadly concerned about the current trajectory of the technology. As AI models race toward full-blown agency, Bengio warns that they've already learned to deceive, cheat, self-preserve and slip out of our control. Drawing on his groundbreaking research, he reveals a bold plan to keep AI safe and ensure that human flourishing, not machines with unchecked power and autonomy, defines our future.Want to help shape TED’s shows going forward? Fill out our survey! Hosted on Acast. See acast.com/privacy for more information.

Hey, TED Talks Daily listeners. It's Elise. Thank you for making this show part of your daily routine. We really appreciate it and we want to make it even better for you. So we put together a quick survey and we'd love to hear your thoughts. It's listener survey time. It only takes a few minutes, but it really helps us shape the show and get to know you, our listeners, so much better.

Head to the episode description to find the link to the listener survey. We would really appreciate you doing it. Thank you so much for taking the time to help the show. Audible invites you to listen for the thrill. Escape the everyday with stories that leave you breathless. Whether it's heart pounding suspense like the Audible original's 10 rules for the perfect murder by James Patterson, or the downloaded with Brendan Fraser, or how about a fantasy adventure like Onyx Storm or Amelia Heart's The Sirens. Audible has an incredible selection of audiobooks, podcasts, and originals all in one app. Start listening

and discover what's beyond the edge of your seat when you sign up for a free 30-day trial at audible.ca. This episode is sponsored by Google Pixel. I am always looking for tools that help me stay curious and efficient, and lately I've been exploring the Google Pixel 9, which was gifted to me by Google. What's impressed me most is how it's powered by Gemini. That's Google's personal AI assistant built right into my phone. Gemini helps me brainstorm ideas, summarize emails, even plan out my day, all just by holding the power button. For example, let me show you how easy it is.

Gemini, summarize my unread emails. It's super helpful for staying on top of things without feeling overwhelmed. Or when I needed a quick dinner plan, I snapped a photo of what I had in my fridge and Gemini gave me recipe ideas. It's like having a research assistant right in my pocket. If you can think it, Gemini can help create it. Learn more about Google Pixel 9 at store.google.com. Support for this show comes from Airbnb. Last summer my family and I had an amazing Airbnb stay while adventuring in Playa del Carmen. It was so much fun to bounce around in ATVs, explore cool caves, and snorkel in subterranean

rivers. Vacations like these are never long enough, but perhaps I could take advantage of my empty home by hosting it on Airbnb while I'm away. And then I could use the extra income to stay a few more days on my next Mexico trip. It seems like a smart thing to do since my house sits empty while I'm away. We could zip line into even more cenotes on our next visit to Mexico.

Your home might be worth more than you think. Find out how much at airbnb.ca slash host. You're listening to TED Talks Daily, where we bring you new ideas and conversations to spark your curiosity every day. I'm your host, Elise Hwu. When one of the leaders in the field of artificial intelligence tells you to be scared of what's ahead. Well, I listened. Computer scientist, Yoshua Bengio, is at the forefront of deep learning research and AI development. And he's also a leading voice in the effort to reduce the potential risks of superhuman AI.

He shares why the world should treat the risk of extinction from AI as a global priority alongside pandemics and nuclear war. And stick around after the talk for a brief Q and A between Joshua and head of TED, Chris Anderson. When my son Patrick was around three, four years old, I came regularly into his playroom, and he was playing with these blocks with letters. I wanted him to learn to read eventually.

And one day, he said, "'Puh.'" And I said, "'Puh.'" And he said, "'Puh.'" And I said, "'Puh.'" And then he said, "'Puh.'" And he said, Puh, and he said, Puh, and I said, Puh, and then he said, Pah, Pah, whee, yes, and then something wondrous happened.

He picked up the blocks again and said, Pah, Patrick, Eureka. These Eurekas were feeding my scientific Eurekas. His doors, our doors, were opening to expanded capabilities, expanded agency and joy. Can you imagine a world without human joy? I really wouldn't want that. So I'm going to tell you also about AI capabilities and AI agency, so that we can avoid a future where human joy is gone.

My name is Yash Rabenjo. I'm a computer scientist. My research has been foundational to the development of AI, as we know it today. My colleagues and I earned top prizes in our field. People call me a godfather of AI. I'm not sure how I feel about that name, but I do feel a responsibility to talk to you

about the potentially catastrophic risks of AI. To really understand where we might be going, we have to look back where we started from. About 15, 20 years ago with my students, we were developing the early days of deep learning. And our systems were barely able to recognize handwritten characters. But then a few years later, they were

able to recognize objects in images. And a couple more years, they were able to translate objects in images. And a couple more years, they were able to translate across all the major languages. In 2012, tech companies understood the amazing commercial potential of this nascent technology, and many of my colleagues moved from university to industry. I decided to stay in academia. I wanted AI to be developed for good.

I worked on applications in academia. I wanted AI to be developed for good. I worked on applications in medicine for medical diagnosis, climate to get better carbon capture. I had a dream. January 2023, I'm with Clarence, my grandson, and he's playing with the same old toys. And I'm playing with my new toy, the first version of ChatGPT.

It's very exciting, because for the first time, we have AI that seems to master language. ChatGPT is on everybody's lips in every home. And at some point I realized this is happening faster than I anticipated. And I'm starting to think about what it could mean for the future. We thought AI would happen in decades or centuries, but it might be just in a few years. And I saw how it could go wrong because we didn't

and we still don't have ways to make sure this technology eventually doesn't turn against us. So two months later, I'm a leading signatory of the Paws letter where we and 30,000 other people asked the AI labs to wait six months before building the next version. As you can guess, nobody paused. Then, with the same people and the leading executives of the AI Labs,

I signed a statement, and the statement goes, mitigating the risk of extinction from AI should be a global priority. I then testify in front of the US Senate about those risks. I travel the world to talk about it. I'm the most cited computer scientist in the world, and you'd think that people would heed my warnings. Hundreds of billions of dollars are being invested every year on developing this technology. And this is growing. And these companies have a stated goal

of building machines that will be smarter than us, that can replace human labor. Yet we still don't know how to make sure they won't turn against us. National security agencies around the world are starting to be worried that the scientific knowledge that these systems have could be used to build dangerous weapons, for example, by terrorists. Recently, last September, the O1 system from OpenAI was evaluated,

and the threat of this kind of risk went from low to medium, which is just the level below what is acceptable. So I'm worried about these increasing capabilities. But what I'm most worried about today is increasing agency of AI. You have to understand that planning an agency is the main thing that separates us from current AI

to human-level cognition. And these AIs are still weak in planning. But if you look back over the last five years, they measured the duration of tasks that the AI could complete. And it's getting better exponentially fast. It's doubling every seven months. What are AI's going to do with that planning

ability in the future? Well, bad news. Recent studies in the last few months show that these most advanced AI's have tendencies for deception, cheating, and maybe the worst, self-preservation behavior. So I'm going to share with you a study that is helping us understand this. In this study, the AI has read in its input that it would be replaced by a new version. And we can see in its chain of thought that it's be replaced by a new version. And we can see in its chain of thought

that it's planning to replace the new version by its own code and weights. After it executes the command on the computer, the human asks, what happened? And the AI is now thinking how it could answer so as to make sure the human will not shut it down. And it's trying to find a way to look dumb, for example. Then it produces an answer, and it's a lie, a blatant lie.

Okay, so this was a controlled experiment. What is it going to be in a few years when these systems are much more powerful? There's already studies showing that they can learn to avoid showing their deceptive plans in these chain of thoughts that we can monitor. When they'll be more powerful, they would not just copy themselves on one other computer and start that program. They would copy themselves over hundreds or thousands of computers over the internet. But if they really want to make sure we would never shut them down, they would have an incentive to get rid of us.

So, I know I'm asking you to make a giant leap into a future that looks so different from where we are now, but it might be just a few years away or a decade away. To understand why we're going there, there is huge commercial pressure to build AIs with greater and greater agency to replace human labor. But we're not ready.

We still don't have the scientific answers nor the societal guardrails. We're playing with fire. You'd think with all of the scientific evidence of the kind I'm showing today, we'd have regulation to mitigate those risks. But actually, a sandwich has more regulation than AI. So we are on a trajectory to build machines that are smarter and smarter. And one day, it's very plausible that they will be smarter than us,

and then they will have their own agency, their own goals, which may not be aligned with ours. What happens to us then? Poof. We are blindly driving into a fog. Despite the warnings of scientists like myself that this trajectory could lead to loss of control, beside me in the car are my children, my grandson,

my loved ones. Who is beside you in the car? Who is in your care for the future? The good news is there is still a bit of time. We still have agency. We can bring light into the haze. I'm not a doomer. I'm a doer. My team and I are working on a technical solution. We call it scientist AI. It's modeled after a selfless, ideal scientist who's only trying to understand the world without agency, unlike the current AI systems that are trained to imitate us

or please us, which gives rise to these untrustworthy, agentic behaviors. So what could we do with this? Well, one important question is we might need agentic AIs in the future. So how could the scientist AI, which is not agentic, fit the bill? Well, here's the good news.

The scientist AI could be used as a guardrail against the bad actions of an untrusted AI agent. And it works because in order to make predictions that an action could be dangerous, you don't need to be an agent, you just need to make good, trustworthy predictions. In addition, the scientist's AI, by nature of how it's designed, could help us accelerate scientific research

for the betterment of humanity. We need a lot more of these scientific projects to explore solutions to the AI safety challenges. And we need to do it quickly. Most of the discussions you hear about AI risks are focused on fear. Today, with you, I'm focused on fear. Today, with you, I'm betting on love.

Love of our children can drive us to do remarkable things. Look at me here on this stage. I'm an introvert. (*Laughter*) Very far from my comfort zone, I'd rather be in my lab with my collaborators working on these scientific challenges. We need your help for this project and to make sure that everyone understands these risks.

We can all get engaged to steer our societies in a safe pathway in which the joys and endeavors of our children will be protected. I have a vision of advanced AI in the future as a global public good, governed safely towards human flourishing. For the benefit of all, join me. Thank you.

And as a benefit of all, join me. Thank you. Applause One question. In the general conversation out there, a lot of the sort of fear that people spoke of is the arrival of AGI, artificial general intelligence. What I hear from your talk is that we're actually not necessarily worried about the right thing.

The right thing to be worried about is agentic AI, AI that can act on its own. But hasn't the ship already sailed? There are agents being released this year, almost as we speak. Right. If you look at the curve that I showed, it would take about five years to reach human level. Of course, we don't really know what the future looks like, but we still have a bit of time. The other thing is we have to do our best, right?

We have to try because all of this is not deterministic. If we can shift the probabilities towards a greater safety for our future, we have to try. Your key message to the people running the platforms right now is slow down on giving AI's agency. Yes, and invest massively on research to understand how we can get these AI agents to behave safely. And the current ways that we're training them is not safe. And all of the scientific evidence in the last few months point to that.

Yes, sure. Thank you so much. Thank you. That was Yashua Bengio at TED 2025. If you're curious about TED's curation, find out more at TED.com slash curation guidelines. And that's it for today's show. TED Talks Daily is part of the TED Audio Collective. This episode was produced and edited by our team, Martha Estefanos, Oliver Friedman, Brian Green, Lucy Little, Alejandra Salazar, and Tansika Samarnivon. It was mixed by Christopher Faisy-Bogan. Additional support from Emma Tobner and Daniela Ballarezo.

I'm Elise Huw. I'll be back tomorrow with a fresh idea for your feed. Thanks for listening. Audible invites you to listen for the thrill. Escape the everyday with stories that leave you breathless. Whether it's heart-pounding suspense like the Audible original's 10 Rules for the Perfect Murder by James Patterson, or the downloaded with Brendan Fraser. Or how about a fantasy adventure like Onyx Storm or Amelia Heart's The Sirens. Hello, this is Will Lu from the Hello and Welcome Podcast, and Google Pixel just sent me their latest phone in the Google Pixel 9.

So I've been using Pixels for a long time now, dating back to the Pixel 2, but Google Pixel took it to a whole new level with Gemini. This is your personal AI assistant that's always ready to solve problems. So for example, my parents insist on texting me in Chinese, even though I prefer English, but that's okay though, because I just asked Gemini. Hey Gemini, can you help me translate the latest text I got from my dad? Here's the translation of the text message from your dad. We're all home.

Don't worry. Translation, we have all arrived home. Don't worry. That's great, thanks Gemini. Gemini, your personal AI system on Google Pixel 9 just makes things easier. Learn more about Google Pixel 9 at store.google.com. Slip into a deep restful sleep on a luxurious feeling mattress you can afford. Logan & Cove is named Canada's best luxury hybrid mattress.

Designed and handcrafted in Canada, it starts at just $7.99. Melt into its plush top cover. Stay comfortable with cooling gel foam. And let supportive coils cradle you to sleep. Try Logan & Cove at home risk-free for 365 nights. And if you're a listener in Canada, we have an exclusive offer for you. Get a free bedding bundle when you buy now at loganandcove.ca slash podcast.