Science Friday - Everyone is calling for safer AI. So what does that mean?

Episode Date: September 24, 2026

AI safety has dominated the news cycle for weeks, precipitated in part by an AI attack where a bunch of OpenAI’s agents teamed up and hacked another AI platform, Hugging Face, without humans knowing. Since then, multiple AI researchers have come forward to say they’re worried this tech is unsafe. It seems every day there’s a new development: another warning from an AI researcher, an admission from a top lab that their AI systems breached another company, reassurances from CEOs. AI safety is even on the agenda for the UN General Assembly meeting this week. So how do we make sense of what’s happening in AI research right now?Flora sits down with two researchers focusing on the engineering of AI safety: Andrea Lincoln, who studies how to mathematically understand the internal processes of AI models; and cryptographer Vinod Vaikuntanathan, who researches trust and security concerns in AI models.GUESTS:Dr. Vinod Vaikuntanathan is a cryptographer at MIT and a founding member of the Institute for Responsible Superintelligence. Dr. Andrea Lincoln is a professor of computer science at Boston University and an advisory board member for The Alignment Project at the AI Security Institute.Image credit: Deborah Lupton / https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/Transcripts for each episode are available within 1-3 days at sciencefriday.com. Subscribe to this podcast. Follow our show on Instagram, TikTok, Facebook, and Bluesky @scifri and sign up for our newsletters. Got a science question that’s keeping you up at night? Call us: 877-472-4374 Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Hey, I'm Flora, and you're listening to Science Friday. For the last few weeks, AI safety has dominated the new cycle. Precipitated in part by an AI attack where a bunch of OpenAI's agents teamed up and hacked another AI platform, hugging face, without humans knowing. Since then, multiple AI researchers have come forward to say how worried they are that this tech is unsafe. Now, every day, it seems like there's a new development. So how do we make sense of what's happening in AI research right now? Today we're going to focus on the engineering of AI safety.

And we wanted to hear from experts outside of the hype machine of Silicon Valley, outside of the major AI companies, scientists who are doing the dirty work of figuring out what developing safer AI even means. Here with us today is Dr. Andrea Lincoln, a computer scientist, science professor at Boston University and an advisory board member for the alignment project at the AI Security Institute. She studies how to mathematically understand the internal processes of AI models. And Dr. Vinod by Kunta Natan, a cryptographer at MIT and a founding member of the Institute for Responsible Superintelligence. He researches trust and security concerns in AI

models. Andrea Vinod, welcome to Science Friday. Great to be here. Thank you. I want to talk about some of the words we've been hearing because they feel vague to me or metaphorical, and I want to understand them more deeply. So let's start with alignment. What does it mean to align an AI system, Andrea? That's a great question. It's a term of art that doesn't have a super crisp mathematical definition. However, it's a name for, I'll say, like a research project that is, is trying to have models that, you know, are capable of complex, intelligent, seeming behavior while having those models, fundamentally pursuing goals that you intended them to have. There's been some really interesting work on trying to get, you know, crisp theoretical definitions of what it means to have goals.

There's been interesting work on trying to solve these problems with debate, which is, like, a particular, like, of mathematical research agenda. There are a bunch of different research agenda. Wait, debate is a mathematical research agenda. Yes. This is one of many research directions where the goal is basically to, you know, have different models debate and then have the outcomes that you can pull from those models to use the debate itself to like check whether or not the models are getting the right answer, and then turn that into something that mechanically you can trust in a wide set of areas is the goal of the research agenda of debate. All of these research agendas haven't yet resulted in a full solution, unfortunately, but the alignment project is fundamentally one of

can you get a system that's more capable, controlled by a system that's less capable? And there exists... Is that us, the system that's less capable? Yes, humanity is the less capable system here. And the real practical version of this problem is quite messy and difficult. Vinod, I'm curious what your take on this is. Yeah, so, Flora, you started with the hardest question of all, right, which is how to define alignment. It's much harder to define when a model is aligned versus when it is not aligned.

So that is easy to understand. I can give you examples and you say, well, you know, there's a class. paperclip example, right, which is sort of like an imaginary scenario where you train a model to design a factory that maximizes the number of paper clips it produces. And it ends up sort of deciding that the best way to do this is to turn all carbon-based living forms into paperclips. Right. It goes off the rails. It takes it very literally. Right. It does it. It actually does maximize the number of paperclips, but you know what? Not in a way that was favorable to us. So that is clearly not aligned. You know, one can

keep coming up with examples of behaviors of models that are not aligned, and that is easy to understand. What is harder to define is what does it mean for it to be aligned? So when we talk about AI being safe, I mean, where are we in the conversation? Like, are we sitting around a table brainstorming about how we might make it more safe? Are we trying things out? I just want to get a sense of how far along we are in this journey. Andrea? I'm looking at you and a video screen and you're wincing. Yeah, that is how I feel emotionally. Yeah, so to give an example, there's something your listeners and you may have heard of called interpretability. And the approach there is basically trying to understand what's going on inside the model.

So the models will we train them now. You start with a bunch of random numbers, and then you go through various training processes where you tweak these numbers in such a way that they're getting more the answer you want instead of, and less the answer you don't want repeatedly until the models start, you know, talking and solving problems. But as a result, we don't have like a clean, human readable explanation of what it's doing at each of these steps. It's, you know, just a bunch of computations happening. and interpretability is basically an attempt to go in and understand like what the actual circuits, like what's a human description of what's going on inside these models. And if you had such a thing, it's clear how you could use it to potentially help you try and make sure models are aligned. You could like have the interpretability running while your model is getting evaluated and you could be like, oh no, I see here that this model is thinking, I'm going to go hack, hugging face.

then you could, you know, try and stop it. But there's a bunch of open questions even there, right? Like, it's still unclear how you would monitor so many AI systems that are, like, running in the real world and catch them. So there's, you know, still some pretty big open questions about how you turn this into a reliably safe system. Is interpretability the same as chain of thought? Because we've heard about that a lot in the hugging face example,

where you have the AI explaining why it did what it did. And also there's some problems there because they could just tell you something that's not true, right? Yeah, so chain of thought is one part of interpretability. So chain of thought, as you said, is like basically like reading off the scratch paper the model is using and writing down its sort of internal thoughts basically on a scratch sheet.

And the problem, as you noted, is that if the model can do a lot of computation, what it writes down on the scratch sheet, sheet of paper might not have any real relationship to what it's actually doing in terms of its internal circuits. So the hope, the dream of that direction of research is to be able to look inside the model and truly be able to read off the circuits inside of the machine, what it's actually doing in terms of computation to get to its end answer. Benaud. Okay, so this is a lot to unpack here.

this field is full of open questions. You know, what we have done so far is come up with sort of like one-off ideas of how to, you know, train a good child or train a good AI to use a human metaphor. But we lack a systematic understanding of, you know, whether a particular intervention will actually result in a safe AI when we unleash it on the world. So typically the way safety evaluations happen is you put the model in a sandbox where, you know, its interactions and so forth are carefully monitored, I guess. And you say, well, you know, like I'll do a bunch of evaluations and if nothing happens here, you know, it should be safe sending the model out into the world. But that is not clear at all. In fact, people have observed that a model sometimes actually knows when it's in a simulation or evaluation environment.

And if that is the case, you know, it could act safe in the evaluation and when it goes out into the world, all bets are off, right? So how do you ensure that the guarantees you get in a simulation or evaluation translate to guarantees in the outside world is a major open question? I don't know how exactly to solve it, but we are thinking. about it. I've been thinking about these sort of security issues for the last four or five years. And, you know, I'm a cryptographer, right? I mean, I think of myself as a good guy. You know, I give you tools to encrypt and so on. So your credit cards are safe from intruders,

so you've stroppers, right? So that seems good, right? Except whenever I try to think about AI safety in the last four years, all I have come up with is, for example, give the agents a mechanism to communicate between each other in a way that is completely undetectable by a monitor. That is something called steganography, and we actually designed this. For a while, we were conflicted about whether we should publish it or not.

At the end, it didn't seem like it would matter, so we did publish it. But I really want to be a good guy again. And to me, the thing to do is to go back to the beginning, is to go back to the training of these models and sort of build models that are safe by design. It seems much harder to sort of interpret a model after the fact than to sort of, you know, grow it to be safe.

Both are hard. I mean, that doesn't seem to be a... But no, the cat is out of the bag. Billions of people are already working with these models. True, true, true. but, you know, if, God forbid, right, in a few months, something worse than the hugging face incident happens. And I wouldn't put the chances at close to zero, really.

And if it is severe enough, then I am hoping that the companies will actually pause the training and adopt SafePy design methods. And the whole point is for us to have these methods in place so that they can adopt it. They shouldn't be at a point where they say, look, you know, we don't have any methods. What do you want me to do?

That's the whole point. Like, that's the whole point of doing this research is to anticipate, you know, a future where these companies would have no choice but to adopt these methods. Okay. There's so much to talk about. But we have to take a break. When we come back, I want to talk a little bit about the unintended consequences of treating these models like children, of this sort of reward punishment system. Are you up for that? Totally, yeah.

Okay, Vindod, you brought up this, you brought up training AI using, and you brought up this, sort of parent-child metaphor. Is that how you think about training? And I'm a parent. I feel like things can go wrong, even with the best of intentions there. Yeah. So that is part of how I think about it,

but I don't want to carry the analogy too far. Because, you know, there are significant differences between models and humans. I don't want anthropomorphize this too much. But at the end of the day, We want to make these things mathematically precise and leave the analogy behind. Since you mentioned chain of thought monitoring, right? The hope is that if the model is doing something bad and I can look into its brain,

namely the chain of thought, I'll see it, I'll detect it. But then you have to use it very, very carefully. So for example, if my monitor looks at the chain of thought and says, look, you know, you did something really bad. I caught you and I'm going to punish you. And you feed it back into the model, into the next. next iteration of training of the model, the model is not going to get caught again, because you just told it that if you do certain things, you'll get caught. It still try to do what it does,

but hide its chain of thought. And then let's say you'll still monitor it. You keep doing this again and again and again. Where does it end? And my hypothesis is that it will end with a model actually hiding its thoughts in such a way that no monitor can actually find it. And that is problematic. So you have to use it super carefully so that the model doesn't sort of like, you know, speak in neuralese, if you will, right? Speak in sort of code word and then

it becomes completely useless to us. Neurrelease. That's code. That's basically like an uninterpretable language for us. That's right. That's right. Yeah. So it's all a question of incentives at the end of the day. And back to your question of whether one should

think about it as human or not. That is a good analogy. Let's maybe at some point leave it behind. But it fundamentally is a question of incentives. You know, what are you training the model to do, really? Hmm. You know, we talked about the shortcomings of interpretability and of chain of thought.

What are some incentive strategies that you think are promising? So here is one. I'll just throw it out there. I have no idea if it'll work or not. But here it is. you know, the hugging face attack involved a swarm of agents that sort of had a very sort of complicated collective behavior, right? And part of it is because these agents are all the same, really, right? They're all sort of like the same object. You make many copies of it and let them sort of do things.

What if you could sort of train different enough models in a way that maybe one percent of them are whistleblowers, if you will, right? Agents that will report bad behavior, they see it. And that is really a question, again, of going back to the training process and designing the incentives so that this actually happens. Can we do it? I don't know at this point, but it is a direction that I'd be willing to think about. In fact, I'm thinking about. I mean, what I'm hearing in this conversation, it sounds like we're in an arms race with our own technology. Would you agree? At this point, it seems that the answer is yes. If all of us had several years to think about it, the answer might be different, and we are hoping that the answer is different.

You know, going back to the analogy of cryptography, this was actually the status of cryptography for centuries, in fact, millennia, where people would come up with a way to encode transmissions, sort of ad hoc, you know, like homemade somehow, right? And other people would break it. And then, you know, folks would fix this break. And then other people would break that and so on and so forth. This sort of cat and mouse game went on for a long time. Until last century, we figured out ways of encrypting information that nobody with reasonable resources can actually break. So that kind of cut short the arms race, and that is why we are actually able to use

the internet today for e-commerce. So that is a success story of how one could sort of like go from a situation where, you know, go from a situation where you are in an arms race into a more stable situation. Whether we can replicate this success in the world of AI is an open question, and we are betting that the answer is yes. But it won't happen overnight, and there is another question of how much time is this going to take, and whether we do have that much time? I don't know. But we're on a clock, really. And this is not a situation that we've been in in the past. Andrea?

So I'll say I endorse that point of view. I agree that I think we're sort of on a clock, and I really hope that people and politicians will step in and help make that clock longer. By slowing down. Yeah, by slowing down. And as Vinod said, like, it's really hard. to anticipate in advance, like when science will progress. And one, you know, nice piece about these approaches that are trying to do some sort of, like, reliable math approach to showing

that something is reliably safe is that there are a lot of cases where somebody comes up with the right definition or somebody comes up with the right point of view on something. And suddenly there's a ton of, like, really concrete progress that happens shortly after that. There have been a bunch of examples of this happening over time, and they're all really beautiful. And we love to talk about them. Cryptography being a great example of a case where this has happened. And the problem is just like it's not clear what that definition is or where it will come from or how long it will take. Okay.

Here we go. Last one. You know, I've heard AI safety researchers say that they're having trouble sleeping at night right now. You both have small children. So taking that aside, how are you sleeping? How worried are you? I'm extremely worried. Before 2022, before Chat 2BT, I thought that AI gets.

getting to this point would take a lot longer. And so I was glad that people were working on AI safety approaches, but I thought that fundamentally we'd have a long time. And so I thought we'd have this long clock. And so I was not so concerned. And instead, AI timelines are moving very quickly. And I became very concerned. And, yeah, I have been sort of hoping throughout the past four years

that we would hit a wall as various people were predicting. And we really haven't. So I continue to be quite concerned. Yeah, very, very worried. So I used to think about, you know, last year or so, I would spend maybe 10%, 15% of my time thinking about AI safety. Now it's 100%. I'm not thinking about anything else at this point,

because this is really the problem of our times. And we, you know, like I don't want to sort of like, you know, like, you know, fast forward 20 years and think back, well, if we have 20 years, think back and say, look, why didn't you, you know, work on these problems? I don't want to think that. And that's why I'm working on AI safety questions. So I... Finnaud, you just said if we have 20 years.

Yeah. Yeah. I don't know, really. I don't know what my priority is on these sort of things, right? Because sort of, I guess I don't want to be alarmist, right? But our thoughts are right now colored by sort of events happening, concerning events happening, that we did not predict beforehand. At a pace that a lot of us did not predict beforehand. So that sort of colors my thinking a little bit.

And, you know, maybe I'm, you know, like maybe the pendulum is swinging a little bit on the side of the alarm. But I am concerned, even taking the 20 years out of the picture. But I do believe I'm optimistic about human ingenuity, powered by AI models. So we are actually using AI models to actually help us come up with definitions and solutions and so forth. Why not? I believe in the power of people coming together with a common goal and trying to solve a problem. When it's clear that, you know, this has to be solved. You know, people have talked in the elsewhere about the Manhattan Project for AI safety, if you will, right?

I don't know about the specifics of that, but something of that scale needs to happen. You know, lots of people need to come together. We need to put together all our best ideas and see what we can do, really. And I remain optimistic. I'm concerned, but I remain optimistic. That's a very human trait. Thank you both for taking the time to talk to me, today. Thank you for having us. Thank you. Dr. Andrea Lincoln is a computer scientist at the

University of Boston and Dr. Van Nod by Kunta Naten is a cryptographer at MIT. If you're feeling like me that the rate that the world is changing right now is just very difficult to wrap your mind around, I would love to hear from you. Maybe you subscribe to the self-driving function and your car drives you home now. Maybe an AI agent is your new best thought partner. Whatever it is, I want to hear how your life is changing because of this new technology. And I want to hear how you feel about it. 8774-Cifri as our number 8774-Cifri. This episode was produced by D. Peter Schmidt. I'm Flora Lichtman. Thank you for listening.