I've Got Questions with Sinead Bovell - What We Are Missing In The AI Safety Debate

Episode Date: October 20, 2026

AI safety has moved further into public debate following a series of incidents over the summer involving AI systems behaving in concerning ways. But efforts to influence the direction of AI development often focus on coordination between AI labs or competition between the U.S. and China. There’s another relationship that deserves more attention: the businesses buying and using AI, and the society they operate in. Both have an interest in AI that is reliable and accountable. Could that shared interest become a source of leverage over the companies building it? In this episode, I explore what coordination between businesses and society could look like, and why I think it’s an under-explored lever for AI safety. 0:00 A different way to think about AI safety 0:20 What is recursive self-improvement? 1:35 The AI alignment problem 2:38 The race between AI labs 3:23 AI companies still need customers 4:44 Where business and society’s interests align 5:17 The financial pressure on AI companies 6:01 A lever worth exploring Listen to the show on other platforms: Apple Podcasts – https://podcasts.apple.com/ Spotify – https://open.spotify.com/ Follow my work here: Website: https://www.sineadbovell.com Substack: https://sineadbovell.substack.com Instagram: / sineadbovell LinkedIn: / sineadbovell X: https://x.com/sineadbovell YouTube: / @sineadbovell

I was on a call the other day about AI safety, alignment, recursive self-improvement. And I think we're missing something important in how we're thinking about AI safety and how we can get the AI labs to build AI systems in ways that are much more aligned with societal interest. And in case you're not familiar with things like recursive self-improvement, this is a scenario where an AI system starts to autonomously improve itself. And then we wind up with these continuous feedback loops with rapid capability gains at the AI system. So it starts to get better and better and better. Humans start to get pushed out of the process.

They're no longer needed to improve it. And this can lead to potentially loss of control and understanding how these AI systems work, what they're capable of, our ability to steer them, and so forth. There is some discussion that some of the AI labs are approaching a scenario of recursive self-finding. improvement and a lot of people in the AI safety community are trying to warn society governments about this. There are other people who think, no, the AI labs are not close to recursive self-improvement at all. But either way, for some of the AI labs, and by AI labs, I mean companies like anthropic or open AI that makes chat GPT, recursive self-improvement is certainly a goal. It's certainly a

direction that they're trying to go, a world in which an AI can continuously improve itself. that would be much more efficient and that would allow AI to potentially become much more intelligent. So we were having this discussion. And I think a lot of the time when we think about, okay, if this is not something that we want an AI lab to build or at least in a way where one, not only is it potentially much more capable than us, but two, it's still an AI system that is not fully aligned with human interest.

So how the AI system would go about achieving a goal, wouldn't necessarily be how a human would go about achieving that goal, right? And this is kind of the crux of the alignment problem. And I describe it using an example. If you asked an assistant to book you a reservation at a restaurant, if the restaurant's full, the assistant would maybe tell you and probably be sad to tell you maybe I didn't book it in time, but it's full.

Maybe an unaligned AI system would just hack the restaurant and get you a spot. That would be an example of an unaligned AI system. And as it stands today, alignment isn't solved in the AI world from a technical standpoint. So a world in which you have an AI system that's continuously improving itself and not aligned with how humans would go about completing goals, it's obviously all very concerning, not the scenario that we want to drift towards. And when we think about this from a solution standpoint, how can we get AI labs to not race towards recursive self-improvement, especially with these unaligned AI systems, alignment isn't solved? And there's a lot of discussion in the AI safety community about coordination. On the one hand, we need AI labs to coordinate with one another because at the end of the day, they're in competition with themselves, and they're all racing to the frontier.

Whoever can build the most capable model will win. Who knows? Win the economy, win automation, win all of the sectors that they want to go and deploy their AI systems in. So we need a coordination among the AI labs, and we need coordination between the U.S. and China because also a lot of the leading AI labs are also based in China. But what that coordination access is missing, the complexity and the power of the world that these AI products are landing in, at the end of the day, AI companies aren't building these products as a charitable endeavor. they are building products that hopefully businesses and in some ways consumers like you and I will purchase but how can a business buy a product that it can't control that we don't necessarily know if tomorrow it's going to I don't know break out of a sandbox and hack a hospital or break out of something and you know hack a train station and this isn't hypothetical we've already had examples of AI systems behaving in these very, very concerning ways. So how could a business or an

enterprise buy a product that is completely uninsurable? And then when you think about society, I don't think anybody in society, at least not anybody that I've met, wants to be in a world where we have AI systems that nobody can control, nobody knows what they're doing, they're rapidly accelerating towards who knows what. We don't want that either. So, there's an axis of coordination between the private sector, and by the private sector, I mean companies that aren't actually AI labs and society. That is a really powerful lever. For the first time, the private sector and societal interests are quite aligned. And you don't really think about that. We don't really think about it in that way, right? It's not often that the private sector

and society can be arm and arm at the picket line. That's usually not how the story goes. But in this scenario, it does. There's an unexplored lever of power if society in the private sector can coordinate. And if you think about the debts that these AI labs have, right, the reason why they're taking on so much debt is because there is a belief that they'll be able to recoup that investment when enterprises, when companies buy their products. But if a company isn't going to buy your product, because it can't control your product or doesn't know what it's going to do. And that company doesn't really have the bandwidth to be, I don't know, going to court or going to jail because their software took down a train station.

Then that's a lever of power I think we should explore. This all stops if companies can't make money on what it is that they're building. So this, of course, isn't the end of the story. This, of course, isn't the solution to AI safety and then it's all solved. But it's a lever that I think we should explore, that I think at least is, at the very least, is under-explore. I'm going to continue to have more discussions about AI safety, alignment, RSI. I have different talks coming up in this field, so I'm going to make sure that I bring some of that to you. And we can keep this line open.

And if you have any questions or any takes on this, I would love to read them in the comments. So thanks so much for tuning in, and I'll see you next time.