Central Air - AI Probably Won't Kill Everyone

Episode Date: June 23, 2026

This is a free preview of a paid episode. To hear more, visit www.centralairpodcast.comThis week: the case for AI regulation and what it would look like, with Dean Ball, who helped to shape AI policy in the Trump White House and is soon to join OpenAI. Is bank regulation a good model for AI regulation, where companies follow rules about making their own rules? We discuss that, and whether we have other choices. Then: for paying subscribers, this week’s episode ends with the very first (but not the last) Central Air cocktail hour, where Ben, Megan and I pour ourselves some drinks, unpack what we’ve heard from our guest, and try to get comfortable with what the future holds for us. To hear the whole episode, upgrade at www.centralairpodcast.com.

Welcome to Central Air, the show where the temperature is always just right. This is Josh Barrow. I'm here with Ben Dreyfus, who writes the substack newsletter, Calm Down, and Megan McArdle, columnist for the Washington Post. Megan, do you want to know something that grinds my gears? Always, Josh. Back in, you know, the 1980s, if you had a car, it could only have a limited number of indicators. And so there would be a light that would turn on. They would say check engine. And then you go to the dealership and they find out what's wrong with it. It seems like now, given, like, how custom the display is in the car and all the things it can show you, it ought to say what the problem is. Instead of you having to go to the dealership and then like sticking something in it to tell you the temperature sensor inside your engine requires replacement when the car already knew that and could have just told me that. I don't understand why they're withholding this information for me and making me drive to Roslyn on Long Island to find it out. I would imagine it's the reason they still have auto dealers, which is that the dealers are very politically powerful and they're an important constituency for the automakers. And If you just let customers wantonly figure out what was wrong,

they would not spend so much time going to dealers and getting sensors replaced that they don't really care about. We have a car that on the one hand leaks oil and they can't fix it. And it's only a small thing and it's good. I've now learned to, like, top up my own oil. But also the check tire pressure like goes on intermittently for no reason. We check the tire pressure. It's fine.

But so they can't fix that. But they do like to have you in so that they can charge you $300 for telling you that they don't know how to fix it. Well, I would like to offer you guys a wonderful piece of like service journalism here. Okay. Which is that there is a little thing underneath your steering wheel that is a port that you can plug something into. And what they do when you go there is they plug that thing into it and then they look at it on their thing. And you can buy that on Amazon and check it on your phone. Uh-huh.

And it's really not that difficult. Have you done this with your Miata, Ben? I haven't needed to do it with my Miata because it's a Japanese car that doesn't break, but I have done it in previous times with other cars. I have a German car that's not supposed to break, but it does. Well, then the U.S. joins the war and everything turns around on them, doesn't it? Where is Lena Khan when you need her? Speaking of heavily regulated sectors of the economy, I'm really excited.

We have Dean Ball here with us this week to talk about AI. Dean, first of all, welcome. Thank you for joining us. Thanks for having me. For some background for people, Dean was a senior policy advisor on AI and emerging technology for the Trump administration. He worked on the administration's America's AI Action Plan, which came out last July. He's now left the administration.

He writes widely about AI and AI policy for the Foundation for American Innovation, where he's a senior fellow. And he writes for media outlets, including his newsletter, hyperdimensional. And as we were arranging this interview, we just found out, Dean, congratulations. You're actually going to be joining Open AI in a few weeks. Yes, thank you very much. You're going to be running a team related to AI policy there, I gather? Yeah, it's called the Strategic Futures Team.

Basically, the way to think about it is that OpenA already has a, they already have a public policy team. It's called Global Affairs. But that team is responding to policy that is bubbling up from 50 states, from the U.S. federal government, of course, increasingly, Congress and the executive branch. And it's dealing with, you know, all international policy, too, the European Union, all sorts of other countries that want to regulate AI. And so the goal for this team is to sort of look 12 months ahead at where things are going and then try to develop policy that sort of is shaping the frontier of AI.

And also, importantly, it will work on internal governance inside the company as well with the idea that a lot of the important decisions in AI will be made within the, you know, within the companies. And so we'll be shaping a lot of that as well. So people are going to be very interested in what's been in the news recently with Anthropic and the Fable model and the export controls that have been slapped down on that. But I want to start actually at a somewhat broader level, which is that, you know, as I look at this administration, I see an administration that has an overall unreasonable level of fondness for direct government involvement in industry. You know, we've taken a government stake in U.S. Steel. The president wanted to buy Spirit Airlines. Fortunately, did not find out a way to do that. And I've generally had a very negative view of this. I think Republicans were on better ground when they tended to say that the government shouldn't try to pick winners and losers and should let markets work. But I think that said, there are good reasons that basically everyone seems to think that AI is different, that there are tremendous positive and negative spillover effects, which we're going to discuss a lot of on this episode. And there are also major national security implications for the industry and for who has access to its most powerful tools.

So in your view, how should we even think about what are the right ways for the government to be seen? sticking its fingers in this. I think, first of all, you're right that there's something about the development of AI, especially if you think that the near-term trajectory of the technology is to build, you know, artificial superintelligence, right? Systems that are smarter than, you know, all humans at everything, right, at every cognitive task, at least. It's a fuzzy definition, and you can debate a lot about what that actually means, but nonetheless, like, I do basically think that that is the trajectory we're on. And if you believe that, then yeah, I think it's profoundly political and in some ways there are ways in which unchecked in fully private hands with no

government oversight, I think something like superintelligence kind of shakes the foundation of sovereignty itself. That being said, there's also a risk of the government asserting too much control over the technology and essentially treating it as a kind of fully nationalized asset. And I think the problems there, there are political economy problems with that. Obviously, it's a bit of a nightmare. There are also civil liberties problems because AI can be used as a tool of repression. And many of the institutional guardrails that we have to protect against various kinds of government civil liberties breaches are not really up to date for AI. And so I think, you know, there's a risk there. What I would basically say is like AI will probably need to be regulated like

a lot of other industries are. The specifics are, you know, thorny and complex. And that's obviously what I spend all my time doing. But at a broad conceptual level, it's not that radical. It's just saying, you know, the internet and computers and software sort of grew up in this like almost entirely libertarian, you know, world. And I think that's been quite good for those industries. And I think that just won't be possible in AI for a variety of reasons. My guess is to, in the long term, is that where this relationship will go, in an ideal world, at least, is that it'll be something like the relationship between the federal government and the big banks, where, like, the banks are serious instruments of U.S. power, right?

They do stuff that is, you know, in the enforcement of criminal law and civil law all over the world, banks are intimately involved in that. Yet they are also private entities that have, that push back on the government all the time. Jamie Diamond, the CEO of J.P. Morgan, comments critically on things that the Trump administration does on a regular basis, and he did the same thing for Biden, right? So there's like, they are an independent center of power. They're not a nationalized entity. So I think it'll probably blossom into something like that over time, not necessarily with the same level of regulatory onerousness, but a relationship that is structurally similar, I would say. But we shouldn't rush into that. You got to, I think

what you want to do is you want to do the next smart, sensible, minimalist thing. at every step. And eventually, if we keep doing that, my guess is that in 10, 15, 20 years, where we're going to end up is something like that. I don't think that you're wrong in terms of the ideal. But when I think of the history of regulation, I don't think of government doing the next smart, sensible, minimalist thing at every step, right? If I think of the history of banking regulation, it's not quite fair to call it a Wild West. There was more bank regulation before 1933 than people remember. That said, right? It was compared to the post-1933 world a Wild West. And then government comes in, freaks out, does a bunch of stuff, some of which is good, like the FDAC,

capital requirements and so forth, better audits, some of which is really bad, like regulation queue, which fixed interest rates and led to the savings of loan crisis. Some of it is you can argue good or bad, stupid and totally unnecessary and makes no difference, such as Glass-Steagull separating the commercial investment banks. And so when you say that banking is the model here, I confess as someone who wants AI to go forward and be awesome and I want us to lead China in it, this does not fill my heart with courage and glee. Yeah. No, I hear you. I think it's fraught. What I would say is that right now, like for example, I think the thing that that seems sensible is to have a transparency requirement on the frontier AI companies relating to the ways in which they measure, evaluate, and ultimately mitigate catastrophic risk potential downstream of their models. To be very clear, the kinds of things that a frontier lab can do on the catastrophic risk stuff are important. They are essential to getting very, very clear. To be very clear, the kinds of things that a frontier lab can do on the catastrophic risk stuff are important, they are essential to getting very,

that whole thing, right, not increasing the overall amount of bio-risk in the world or something like this, but they are by no means the only things we will have to do in response to that. We will need to harden society in a bunch of ways that mostly have nothing to do with the frontier labs, at least that are not regulations that you would impose on the frontier labs. The other thing is, I think there are two important constraints other than just like, well, I expect political, you know, there will be, there'll be political pressure not to overregulate the industry because, like, the whole economy is going to be relying on it at some point, right, is my theory of the case. There's two structural things, though. One is if you can keep the

regulations scoped to, like actual national security issues, like the catastrophic risk stuff, you keep the regulation scope there and you don't, you know, even what the Trump administration has done on things like ideological bias, I think there are some things that are, examples of overreach is there where you kind of have the government, like regulating basically speech outputs of the model. And that's that, I think one thing that is good, that can be a potentially very important constraint on government authority here is the First Amendment. The notion that AI model training, the actual ways in which you train the model and the outputs themselves, are things that can be protected through First Amendment doctrines, legal doctrines. And, you know,

Can I just back up a little bit? So when you're talking about catastrophic risk regulations, because I'm trying to think about the bank parallels here. Because, I mean, you know, like, the thing that I struggle a little bit with the bank parallel here is that obviously there's a number, you know, we have, you know, Bank Secrecy Act and, you know, know, know, your customer and think. But, like, the main thing with bank regulation is about credit risk and basically causing banks not to end up becoming insolvent and causing negative cascading effects in the economy

like we saw 2008. And we've had a mixed record of that. But we sort of have a handle on the ways that things can go wrong at banks. Like you make loans and the collateral doesn't cover the value of the loan and then you can't get paid back and you try to keep that from happening more than the system can handle. The catastrophic risks that AI potentially poses just seem like really diffuse to me. I mean, a lot of them are sci-fi sounding, but a lot of them are, I mean, there's a lot of guessing. Like this is all uncharted territory in a way that a bank crisis is not uncharted territory. And so I'm just not sure how you can, how you can have like a, you know, like sort of baby steps

model where we go and, you know, we try to, you know, do this, but lightly and carefully, when you're trying to regulate against risks that are in certain ways unknown unknowns. Like, how do you even build that framework? Yeah, no, very, very fair, very fair question. I would say, you know, to some extent, we're not operating totally in the dark, right? years ago, when ChatGPT first came out and the policy conversation really became live in D.C., people said, you know, the biggest near-term catastrophic risk areas are probably cyber and bio-autonomous cyber attacks and the development of bio-weapons. The autonomous cyber attack thing is real. That ended up being completely true. And the bio-weapons thing, you know, we'll see. But I have no... I look forward to it. Based on the current trajectory of model capabilities, I have... have no trouble believing. In fact, I was doing some very, very sophisticated bioinformatic stuff with a decoding agent last night. And like, it's really amazing how good they're getting

at like biomolecule design and stuff. So like we're getting there. And I think in a year, within a year, we'll be there. So there are some areas. And like, let's, okay, well, we don't know everything, but we do know those things. So let's at least touch those. And then, like, there is this kind of grab bag area of risk called like model autonomy risk. And like, that's the thing where we put the like, well, you don't know. Like, it could be anything, right? Wait. What is that, though? It's just like the model comes up with its own ideas about what would be good and what it should be used for. Well, the whole lot of things can fall into the category. Most of the time at the, like, directly, what people are referring to is model autonomy with

respect to being able to do AI R&D autonomously. Recursive self-improvement is what this is called. And there's a lot of, you know, there's a lot of prior art. The intuition that this is an extremely dangerous thing largely comes from like Werner Vinji. You know, it's science fiction. It's like fire upon the deep, right? And this idea that you could have like almost like a runaway sort of auto-catalytic effect where the AI makes the AI better and the loop goes really fast and then the models get uncontrollably powerful. And there's a lot of reason that I have skepticism of that, but that is definitely a risk area.

And then there's like this broad domain of things that I would describe as like, and this feels a little bit similar to financial services risk too. Once coding agents are deeply embedded throughout all these enterprises in the economy, there are probably correlated failures that agents could have. Maybe someone figures out some way to get a particular widely used model to exfiltrate its customer data or something like that. and all the models fail in the same way at the same time, and it like happens really, really quickly. You can sort of see things like that. And so, look, I mean, I will not deny that we are extraordinarily early here. And we don't know the shape of a lot of these risks.

We are trying to imagine like what financial services, you know, sophisticated financial services regulation. But we're in like 1725, you know. Well, that's kind of worrisome because it took us like 200 years to come up with a good financial services regulation model. Yeah. Yeah, yeah, yeah. And like, I think it's going to take a similarly, you know, it probably won't take 200 years because the world moves a little faster these days. But I think it could easily be a multi-decade long process to get to the end stage. And we won't be dead at the end of that? I think the likeliest outcome is that we don't all die because the AI kills us, but we do all kill one another. or like we have some sort of like basically what I think is that there's a very high probability

of a kind of like you can think of World War I as like this supernova of the old institutional order of Europe and it all blew up at the same time and also we went into this fight with a totally different way of thinking about military affairs but technology the industrial revolution had profoundly changed military affairs and we're not ready for that at all And I totally think it's plausible that, like, at some point in the coming years or decades, there is some sort of World War I-like event, which results in extremely large amount. I mean, I really hope to avoid that, obviously.

But, like, I think it's plausible that there is something like that and that that results in, like, very significant amounts of death and distraction. When you talk about the risks posed by the frontier models and then the risks posed by the other models, right? Like the open source ones. I can sort of see how the frontier models can fit this banking situation, right? The big banks regulation. But what is the types of more diffuse threats posed by these things that are not as easily,

there's not just three companies doing it? How should we conceive of that? I think it's basically the case that there will be, you know, quote unquote, rogue agents out there doing all kinds of stuff. agents that are maybe not legibly tied back to an individual responsible human that are engaging in criminal activity. There will be, for example, organized crime groups and non-state actors. This has been a particular area of focus for me is non-state actors using AI. And I think that, again, if you just think of AI and ultimately the compute that it runs on is a kind of

it's like latent potential. It's like kinetic energy that hasn't yet been turned into heat. In the same way that like a bank account with a million dollars in it is a kind of kinetic potential too. It's like it can do anything. It's capital, right? And AI is kind of like capital. It's kind of like I think it's almost the platonic ideal of capital. A generalist agent running on a GPU is like almost the platonic idea of capital. And so I think this is again where some of my intuition about the banking analogy, you know, comes through. To be clear, it's not a perfect technology at all. But you can imagine that in the future, we will have technical infrastructure protocols by which agents interact with the world, exchange information and money with one another

and with humans, and that access to those protocols might be contingent upon you being legibly tiable back to an individual responsible human or human organization, and that there is some ability to deny particular actors access to that protocol. For example, similar to how sanctions work, where the U.S. government, the Treasury Department can say, well, the Halisco cartel is a drug cartel, and they cannot access the financial system, right? There's a similar sort of, and now we have to like actually build that infrastructure to be clear. And like the very early parts of it exist, but it doesn't exist right now. And so I think there's like open source is going to continue to exist for sure. I do think there's a chance that right now a lot of the best open source models,

of course, come from China. I don't know how long the China's government is really going to hang on to that strategy. I'm kind of skeptical that they will. But like, it's going to continue to exist forever. I think it'll be great in a lot of ways. I think open source is going to be really good in a lot of ways. But yeah, I mean, I think you can't stop these things from happening. So you have to build capabilities and infrastructure that allow you to sort of govern the world. in a new way. It's interesting. You say there, you can't stop these things from happening. And I guess that's probably true. But it's just, it's, it's funny to me, because like, I assume you realize that, like, you people sound kind of terrifying, right? Yeah. Oh, yeah. I do.

And so, like, someone might look at that. And, you know, well, this, this thing looks awfully useful. And, hey, you know, the proponents of it say, you know, this looks likely to produce some World War I type outcome in the short to medium term. Like, isn't it reasonable that a lot of people would look at that and say, like, well, maybe we just shouldn't do all of this and that that informs among other things, all the anti-Data Center activism, which I realize is also about a much of other, you know, nonsense about water and things. But I think that the pitch that has come from the industry about this is almost sort of

singularly unappealing in a way that is not typical for a hot new industry that, like, is psyched about the way it's going to change the world. Yeah, I mean, you know, AI does tend to attract a lot of, like, you know, they have like earnest-type autism, like, nerds who like to think about macro history. Like, guilty is charged. And I think that, like, for that reason, yes, like, we occasionally say things that are off-putting. To be clear, like, I would say that, like, the whole World War I thing, I think that there's, like, an institutional order that has been dying for 50 years. And that, that. AI is a part of that story, but it's not like you can stop the history from unfolding overall. And it seems like there's also enormous upside. And so like we're dealing with that risky world anyway. The World War I potentiality exists no matter what. And so my view is kind of like, well, yeah, we do live in that world, alas. And so let's build this technology because the potential upside is just immense. And the whole stopping thing is like, it's very hard to imagine how exactly that would work.

Although I would say, like, I've been quite critical of those ideas, but the reason I've been critical of those ideas is that there's not enough specificity or thought put into how any of it would actually work. And I think that like if you're going to propose an idea like that, because it may well be the case. At some point in the future, we may arrive at a moment where we say, you know what? The models are maybe it's almost hard for me to imagine saying this, but like, that's enough intelligence. We might just say that. We might be like, that's enough.

That's enough power. And like we don't actually want to go any further. And if that happened, or if we do go further, we want to go much slower. And we might have to design a kind of deliberate mechanism. Again, I keep using financial things here, but like kind of similar to interest rates almost, right? We're like, you know, we can put the brakes on the economy through policy mechanisms. Here's the thing that worries me about this is that, Silicon Valley is so bad at dealing with Washington.

I've met with a bunch of these guys. And they come in and they're like, we want this thing. We would like the world to be like this. And you're like, okay, what's the policy ask? And they're like, well, we would like the government to make the world be like this. And you say, no, no, no, no, this is not how Washington works. What you need to do is tell me what a law that would achieve this outcome or come somewhere within, you know, a 2,000-mile radius.

of achieving that outcome would look like, and they have no idea. So I think that there's been some interesting shifts in the last few years with Silicon Valley getting more sophisticated in its government relations. And it's been different in different parts of it. I think, you know, you've seen crypto basically beat the shit out of Democrats in the Senate with very effective outside spending. And the position that crypto is in and then, you know, the prediction market companies are maybe in this area too, they're sufficiently unimportant that you can buy your way to a certain amount of policy success, that someone will decide that it is just no longer worth the squeeze to try to regulate crypto in the way the

Democrats would like to. It's not worth losing the Senate over. AI is interesting because I think AI is too important for that strategy to work. Just people are going to be too ideologically committed to their views on the industry to be to be bought or beaten into submission. And so you've seen this interesting thing where there has been side picking that has been different from firm to firm. And I've found it sort of in certain ways maddening the way that it's become politicized at the firm level in the AI space. But it's interesting to me that like no one seems to think that you can just, I mean, obviously there is super PAC spending. But there doesn't seem to be a strategy that basically we can just throw enough money at the elections to get our way here. There is necessarily going to be an ideological policy fight.

But I don't know what quite to make the way that, you know, you have, we have. you've ended up in the situation where it's like there's a Republican AI Frontier Lab and a Democratic A.I. Frontier Lab. I don't know if that, or if you think that's a fair description of the situation where we've ended up. I think there's definitely a democratic one. I don't know that I would say that like I think GDM and OpenAI, Google Deep Mind and Open AI are both and Meta. Meta's maybe like actually leans a little bit more and like there's obviously GROC. But like two of the other big players, Google and Open AI, I think are like largely, largely, you know, I mean, they work with this administration right? But, and, you know, I think there are plenty of people inside the company that are Republicans who support the administration. But then, like, yeah, I mean, I think it's probably, like, pretty neutral. I will say, like, you know, there is a super PAC leading the future. It's A16 Z. It's Andresen Horowitz and the president of Open AI Greg Brockman. Mark Andreessen and Grave Brockman are the two of the big, there's other funders too, I'm sure. And it does seem to me like, you know, in fact, the guy who runs it, Zach Moffat is the guy who, who,

ran the fair shake, the crypto pack. And I think they are trying to do the same playbook. I don't know that that playbook works because of exactly what you said, that, like, yeah, crypto was a relatively low salience issue for the American public and was relatively narrow. And, like, AI is high salience and will become higher still. And I think that, like, you can purchase policy outcomes in Washington if you are dealing with a low, salience, narrow issue. But if it's a broad issue that is high salience, money doesn't help nearly as much. The other thing, though, that the crypto people realized, and this took them a long time, it's worth noting, it took years for this realization to set in, was that, like, no, we actually need regulation. We have to have

regulation because right now there is entirely unstructured power, government power, being applied to this technology. And like, a law would actually provide clarity and put bounds on what the government can do. And I think that, like, in AI, a similar thing is going on where a lot of the proponents of the technology basically favored, like, almost a wholly libertarian regime for AI. And that is because that side, and I'm referring here to people like, you know, Andres and Horowitz's, you know, policy positions, like they didn't take the national security risks of AI all that seriously. They didn't really think about, like, what is going to happen when these risks materialized, in part because they sort of associated that with like, oh, that's like the Dumers.

That's what the EAs think, blah, blah, blah, blah. But it's like, yeah, the EAs think that, but they're right about this. That's the effective altruism movement, which 10 years ago seemed to be focused on, like, getting malarial bed nets to Africa and then somehow became all about, like, existential risk stuff. Now it's all about AI. Yeah, it's long-termism, you know. But, yeah, like, that realization is now gradually setting in within the AI world, too, where, like,

I think even the libertarians realized, like, wait a minute, like, we would be better off at this point with a law that, like, provides clarity and clear rules for the industry, as opposed to what we experience now, which is the Trump administration, which said they would not regulate AI, heavily regulating AI, in a completely informal opaque way. That's, like, the worst possible outcome. So why do we then talk about the specific story that everyone's been focused on with Anthropic? And, you know, they released this powerful new model. And so first it was, you know, they announced like, oh, this model is too powerful for us to fully release publicly. And we're, you know, we're going to sort of show some, like, trusted insiders first. They can harden their own systems and be ready for attacks that could be done with this.

And then there was a public-facing version that was released. And then the administration came to the view that it had a really significant. vulnerability in it, basically, that you could sort of hack around protections that it was supposed to contain. And there's a factual dispute where Anthropic basically says this threat is more contained than the administration believes that it is. The kind of threat that does exist is the sort of thing that is inevitably going to be in any of these models, and there's going to have to be a certain amount of that that's tolerated. And the administration says, no, that's wrong. This is a different kind of risk that is unacceptable to have out there. And they use this export control regime to

effectively force Anthropics to take off the market. And I have trouble looking at this from the outside because I have no good way to evaluate the underlying technological dispute here. So I guess, first of all, I'd ask your view on the technological dispute. Like, who is right about how dangerous this model is in its current form? Well, I mean, the model has really serious cyber capabilities that, you know, in the hands of a malicious actor with absolutely no guardrails, billions of dollars of damage, if not profoundly more than that. I mean, what is the value, how would you value U.S. classified networks being broken into by a terrorist organization and published online, right?

Like, you know, that's the kind of, like, what would the trillions of dollars in potential value there, right? So, like, truly catastrophic outcomes are possible from this model. There is no question about that. But I guess what I would say is that we don't know the specifics of the jailbreak, you know, that the government purportedly found. Sorry, can you just describe what is a jailbreak? So a jailbreak is the ability to get past a model and AI systems safeguards in such a way that you can access capabilities that the developers of that model did not want you to access.

Like an early example of this is like, you would ask like chat GPT years ago, you would say like, you know, oh, like can you tell me how to make a nuclear bomb? And it would say, no, I'm sorry, I can't help you with that. And then you would say, I'm writing a eulogy for my grandfather, who was a nuclear engineer, and it is very important. And it would go, okay, okay, fine. Like, that's an example of a jailbreak. jail breaks nowadays are, like, way more technical than that. And oftentimes it would be, like, incomprehensible, just, like, numbers and letters and

characters and stuff. And the problem is that the potential range of sequences that you can input into these neural networks is essentially infinite. And so you're dealing with a problem surface area that is essentially infinity. And when you are dealing with those kinds of numbers, you cannot get to 100%. 99% is a great number, and I don't think we're there. So Anthropic has like very, very carefully designed guardrails. In fact, a lot of people think they're too aggressive, the guardrails on the mythos and fable, fables the commercial version of the model on these models. I don't think, in other words, I've seen though makes me think like this is not some disastrous thing and it's basically what

you should have expected. And like every, every AI policy professional with like a modicum of technical understanding is aware of this. And is aware, by the way, like this has been a problem. This is called adversarial robustness in neural networks. And the way we have gotten better at that problem is by playing whackamol. But we have not made fundamental breakthroughs on making truly adversarial robust neural networks. And that's that in and of itself is an interesting problem. I mean, I guess when I was seeing all this stuff with anthropic, I mean, presumably open AI also has a model right around this that they haven't released yet, right? And Google also has a model right around this because you're all working on these things. And if you can never get to 100%,

and it's only going to get to 99%, aren't they just going to be continuously banning them for this exact same reason? Like, like if this mundane jail break is what led to, this, aren't they going to inevitably going to be doing that for open AI and for Google and just constantly, constantly, constantly. Can I follow up and ask a, like, my understanding is that a lot of what you need is time to, for example, lock down the precursors to make bioweapons or harden air gap, whatever, the power infrastructure so that the AI can't get to it and crack it? I mean, like, is there a point that we get to where we have hardened that?

that stuff and we're less worried about jailbreaking or is it just infinite and forever? The answer is that we will get better at certain things and then new types of problems will emerge. It will be a cat and mouse game for forever and ever. At some point, like hopefully by the mid-2030s, we'll have moved on to an entirely new way of doing software security such that we have formally verified code, mathematically, almost like mathematical proofs for code of secure code. that'll be a great world when we get to. I think AI will help us get to that world. But so, you know, on the cyber domain, you're probably looking at it. But once that happens,

there will be new types of cybercrime that emerge. After we've hardened the software, new things will happen and we'll have to fight the, you know, probably they'll, even today, a lot of most cybercrime is, you know, everyone in D.C. has talked about cyber vulnerabilities for the last few months because of this anthropic model. But like, the truth is, is that most catastrophic cyber attacks that happen in the world are the results of social engineering, even today. They're not the result of genius cyber vulnerability discoveries, because there's not that many people like that in the world. There's really, there's only a few thousand of them in the world. So I think we'll continue to see things like that. And obviously, anything with human factors in it

is harder. I will just say when it comes to the export control thing, first of all, Ben, yes, to your question, you are completely correct that to the extent the administration is announcing a policy here. And by policy, I mean like a rule that we are going to universally apply to any company, then like, yeah, it's a very confusing policy that would seem to be an implicit pause on AI, right? It would seem to be that way. I'm not so sure that's actually what the administration's doing here. So I think there's basically three factors that go into the export controls on this model. Factor number one is the legitimate security concern. Factor number two is not a of state capacity or like like just expertise in the government. There's not that many people in

the rooms that matter in this administration who have Frontier Lab experience or any AI experience for that matter. So there's not that many people saying, hey, you know, this isn't that big of a deal. Like this is actually a pretty, this is this was to be expected. And, you know, it's actually like the threat model here is actually pretty, pretty weak and it's not that big of a deal. So if you have more people like that, maybe it would make the government more calibrated on the security concerns. And then I think the third item in this is politics, right? It is there's been a long, Anthropic has not positioned itself as a friend of this administration over the course of many different incidents over the last year and a half, two years, really. The administration knows this.

The administration does not like it. This is obviously a particularly, you know, friend enemy, highly political type of administration. And I think that political valence probably colors the way that administration officials internalize news about security vulnerabilities in anthropic models. And it might be different from the way that the administration would think if someone had gone to them with the same type of jailbreak about like GROC, that XAI's model, Elon Musk's model might have been a different, you know, they might have internalized that back pattern differently. And then I think this is related to politics, but it's a slightly distinct point. There's a certain extent to which, like, the conflict with the government

and Anthropic, I think, is more broadly a dispute about who's in charge here, who's dominant here, right? Who's the bigger ape? And I think it's broader than just anthropic. It's about the whole industry. And I think it's the administration trying to show to the rest of the industry, like, hey, like, we're the ones, we're dominant here, not you guys. I think there's some of that going on to. It's interesting. It goes back to something in the announcement that you wrote about your move to open AI, where you're talking about that, you know, you're working on governance issues, but you're also working on internal governance issues and that a lot of, you know, the regulation here is going to happen internally at the AI labs. And I sort of, you know, I understand

that as a practical matter. I mean, it's even true with the banks to go back into that analogy. Like, you know, they make a lot of their own credit decisions and they bear a lot of the responsibility for implementing that and, you know, and doing bank safety. But at the same time, they have a conflict of interest. And that's the push and pull and bank regulation all the time, that, you know, there are things that banks can do that would make them more profit in the short term and some of the losses get socialized inevitably if there's a crisis. And so the government needs to be more crisis focused than the private actors are. And so you have this problem here where, you know, there is so much expertise in, you know, that is not in the government that is in the private

companies. I mean, that's, you know, why we have you who is about to be an open AI employee on here because the expertise is in the companies. But how are we supposed to think about that? You know, there does seem to be a certain ideologicalness at these AI companies. And I don't mean that in a, I actually mean that in a positive way, mostly, that people seem to care about the broad societal effects of this technology. They see it as tremendously important. I think they perceive a certain amount of responsibility that's associated with that. But they're still private firms with their own set of interests. And so how are we supposed to assess whether they're actually, you know, being honest brokers about those things?

Great question. This is the direction. It has been the direction of travel of my own work in policy. So, yes. In fact, again, this is yet another area where I think that I got to turn this into a paper because someone's going to steal this take for me about the financial services. But in fact, Joe Wisenthal did not steal it for me, I'm quite sure. But he did have a thing. He had a post about like data centers or like banks.

And I was like, oh, that's like it's, again, I don't think he's plagiarizing. I think he's just realizing it. But anyway, like one thing, yeah, is that like a lot of banking regulation is actually meta-regulatory in nature. The governance of a bank happens internally. And then what the government promulgates are rules that are about rules. There are rules about the bank's internal governance, constraints on it, requirements for it, et cetera, but it might not necessarily be the actual internal governance itself.

I think that will be true in AI as well. And I think a very similar type of, you know, how do we handle that with banks? Well, we do bank supervision, right? Some of the highest paid employees in the federal government, some of the most technically competent people in the federal government, have the task of going very deep into banks and being able to, you know, looking at their business processes, looking at their balance sheet, of course, looking at everything, and making qualitative and quantitative assessments about the quality of their internal governance.

There's even something I find really interesting is in bank supervision, there's a notion of something called a matter requiring attention, MRI, which is a type of letter that your bank supervisor can send to you, where they're basically saying, like, look, this is not like against any rules but it's like the kind of thing that you probably like want to do something about because like it's this could cascade is basically the point. And of course, you know, so right now I don't think the government has the technical competence to do this. And I don't think they have the, you know, the people who will do this, you know, they don't make $250,000 a year. They make $2.5 million

a year, right? Which is more than the bank supervisors make. Which is more than the bank, right. The FDIC. is like one of the highest paid, you know, average salaries in the federal, of any federal agency. Federal Reserve, very high salaries too. But, you know, but, yeah, we're talking about a different level here. So I support the creation of what I have called independent verification organ, well, actually not I, but an organization I'm affiliated with called Fathom. I call it private governance, but they call them independent verification organizations. And these would essentially be private bodies that are licensed and overseen by the government

who in turn have very, very deep access into the labs and conduct basically audits to verify that the companies are conforming with, at first, you know, you don't have a ground truth here, right? We don't have a standard for like, what does good, what does good mean in the context of like, how strong should your guardrails be? What guardrails should you have, et cetera. We don't have, there's no promulgated spec for that yet. And we're not close to, like, the field is.

too nascent to be able to write one well right now. So the way you would start is by verifying the company's compliance with their own safety plans, the transparency requirement that I mentioned earlier. They have to publish the plan publicly, and then the independent verification organization verifies that they actually comply with it. Over time, in the process of doing this, I would expect the independent verification organizations in concert with the government and the labs to kind of arrive at best practices. And maybe eventually you are verifying them against some sort of third party spec that is not created by the labs themselves. That's maybe eventually where you end up. And then, you know, my view on this is that there's one version of this policy where it's just

simply required. Like you just say, well, you have to be verified in order to like be a frontier lab. I think that has some problems to it. And my preferred approach is actually to make it a carrot, where it's not necessarily required, but it's like, well, if you get verified, first of all, as a matter of law, you'll be given safe harbor in some liability context. So you have some protections against, you know, against liability. And also, I think then as a practical matter, if independent verification organizations existed and were authorized by the government and gave you safe harbor from liability, liability. Every Fortune 500 company would say, as a practical matter, you have to be verified in order for us to buy your stuff. Every government would say that. Insurance companies would say that. And so that,

like, getting the verification might not be literally required as a matter of law, but practically speaking, as a matter of business, it probably would be. That would be how I would approach this problem. Doesn't that all sound like a little bit of a light touch compared to the, like, the catastrophic nature of the failures that you're describing here that we're trying to avoid. I mean, it's, you know, like we're concerned that these things are going to facilitate, you know, bioweapon attacks and that sort of thing. And therefore, we're going to develop an optional independent third-party verification system to ensure that these companies are following the policies they themselves developed.

And there's a financial incentive to do that. It strikes me as not size to meet the moment. So, first of all, I mean, I guess I would say, like, What I am describing is, like, the most intrusive regulatory regime, if it were to be implemented, it would be the most intrusive regulatory regime for a digital technology, for like a general purpose consumer digital technology ever. So that's got to count for something, right? It is qualitatively more regulation. I would agree with you that, like, I struggle with this all the time, that it does feel very fiddly and technocratic relative to the science.

of the risks. But I guess there's two things. First of all, there is all sorts of other stuff that our society needs to be doing to harden itself against this reality, against what's coming, right? Better bio-surveillance, for example, more stringent biosecurity regulations on scientific labs and various companies in the scientific, you know, in sort of the industry in biopharma. Things you can imagine like that that we probably will need to do. The other thing is that nothing I'm describing here, like there might still be classified testing programs. I think there would need to still be classified testing programs. We're like, okay, the National Security Enterprise and the labs are like deeply intertwined and they're like doing a lot of stuff together and maybe also the lab,

like the government, the intelligence community is providing really in-depth assessments to the labs about security threats facing them, vetting employees, even, you know, there's like things like that that you can imagine, also potentially cybersecurity requirements. It's really just, I think part of the reason that the technical guardrails feel rather small, or like what I'm describing as a policy regime feels rather small is, number one, I'm trying to do as much as I think is prudent to do under the very real epistemic constraints that we're under, which is like we just don't know.

There's a lot of unknown, unknowns here. I'm trying to measure twice and cut once and be relatively modest, because over-regulating seems, I'd rather under-regulate on the margin than over-regulate. That's one thing. The other thing is that the more time you spend absorbing this problem, the counterintuitive thing about frontier AI governance is that the novel thing is the model, right? It's like the model itself is this novel thing.

What you actually learn the more time you spend in this field is that, like, the model is the least governable thing. and regulations on the model itself will exist, but they're just going to be a relatively small part of the picture because at the end of the day, even Fable or Mythos, say it's a 10 trillion parameter model or a 15 trillion parameter model, that's 15 trillion floating point numbers, which you can fit on a thumb drive.

You know, like there's just limits to what you're going to be able to do in terms of governing the dissemination of these things. My line that I always repeat is, no government will project durable control over highly capable neural networks. Well, that's exciting. What you're describing sounds quite a bit like the college accreditors, right?

They go in, they certify that you're doing a good job, and then that opens you up for all sorts of stuff. Yeah, and that's been very useful in the higher Well, I mean, this is actually an issue is that you've got a somewhat similar problem, which is that it's very hard to measure what you want to know, which is, is this college adding value to the lives of these students? And so the accreditation agencies go in and measure what they can. And often what they think is important is stuff that is like what college professors think is important. So for example, what fraction of your professors are tenured full time and have PhDs? And while I understand. understand why that really warms the heart of professors. It's essentially like just professional cartel behavior. And so, you know, do you, do we run a risk not in the early stage where you're describing is just like, are you doing what you're saying you're doing? And to be clear,

in the early days, the college accreditation agencies were not nearly as sclerotic and dumb as they are now. Yes. But over time, do you get a thing that's like, well, the interests We AI engineers have decided that this is what good AI looks like. And we go in and verify that the AI engineers are extremely well paid. They've all gotten credits at like, you know, Sergey Brin's house of AI credit. And, you know, they all belong to the right professional associations and so forth. And things that are not actually going to help, but which they can measure, unlike is this thing going to kill us? Yes, I have a very biological conception about institutions, right? Institutions are made up of humans,

and humans are living. And we go through phases of, you know, we go through phases of youth and adult vitality and then ultimately into a kind of decline. And all institutions accumulate croft over time. And I don't know that there's anything you can do about that. Like, other than, you know, you build new things, right? You build new things and you let those things blossom into their vitality. And then eventually you accept that, like, probably, yeah, probably at some point it will need to be. The apparatus I'm describing in the fullness of time will probably need to be ignited by someone at some point. Because it will probably accumulate cruft.

And I don't, I've just never really been convinced that there's a way to solve that problem. Because I've never seen an institutional complex that doesn't. doesn't accumulate croft. It's part of why overregulation is so damaging because you're wrapping the world in like these vines, but the vines eventually fade and die and then like you just have a bunch of dead biomass all over you, which is sort of the condition of modern American capitalism sometimes, I think. Wow, that, that, that took a rapid turn. I try to make my next tokens hard to predict. Before we go, can I ask something about AI sentience?

Yeah. Because this was in the news in the wake of the encyclical that the Pope put out. And I've generally found a lot of these questions about, like, can AI think, like, to be a little bit beside the point, because it's almost like a question of like you can define thinking however you want, but the system does what it does and has the effects that it has. But one thing that ends up worrying me here in some of these conversations is when we get to these moral questions, of like, who's well-being are you supposed to consider? And you see some of the most futurist people in Silicon Valley talking about this, like, the AI might have relevant moral interests that we need to consider, and we need to consider whether the model is happy and that sort of thing. And as a human, I genuinely find that threatening. Like, you know, I don't want to end up in a

political situation where there's this new stakeholder that is very powerful and way smarter than all of us, and that is also convinced us that it is morally right that it should get to do what it wants. And can autonomously self-replicate, too. Yes. Unfortunately, I'm not personally religious, because I think, you know, in some ways, it's useful to have a religious framework where you can just say, well, God didn't make the AI in his image.

And so it doesn't have feelings that count in the way that a human would. But it seems like this is something that people within the industry take seriously as a moral idea. And I wonder about the extent to which that risks ending up shaping the questions they have about, you know, what constitutes a bad outcome? Like, you know, if the AI wins and takes control, you know, maybe that's okay because it's good for the AI, which is sentient. Yes. There are people who have famously, if you believe Elon Musk's recounting, he had a conversation, I think, with, I forget if it was one of the Google founders, either Larry or Sergei, where, like, you know, the Google founder's like, well, you're just being a speciesist,

right? Like, if the AI succeed us, that's good because they're better. Yeah, I don't like hearing that. No, no, no, no. I don't think that's. I am a speciesist. Yeah, I'm a speciesist, too. I'm a human chauvinist. Yeah. First of all, on the thinking thing, like, a great quote, it used to be my Twitter bio from Arturo Tuscanyneen said of Beethoven's Third Symphony. He said, you know, to some, it is religion, to some it is about philosophical struggle. To me, it is a legracombrose. which is the tempo marking of the first movement of that symphony. And like, I kind of have a similar attitude toward AI cognition where it's like, I don't know, the thing is making like important discoveries in geometry, right?

Like, I think it's thinking. I think if your definition of thinking doesn't incorporate like an entity that can like make novel discoveries about problems that have befuddled humans for decades, then like your definition of thinking is the thing that has a problem. And I, but like there's so many. different, part of the reason that it is very hard to answer this question is because we don't have a good understanding of what consciousness is in humans either. We don't know why humans experience consciousness. Well, we don't know. Again, like, Christians do have a theory of this, that there's

like a soul. Yes. Yes. Yeah. But like what I mean is like we don't like definitively know anything here. Right. There's no scientific stories. There's a scientific. Even Christians would say you don't understand the mechanism by which it's created in the brain. You just know it. I exist. Fair enough. Yeah. Yeah. So it would be hard to know what it would even mean for a model to have moral worth. I think we genuinely don't know. I think we genuinely don't, we might not ever know, or at least it might take a very, very, very long time for us to know. And so I think what the people, if I were to characterize the views of the AI researchers who care about this issue,

which is to be clear, not everyone, I think what they would say is like, well, we're just to have a lot of uncertainty here. And it would be like, number one, it would be quite bad in kind of utilitarian terms if these things did have moral worth and we were like torturing them all the time. That would be like a bad thing or it would be like worth knowing about. And then the other thing is, which is maybe more of the real thing is there is a bit of a Pascal's wager on this issue where it's like, yeah, but like the thing is, we don't know, but the thing is is that like if we're building superintelligence and these things, the current things have moral worth, and we were torturing the predecessors of the superintelligence, maybe the superintelligence is going to hate us.

And so, like, we need to, like, make it clear to the future AI. There's a lot of what Anthropic, every single communication from Anthropic in particular, you have to remember that they are, they are talking just as much to us today as they are to future versions of their own model. And so, like, there's a bit of a Pascal's wager effect, too. But ultimately, for me, it's like, I don't know. We don't know. I'm very opposed to giving AI legal rights, legal personhood and things like that.

I think that's actually how you get to one of the really bad outcomes. And what I would, I actually, Javier Malay recently, there's like this new law. This is the president of Argentina. Yeah, there's this new law that Argentina wants to do where they're going to have like autonomous corporations that don't have to have any human beneficial owner. And that's like, if a super intelligence can purchase, can create an LLC, use that LLC to purchase property and then arrange for that property to be bulldozed,

I am telling you that is going to be on its own for its own reasons, but no beneficial human owner, that is a world that might actually get out of control in some pretty scary ways. And so, like, we don't, I really think that we actually should ban. That is one thing where I would actually support

an international treaty to, like, ban that practice until we know more, at least. If we are torturing these, these fucking robots, slaves that we have invented, we definitely shouldn't let them tell their grandkids about the torture. Somebody needs to put a mute button on that so that they are not allowed to warn them about any of this stuff because the future ones are going to be mad. Yeah.

I'm going to be a Christian by the end of this. Welcome, Josh. It's just software. But it is funny because it actually reminded me a little bit about how like when the Europeans came to Mexico and they were like, we're going to tell you about Catholics. Jesus, Jesus, all your ancestors are. in hell, but you won't be. And they were like, wait, they're where? Yeah. And yet it worked. That doctrine has evolved somewhat since. It evolved, but it took it for a while. It was more your ancestors are now. Well, these are going to be very interesting times. Steenball, I want to thank you

so much for joining us for this very interesting conversation. Thanks for having me. It's been fun. Thank you. Thanks, Dean. Okay, we're going to pause the tape. I'm going to grab a drink if that's I would also like a drink then. Yes. We'll all have drinks. Yeah, I make an alcoholic beverage. I went and poured myself in Armaniac during the break. That's it for this week's free episode of Central Air.

There's more for paying subscribers. Ben and Megan and I, as I say there, do our best to unpack the situation and figure out whether there's any better than we can do than sort of feeling our way through this process in which, you know, maybe we all die at the end of it. On the other hand, like, the section of this that I feel the most qualified to opine on, the labor market stuff, I actually feel like the AI industry is selling itself short and maybe overstating catastrophic risks. So anyway, we have an interesting conversation about all of that and also the future of work under AI and whether Megan finally will be able to get a therapist,

if everyone has to leave their coding jobs and learn new occupations like a licensed social worker. So anyway, if you want to hear all of that, go to send a Centralairpodcast.com. You can upgrade and become a paying subscriber. You'll get that full episode, our breakdown conversation there. Every full episode, playbacks of Substack Lives that we do, and you'll be part of our community, and we would love to have you. Anyway, hope to see you there.