a16z Podcast - Aaron Levie, Steven Sinofsky & Martin Casado: How Do You Secure a World of AI Agents?

Episode Date: September 26, 2026

Erik Torenberg sits down with Box CEO Aaron Levie, and a16z’s Martin Casado, and Steven Sinofsky to debate how the AI industry should think about safety, security, and regulation as increasingly capable agents move into the real world.They argue that much of today’s conversation is happening before we have clearly defined the risks we’re trying to regulate. Drawing on earlier waves of computing, from computer viruses and the early internet to aviation and automobiles, they ask what AI can learn from industries that developed safety standards only after understanding how their technologies actually failed.The conversation then gets concrete: agents don’t get tired, can operate at enormous scale, and can probe systems in ways human employees never could. That could require rethinking permissions, authentication, operating systems, and the security stack itself. They also discuss why AI innovation may increasingly move beyond the frontier labs and into the software built around the models.Resources:Follow Aaron Levie on X: https://x.com/levieFollow Martin Casado on X: https://x.com/martin_casadoFollow Steven Sinofsky on X: https://x.com/stevesi Stay Updated:Find a16z on YouTube: YouTubeFind a16z on XFind a16z on LinkedInListen to the a16z Show on SpotifyListen to the a16z Show on Apple PodcastsFollow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

If you regulate AI too early, you actually don't solve anything. You still just kind of had the same risk ultimately. You willed the thing into being, but you haven't figured out how to control it. The problem we have now is this rift between the labs and the security community that keeps coming to two conclusions. Sloppy. And you're not complete in what you're telling us happen. An employee is like 10% chance of species extinction.

The post is very reasonable, but the atmospherics are not. Agent Swarms completely flip that. These are just roaming drones. But like time 10,000, and they will easily mistake a good task for a bad one. So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. The U.S. about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose. This could change the nature of software fundamentally.

The center of innovation has just moved. This is the signal that the early Internet. was riddled with viruses, worms, and security failures. We didn't stop building it. We learned how to make it safer. What should AI take from that history? In this episode, I sit down with Aaron Levy, Steven Sinovsky, and Martine Casado to debate AI safety, regulation, and what changes when agents start operating across the software we use

every day. We get into why agents could force a rethink of permissions and cybersecurity. What decades of software security can teach today's AI labs? and why regulating a technology before we understand how it actually fails can create problems of its own. And we look at a broader shift already underway. As models mature, some of the most important AI in inundation may increasingly happen outside the frontier labs in the systems and software built around them. Guys, welcome back to the podcast. Thank you.

Didn't think we'd ever do this again. I can't believe this is great. I mean, Martin's just building these $100 billion companies. company is too busy for this podcast. Well, at least taking credit for us. As VCs do. Exactly. We've a lot to discuss today, but Aaron, when we start with you,

pacing the frontier, how have you reacted and reflected on what's happened there and just the discourse that's followed? Oh, boy. I think we should start with Martina on this one. You were fighting lots of good ground wars. Maybe I'll say one thing that we probably all agree with, and then we can figure out where we maybe kind of fracture off. I think we would agree that any AI lab right now at the frontier

should be building in the safest way possible with the highest degree of governance and security and whatever your definition of alignment is. This is an incredibly important area of research. It's an incredibly important area for the diffusion of AI. You're not going to have AI diffusion without extremely high-quality products that can be trusted by enterprises and that aren't kind of constantly hacking systems.

So when at least I read the Dario Post, I actually didn't disagree with almost anything because it was all about how do you have better security of these systems, unboxing, better testing. There's going to be some debates around the embedded nature of the testers. And do you agree with who those are? And does the industry all align on that? But I think actually all of the major points were probably salient and appropriate.

Then the only question is, does this get sort of user leveraged to do things that maybe we don't agree with, which would be like a slowdown of AI dramatically because of regulatory controls that would sort of not make it easy to compete with the frontier labs? or do politicians kind of end up sort of taking the message and run with it and maybe even worse outcomes happen? It's used to ban data centers far faster and whatnot. And so I think the actual substance of the topic is actually incredibly important and I think very important for AI advancement in general.

And then the question is, what do you do about it, especially what do you do about it from a regulatory standpoint? And that's probably where the industry is going to land on very different points in the continuum. But Martin was putting up a good fight on let's make sure that we don't use this for regulatory kind of capture, which I also agree. with, but I think the ideas in the pacing conversation are important. Again, it's a little bit of a funny concert because maybe it's not even pacing as much as just good hygiene and engineering. And so with good engineering, obviously there is a slight slowdown, but it's a slowdown

that obviously allows acceleration of your diffusion because you wouldn't be able to have any of the AI be diffused if nobody would trust using it. So the post is very reasonable, but the atmospherics are not, right? So an employee is like, this is going to kill whatever. a 10% chance of species extinction. And you know what Dario says? I agree with him more than I disagree with him, right? On TV. On TV, the same day that he landed these things.

And so in some way, you can't have these conversations in isolation, which is, of course, if he's going to agree in species extinction, this post that he has looks like this milk toast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics. And then I have this quibble, but it really bothers me because I'm a pedant, which is... I think pacing is the wrong way to describe this.

Right. Yes. It is orthogonal to security. Right? So you can very slowly build a nuclear weapon. Right. And that doesn't make anybody feel better than a slow versus fast. So that's one.

The second one, it just feels like a capitulation to the pause folks without actually addressing it. So you're saying, well, we're not going to. pause we're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy. Because the pause people are like, well, that's not a pause. That's just pacing. And the regulators are, you're still doing the thing.

Right. And so I just feel like my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal, and I applaud them for that. I think the messaging is wrong because they're trying to like split the difference between an internal kind of fringe faction, which are domer, and then the regulators on the other side.

And the problems are making both of them unhappy. And I think what they have to do is they need to come out. They need to address the X-risk question directly. They need to say, no, we don't think this stuff we're going to do is going to cause extinction. And then I think this becomes this very sensible. And what would you do just to play the other side for one second? What do you do? Do you make room for the one possible Venn diagram,

which is the lab researcher that is both simultaneously super scared, but also works on advancing the state of AI because they believe that it's so important to get right that they want to pursue that. And then obviously the language is right now very problematic, but that person does exist. And that is a real kind of person in our industry, which is like we have to be at the forefront of AI.

I'm also very scared of it. And so that's why I'm working on this. So let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things, that have access. You were the first pacer.

We were part of the first pacer. So if a contingency within the labs that are the most knowledgeable people believe the stuff has existential risk, the answer is to nationalize it and actually put controls that we know that work, right? Now, if they don't actually believe that, and in the private conversations I have,

the most sensible people don't, it's a small fraction that do, this is an HR problem. Right? So to me, an HR problem is a company problem. Like if they are worried that they can't recruit people or they can't retain people, which it seems to me a lot of this is just that concern. It's almost more of this kind of research or currency.

If that's the case, I think that is the wrong reason to cause a national level lockdown on a very promising technology. So listen, I think the post, again, I think the post is actually very sensible. I think there are real concerns around security. We've had many compute epochs that have real security concerns. I don't think you can reconcile discussions on X-risk with the proposal that was put out. You just can't reconcile those two things. And that has always been my primary.

Right, right. Okay, unleashed. Okay. It's just holding it back. So, okay, the first thing is, is there a schedule that they've published that says when all this stuff, whatever bad stuff is going to happen is going to happen? They haven't. So you can't pace it because nobody knows what it was supposed to finish in the first place.

It also just seems disingenuous, too. It's complete nonsense. You can't claim, this is like when the press reports on Apple's latest iPhone is late. From what? Nobody knows exists. They haven't told anybody back. Yeah, my Apple car was very late. Yeah, like, I don't understand.

In order for something to be slower, you need to know the rate which was moving in the first place. So it's all just the utter nonsense. And you can't escape that. And then, by the way, this is another problem that, like, again, from my little quibble on PR is, nobody believes it anyway, right? And so if that's the thing... Wait, which part? The pacing, right?

That they're going to pace? That they're going to pace. I see, okay. They've been at a dead run. They've raised more money than ever before. They've run faster than ever before. There's no indication that they're pacing. So if this is what you're going to hang your...

I see, I see, yeah. Well, well, the issue, obviously, is that the model that everybody has internally far seen what anybody else has access to. So it's really just the pacing of external releases. But again, back to your point, pacing versus what? Right.

We don't know if the one that scares everybody also doesn't work for a legal brief. It might actually screw all that stuff up because it's so... Who knows? Right. So there's that. And it's just disingenuous to claim that you're paid.

Second, why do they have to announce all of this and ask the government to tell them to pay? Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is going to go because of your product, just stop. Don't do it. Like, this is the...

Like, I worked at a missile factory in college, and we had nuclear missiles, and I walked the floor. What's with you guys in missiles? I'm just here with software. I guess. You were one or two people in college when we were, which was either you were protesting to keep them off campus. Or building up.

That was your choice. And like, like nuns from the Catholic Church show up and pour blood all over on missiles. You know, and I'm busy just wheeling PCs around on carts saying, here's a secure PC. And I'm like scared to die. I don't know what he was going on. And I'm like, it's a nuclear. missile. And then you find out that that's what stopped the Cold War. Like that literally,

it was a Persian missile. And that was what did it. And so, but you said something I think is super interesting, which is pacing is this sort of fuzzy non-word between like going and not going. Yeah. And the problem is it, you're exactly right. Like, there's no one is going to be happy with the middle road. And so one of the things that, that people, I think it's almost fun for me to watch as a sport, they think that all these people saying to the government, do this, don't do this, that they think they're going to get what they want. And it's a complete 100% misunderstanding of how government works, which is when you talk to the government, they actually know how these things work. And they know they're just listening to you and they're listening to everyone.

And no one is going to get what they want because they know in order to do anything in our system, it's a compromise. And so everything that all the inputs can make it into the government, the output never makes everyone happy. Right. 100% of the time. It either doesn't go far enough

or it goes way, way too far. Yeah. And so you can't take the view that you're going to talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're going to get the wrong velocity. Well, my favorite thing is like,

who's it? David Sachs was like, I think it was David Sachs, but someone from the government said, you're asking us to regulate you, no. So basically the answer was no. I'm probably just Trump, I think.

But the thing that they know now, that you just know from experience is, once the wheels start on regulated. You can't sell that one down. And now it's become an election issue for every party and every jurisdiction up and down the whole government stack. So there's now this whole basket goods,

basket of regulatory approaches. Well, the next election will 100% be a referendum on AI. So, like, it has to happen that 2028 is, like, the AI election. And you can basically run on... The problem is, like, it's not obvious who would run on the pro-AI story because it's going to be too nebulous to tell that story. So then it's just basically varying degrees of how much do you regulate it or at least try and, like, avoid the topic?

Yeah. But it is too bad that we... we as a country are in a spot where, like, the pro-AI case is just like, it sounds too, it's just like takes too many words, you know, it's way too nuanced. It's defensive. Yeah, it's defensive. Defensive. And we own none of the vocabulary.

Right. So the whole debate is, is pause, it's swarms, it's rogue. Every word has been chosen by the people who don't want to do AI. Yeah. And so it means the first thing you have to do is invent new words and say that their words are wrong. which takes so many words that... Yeah, so we got to be, like, union jobs and, you know, cancer, and, like, you know, there needs to be another word cloud that emerges.

I mean, what I don't understand is why the labs have not taken a position on X-risk. Like, short of that, I think this goes in any direction other than heavy-handed regulation. Yeah. Well, it would just be negligent of the government to be like, there's a 10% chance of species extinction. The CEO of the top company says he agrees with that. Like, how can a government? government not do have he had a regulation.

Well, but what would anybody knowing this ecosystem, though, I don't know that you would be able to pin anybody down on that other than something. No, I was it, Dario said it on. No, but I'm saying you're not going to pin anybody down on a lower. Well, in fact, he does the worst thing about it, which is

he agrees with people who claim that they believe that there's an X percentage of extinction happening. But he specifically is out of his way to say, I'm not going to put a percentage on it, which I just think is the weird No, but that is, to be fair, to be fair. No.

Okay, no. Nobody can be fair. To be fair, the, that's, like, it's not a hundred percent, like, like, disingenuous or whatever. Like, like, he probably doesn't specifically agree that it's 10 percent. Or maybe he does, and it's just too scary if he were to say it. Well, let's just play binary search. But, I mean, is it more or is it less?

But the whole thing is it made of, he knew is it made up concept that we just create, but like, nobody can. You can't quantify any of this. But that's sort of Martin's point. But you just had an official position, though. So the only official position would be, like, I think the only official position you could possibly get would be the PR version of this that would be the only thing that would be intellectually honest would be there are real risks with AI.

There's incredibly positive benefits as well. We are working to mitigate the risks so they are as reduced as humanly possible. I don't think that's. No, I don't think that's true. Listen, so we've been through multiple epochs of technology. Yeah. We've been through compute. We've been to the Internet.

We've been to the web. We've been social networking. We had a discussion about the risks without talking about X-risk, right? So, for example, one thing you can say is we do not think the marginal risk for species extinction is different than it is. Right. Than it is. But what if they do think it is higher? Well, then we should.

Okay, okay, okay. I think the answer is they do think it's more than just the Internet. We have one of two options. You believe that we're going to go extinct and we shut it all fucking down. Like shut it down. They believe that there's a chance that we go. Speaking of cold. Dario thinks less.

No one, like the Pentagon, you know, they ran a lot of simulations on the chances for nuclear war, great movie war games about the whole thing, all of that. But the thing was, it was non-zero. Yeah. And so once you said it was- Can they say non-zero then? Is that allowed?

I think as soon as you think it's non-zero. The problem is if you say non-zero, that could be like, you know, shit, me. You think I think it's 93%. Let's not, wait, just so we're all having a think conversation. Let's talk about marginal risk. Yes, yeah. We're not talking about absolute risk.

Right, right, okay. It's just that I think if once you say it's non-zero, the only answer of like catastrophic. For marginal. The only answer is you have to nationalize it. Yes. And so what he's trying to, what the labs in general that have that view are trying to threat is they want it to be non-zero as a license to do a certain set of things without. the burden of just becoming a nationalized. Well, do you have, I actually don't know all of the precedents you hopefully do, but, but like,

there must be some things that are non-zero, uh, let's say X, X risks that are not particularly nationalized, but the, but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements on, like, like, like, I'm sure like, to develop anthrax, you have to go to a particular kind of lab. Well, BSL4 labs, okay, but they're nationalized. They are nationalized. Okay. Ethics tends to be nationalized.

Yes. Like normal human safety, not so much. Like industry oversight, over the time becomes federal regulation. Right. And the progression, which I think is just super important to this discussion, has been since the post-World War II era,

most industries that are critical to the infrastructure, power and banking and health care, the trend has been to basically be nationalized. Yeah. But just like your examples of K.Y.C. And all the other stuff, the banks are for all practical purposes nationalized. And the financial crisis. Okay.

But you said all practical purposes. Right. They're literally not nationalized. Right. So maybe that, but this might be the intent of the labs is to look like J.P. Morgan or look like Verizon. And it's just like we're critical infrastructure. We get heavily regulated.

It's not good for open source or at least frontier open source. But it's a, it is like, it's like a plausible outcome for this industry in the limit. It's really good for innovation is the problem. I think even that's fine. Just don't use species extinction. Sure, as you're arguing. This literally is like the difference between the things that are like stuck in the lab.

And let me just say something. I actually think the labs are moving in the right direction. I actually think the actual statement was really good. You know, in my discussions with, you know, executives and leaders, like they understand that they have like this tension and they're going to reconcile that. So I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing.

I just think that grew so fast and just trying to figure out how this machinery works. And I think Sinovsky really hit the nail on the head. The political process is its own thing. I don't think, I don't know if it's naivety or hubris, but I just don't think that they kind of know how to navigate it.

Well, I think the tech industry has literally over 100 years consistently relearn the lesson at each technology wave, that we don't understand the regulatory climate and we can't navigate it. even the companies like AT&T and IBM that were born out of basically being government monopolies from the start never figured out. Both got sued for antitrust, both got substantially and structurally changed as a result. And they had hundreds of lawyers in the 1960s navigating them.

And that, you know, Microsoft came along. Like, we were just like, what? Yeah. Like, we had no idea what was happening to us. And there's, you know, there's Bill Gates playing golf with Bill Clinton. And then we get slapped with anti-trust laws from his administration. And Bill was like, I was playing golf.

It was the picture. And that doesn't help. And like, isn't that what I'm supposed to do was go and play golf? That's a shortened version of those stuff. But I think, and I think it's just, it's, I always use this example, which is the Hollywood got together

during the red scare and all, and censorship. When they were worried about the government censoring movies for, for, sexual content, for adult themes, and they all got together. And, of course, they were never able to win in court if they tried to,

but they were going to, threatening to do it. And they got together and they formed the Motion Picture Association. And movie ratings and all of it, and they police themselves.

So how do you, do you like guys like, do you like the FINRA proposal? No, because FINRA is, that's as close to NPAA as you can get with more teeth. No, it's not,

because FINRA is going to, Finera becomes legislation, which comes with direct oversight. Yeah, I think MPAA might not have as much consequence in like how society functions. Well, no, the First Amendment.

Yeah. Did I win that? No, I, first of all, fantastic. But I still don't know if you won it. Like, I agree free speech is really important. But like, but like, you know, like the, I just think. There was no societal risk. I just think what's in our movies will be like, we'll survive.

Like on like a different continuum of. Every history is always relative. And at the time, being a communist was a really bad thing. And 30% of Hollywood got fire. for you know, it was all this stuff. Yeah. So I, it's always a risk to bring up something in that kind of way because it sounds so dumb. Like nobody thinks about movie ratings.

And that's because like actually they were ruled basically you can't constitutionally mandate them. So they couldn't regulate them on cable TV. And so we got to grow up with HBO and all this other stuff. Yeah. But the problem with FINRA is that is a perfect example of essentially nationalizing risk. Yes. Because even though the banks all pay money into it and that's how it's run and stuff, it's all mandated. Okay, wait, sorry, you think we're going to end up with a situation that is better than FINRA?

Finra appears to be the best case scenario. It is the best case scenario. But not even anymore. Like, I do think this technology is in the limit, again, so powerful relative to what it could, I mean, for what it can deliver, that it would be impossible for eventually Congress not caring about that. Like, it's just like not possible. That's what question is, if it's eventually, if it's eventually making every recommendation in your health care process. and it's inside of your medical device as an open weights model.

And it's all, and every trading system for high frequency trading, there's just, and it's on an airplane. Like, there's no chance that the government doesn't say we need something where FINRA actually is like the probably the best case scenario of what that looks like. Right. Yeah.

And so now that's absolutely the most crucial point. Because if all the people in the Senate now were in the Senate when, when the Communications Act was passed, and liability was not passed. through to ISPs and to social networks. And they sat around. The debate at the time was they would literally say to us,

the internet is so big, how did we not have anything to do with it? And that's why Al Gore gets a bunch of abuse for saying he created it. Right. Because he was actually trying to get out in front of that and said, no, the government was instrumental. And it all backfired because it made it look like he was a crazy person. But that, it is absolutely the case that they felt like they missed their chance to be on top of the internet.

With Al Gore being on the positive side to innovation. That was their polk, right? So who is the Al Gore? There's no one. Yeah, yeah, yeah. There's a couple, well, Bessett seems to be like that. Right, right. You know, the defense people sort of wanted private, but not,

which is exactly where they were on the internet. Right, right. And so it's very interesting that a lot of this is just this, like Elizabeth Warren's, Senator Warren's tweets were all like, we missed this for social network. Yeah, yeah, yeah. And it's like, there's a lot of pent up energy against tech that could end up just all siphoning

into AI regulation. That is exactly what it is. That's what's happening. Yeah. I think there's another problem, which is we're all trying to predict what's bad, which is not how we've normally done things. Like, by this time on the internet, we'd taken out like tens of billions of dollars of like economic, well, we've caused tens of billions of dollars of economic damage.

We've had words that took out 10% of the infrastructure. Yeah. The internet infrastructure, which was running critical infrastructure. We had hospitals go down. We really should have blocked the internet in like 97. How did we? We, yeah, okay, okay, okay. We had, I mean, I remember when a time, you, like, I remember when you would, you would, you would buy your CD of Windows 95, and by the time it was done installing, you would have a, a worm potentially.

Way to go, Stephen. No, no. No, this was the reality. No, totally. The reality of the PC until 2001, was you could not install a PC connected to the network without getting infected. Oh, viruses were everywhere. for the time. And there were two level, two rounds of congressional hearings.

You had all the same. That is actually, okay, I'll agree with this. That's a very interesting point. This type of zeitgeist in 94 would have, we would probably not have the internet. Listen, listen. I mean, the automotive, the airline engine, you always have these periods of kind of like teeth cutting where like you learn about the dangers and you learn about the technology and the internet. I mean, we were on the ground floor of this. I mean, things were down all the time.

There's economic damage all the time. And like, there was, like, novel, there was new viruses. There are new worms that are all over the place. But Y2A, Y2K was going to destroy the world. Here's the interesting thing about this. But I just want to say, like, so when we created policy and we did create a lot of policy, it was kind of like with a bunch of very specific data points on what you're trying to do. And so you know that the policy actually fits the fact pattern.

And in this case, I mean, even when you were talking, you're like, well, what if, you know, yada, yada, yada. And it's very hard to be predictive policy. With that. No, one area we can talk about is. is there seems to be novel cybersecurity risk. Great. And what is nice about the discourse,

and it's actually starting to evolve around that. Like you actually hear people from the lab saying, novel cybersecurity risk, this is an engineering problem, we're talking about that. So the more of this becomes concrete around real identified risks, I think we can all fall in line. But that's not been the discussion to date. That's a very recent thing.

Perfect, perfect, perfect point. Because if you look, in 1986, the Computer Crime and Fraud Act got signed, it was out of a very, very specific scenario, was two groups of hackers broke into GTE Telemail. One of them lived in my dorm. Go figure. And sure. No.

It wasn't you. He worked at Cisco later. And the problem was that was 1983. And there was no crime. And it took two and a half years for the bill to make it through. That made it very, very specific. And that's the law that says you can't use you can't access unauthorized.

access unauthorized computer systems. So I think we'd all agree that we probably, you know, we probably have like 90% of laws already in the applied layer. Like you can't hack systems, et cetera. Do you think there's anything that should be from a liability standpoint in the model layer, which of course then- Wait, and now are you making a technical? Well, that's a very legal.

No, the legal, everything, my read of hugging face, open AI, yeah, four, they're all absolutely blatant computer crime acts, except for the fact that there's, they carved out in an, amendments later that if you're a white hat, it's not illegal anymore. And that was because people kept making mistakes. And they didn't want to go around arresting everybody who was actually trying to make the system better because they messed something up. And so the Justice Department wrote a memo that said, we're not going to prosecute for this. And we're also not going to prosecute if you just, like, violate the terms of use of a system versus actually try to breach it.

And so I think the law is ample for this, for the scenario. And it was all. written this GT telemet was used by NASA and Livermore and all the labs. And so that's why it was it caught the attention of D.C. because it was federal systems that were being broken into. And the problem we have now is this is this rift between the labs and the security community that keeps looking at all their postmortems and coming to two conclusions. Sloppy. And you're not complete in what you're telling us happened. And so, of course, the CBE process came about in the 1980s, the computer virus and vulnerability reporting stuff

out of CMU. And for years, they worked on very structured reporting with obligations and how to, and for some reason, they're not using any of that to do this reporting. And so there's this very basic stuff that I look at and say, well, until they're doing that,

they really should stop talking. Like, they shouldn't do a post-mortem on a breach that looks like an intern wrote it, and it's not a post-mortem. It's this selective memory. It looks like, The kind of postmortem you do when you hire outside lawyers to investigate some random thing and you only give them certain stuff.

Because you don't have to give the lawyers you hire all the information about what happened. Like, where's all the slack messages? Where's the actual details of what happened? Can I just interject an annoying aside? Which is more annoying about what I just doing. No, no, this is very in line with this, which is, I've been in the security community, like the actual cybersecurity community for a long time. and it is, you know, it's always been one of these things

where, like, they just don't like practical solutions. So even if you build, like, you know, a secure system, like, well, what if somebody, you know, shows up? Oh, yeah, yeah. You know, like, can Russell Crow can, like, break the encryption or something like that. It's mission impossible. So there's always like these kind of like whatever.

So my favorite thing that happened recently, it was like, Nome Brown. Yeah, it was some podcast. We've all said stupid stuff in podcasts. I've already said this one? No, it was great. No, I thought it was fun. No, it was fantastic.

No, no, no, I'm setting it about it. Okay, okay, yeah. So No Brown is like, listen, you know, you don't know what a superintelligence could do. It could maybe use like the heat of a CPU to exfiltrate itself to another computer, which this brought me back to my... I'm like, now I'm very comfortable. Okay, okay, okay. Great.

Now, I can have endless, pointless discussions on this. But what was interesting is you basically have the X-risk people saying something you thought was plausible, and then you have like the security people having this kind of endless discussion. And so I think at some level, now these communities are being bridged, which... It was like, you know, I actually think that was a very reasonable thing from Rome Brown to say. I think you can pick holes in it, but we all say we're a stuff on podcast. I actually think actual risk is real. I mean, I worked in secure computing environments that were highly classified that the covert channels were unbelievable.

So these are very real comments, but like we actually have a real discourse. And it was the first time I saw really hardcore systems people having pretty, like, I would say constructive. Well, they were calculating the bit rate and all the students. But it was a constructive discussion. And, like, you know, you had like the kind of the typical extras people were engaging. And of course, it was Twitter. So there's like a lot of name calling this and that. But it was actually, I felt like a real discussion for the first time.

So I hope we see more of this. Wow. We see more, you know, you know, cyber-related things. I think the lab should talk more about it. And I think it'll engage the community. And I think once that happens, we can actually. Yeah, Greg's been out there a lot more on this topic, which has been, which has been good. But I think the takeaways, Noam Brown should be doing more brainstorms on podcasts.

I thought it was great. It opened people's eyes to the fact that actually there's a lot of risks in security that most people don't understand. And so that means that there's more stuff that, like you can't make baseline risk, you know, like how is your authentication layer work? You can't just wave your hand and say that should go away and then bring up, oh, but, you know, space aliens can invade. And that's a little bit of what was going on that I felt uncomfortable with. But like, what do you mean? Like, it was sort of like, but you know, there's also this risk. And it's how I view that.

The heat thing. But at least we're now in a domain we're comfortable. Like, I can talk about entropy. And we can actually have a concrete discussion. That's not, oh, well, it's super powerful. Because we reduce it to, like, the laws of physics and the laws of system. And most people did. I would say most people had no idea that that kind of risk was real.

I mean, like, when I'm walking around the Persian missiles, I actually had to test the graphics cards and PCs because a DOD requirement is that the screen memory not be sustained when you pulled the power. But not for zero time. Like the minute that the power went off, the memory image had to go. And if there was like a three second delay, you, that could be read. Oh, see, see, we had people that came into our offices

and would literally measure the distance of the monitors to each other because of Tempest attacks, which is 100% away to use electromagnetic radiation to leak information. I've seen the same thing with spread spectrum from the bios. I've seen it from, I've seen it from audio speakers. Like, we had to remove the speakers out because it's a very high-backer channel. Our building, our building, which was secure. Our building had just Muzak speakers aimed at the windows.

Yeah. Just to produce interference. You need to go run safety at one of these labs. This is like, I've never seen you at more excited than Heath, you know, based communication. Can I tell you the craziest comfort channel I've ever seen? So it turns, remember the old CRT? I know this is like one of me. Yeah.

So I'm glad that someone's older than me. Not really, but acting it. So it turns out if, like, let's say it's night and you're using a CRT terminal in your room. the lightest thing in the room is actually the pixel that the raster beam is on. So most people think it's like the glow of the monitor, but it's actually that given pixel. So somebody figured out that like,

let's say you're in like a hotel room and you're on your computer, if you have something that can sample the color of the window, you can reconstruct the screen. Oh, that's crazy. You just do it at the same hertz that the raster beam is moving.

Then somebody else figured out if you can subvert three pixels, you can use those, because it just looks like bad pixels, you can use those to basically send a message. So like you could literally like sit out, in whatever, like to your own SOS. You sample the message and you can,

it is a relatively high bandwidth one-way communication. And so, like, what no brand was saying, like, maybe heat is not the way to do it, but that level sophistication is actually real. That's a thing. When I was at the missile factory, like I had to lock my keyboard up. That's really an impolite word, but that's what we call it.

I had to lock my keyboard up at night because they didn't want the custodians who didn't have clearance walking by and just noticing which keys were dirtier or cleaner. Oh, yeah. And I once left it out, and there's like a note from security, you know, telling me to report to security and pick up my keyboard. Like the guard and walked the hallways just took the keyboard that night off my machine. And that's like basically, I was not cleared.

I was just had sensitive. You know, I was like nothing. I was an intern. The big problem with this conversation is now this is all in the training data of every AI model in the future. But that's also the threat model. The threat model for these things is always you've got a trusted site and an untrust. NIST has 500-page manuals. Okay, okay, okay.

All right, this is already out there. And the untrusted side, you assume basically an Oracle that can do and know everything. Yeah. And then the question is, can you get information off the trust side? Which, by the way, is what Noam was saying, which, again, is this is actually quite something. Which I do think brings up a super interesting point, which I'm going to bridge to which is just that I think the thing that people really aren't wrapping their heads around and are using the language that's really confusing is just that what AI can do is it can try all of those things in a very short time.

Yeah. And it doesn't get tired, it doesn't get bored. And, you know, and so, but the interesting thing about it is there's a whole layer of security that you now have to go look at every single API, every single service you're running internally on your network as like, you know, like nobody thinks that their internal GitHub or their internal Slack or internal finance expense tool is, is vulnerable to a denial of service attack. But Swarms will do it.

We'll do it. And so now we need a whole layer internally that just is tracking way. more about what authentications are being done, what APIs are being done. And but that's just like, now it's just going to be basic. And all the off-sec people that are old are like mailing me like, why are we explaining to this to everything? It's so basic because nobody did it internally. Yeah.

And well, you didn't have to worry about it for your people. And that's the like. But now your person is just a piece of software. Yeah, right. And like unlimited. And the- As a credit card. Yeah, I mean, we, you kind of got by with information security like to

some extent on the fact that most people will do the right thing 95 to 99% of the time. Wait, is not fine. The malicious employee is like one in 10,000. Yeah, yeah, yeah, yeah. So, like, so all these systems are basically open to whoever wants to access them or like one tap on the shoulder and then you have access. And agent swarms completely flip that because they will, these are just roaming, you know, drones.

Yes. And, but like, yeah, time, time 10,000. And, and they will even. easily mistake like a good task for a bad one and vice versa. So the data security in our systems is a, this is going to be a huge upgrade moment. I actually, Martina, like, I think that actually we're going to need a different access and security model going forward.

Where are we going to go on that? Because the model we have is not granular enough. And it's not performance enough to handle this stuff. So I want to step back, I don't say like a meta point, which is I think this is how these conversations should go. We've identified a novel risk, which is like cybersecurity, which we actually have proof points, and now we're talking about solutions.

I think the entire discourse around AI can be of that form, and the biggest mistake is that's not what it's been. Like, I think the entire industry and community is very happy to engage exactly like this. And I have something to say about exactly what you're asking. I think this is where the conversation should be. We're known for having healthy conversations that everyone should learn from. So that's what we do.

So here's the thing. I don't think there's a technical limitation here. Like, these threat models are very well understood and have been in the literature for a long time. I mean, like the operating systems research, the MLS, the multi-layer security research, has considered these sorts of things from an academic lens. The reason it hasn't been adopted

is just tended to be a usability issue. I guess it's really hard to maintain, and you didn't have to. So you could argue that AI solves the usability issue because its AI is using it. So maybe now is going to be a renaissance in operating systems and network and computer,

languages and we should like go back to the old research and we should kind of start rebuilding systems that are secure by design. And oh, by the way, if we don't think that, you know, these things are safe to put out, we don't put them out until we have these systems built. And no, by the way, the AI is very smart so they can help us build it. And so I think, again, like, computer always like, you know how much of the stack we had to change for the internet? Everything. Right. Tell me about it. And you know how not vulnerable, how vulnerable everything was? Like, like, we could be in one of those moments like, oh shit, we got to rethink everything, and that's fine. We've done that before.

But I think that's a conversation we should have. So I agree. It's time to think about evolving these things. Well, like, look at, like, you mentioned earlier, like the booting a PC and getting a virus and 30 seconds or whatever. So if you look at how, like, you take an iPhone out of the box, which a lot of people are doing this week, you know,

what happens is the whole network is basically shut down except for getting the latest version of the operating system. Because it doesn't, even though it was pressed, you know, six weeks ago, You know, some zero-day thing has been discovered since. And so actually, the whole out-of-box process now involves first step, doing an update where the device can't do anything else,

and it can never do anything else until it's updated. That's the, like, there are all these benign, think of completely benign that we turned off in all of this desktop software of the era that used to be good things. Like it was having a macro for Word, so you could build automatic citations or something. It was like the super cool thing, until it became a virus.

We had a thing where you could put a CD in and it would just arbitrarily run a program. And so then what somebody did was like, oh, well, I'm gonna burn a clone of that CD and replace the program with my virus, but it's gonna look like the thing that's supposed to run and it's just collecting all this stuff and being evil. Yeah, I'd have disabled that.

And so one of the things that's happening right now is, there's a whole bunch of stuff that happens on, like you, on your own box court network that actually is gonna have to just change as the standard procedure, two-factor off five years ago was not standard in most places. Your whole SaaS world, like, your whole SaaS world, like I remember in like 2015 or so

when you would talk to a new company about their, oh, we're going to do enterprise, pricing, or whatever, and then they realized the first thing they had to do would go do Octa integration or Google Off because they could not have their own directory of how to manage it. And then that just became a thing. And there's not a SaaS program anywhere that doesn't just launch

with managed authentication. Right. And so there's just so many things that need to happen before you're even software now. Yeah, well, yeah, we're, I mean, there's probably every layer of the stack has to evolve a bit on this. Like even the, like, lack of granular nature of, like, you know, you have these modes of, like, the agent will either ask you every single time, you know, if you want to, you know, give a permission to do something or the exact opposite of, like, it can just, like, delete your entire computer. Right, right.

And like our OS probably wasn't built for the right level of granular set of tools you want to give the agent. We've kind of done a lot of work in the space because obviously like, you know, do you want to give an agent like your entire file system? Probably not. Right. Maybe in some cases you do. But oftentimes you want to have granular controls of like in this folder, you can do read right. And in that folder, you can only do read. And so how do you kind of make this all intuitive for the user?

So it's very difficult. And so there's going to like. Yeah, he just said is a very deep comment. I know, exactly, yeah. Which is basically the conclusion of 40 years of, right? No, 100%. It's like you actually can make it. But maybe with AI, you actually can.

Like, maybe there is like, you a way. Well, I actually think this is, if you ask me, like, walking in, like what I wrote down on my note, like was my biggest fear. Yeah. Is that Europe decides that GDPR was the best thing ever. Uh-huh. And they're going to just GDPR AI. Yeah.

And, and the AI will be fine. It'll be have one, it'll have one prompt for. when text that's omitted that just says, this vendor is emitting text and it's probably wrong, yes or no. That's going to be, because you can't really, at least in North America, you're not going to send your speech. In Europe, they still will.

And they'll have filters and keywords and blog lists. But then on any verb, any time that an agent or, you know, a background agent or a frontline agent touches a third-party product, I'm really worried that they're just going to say, we need a GDPR prompt on that. And it's going to be...

Oh, back to the user through the agent. Every single right or every single non-lookup becomes like a safety warning, like the airbag thing in your car. And the regulators love that because it's a liability assignment, and so it has this sort of legal precedent. And I really worry that that's actually

the middle ground where we're going to end up. And unfortunately, because the U.S., about 15 years ago, stopped leading in tech. antitrust, the problem is that Europe is going to lead with that because they have nothing to lose. Yeah. And so I don't want to be sad and down about it, but I just can't get out of my head that they love

prompts. They, I mean, look, I have to put in that browser ballot choice thing. Yeah. They, they, it, because it's the same, look, get into a new car. Yeah. Which I haven't done in years, but, you know, like, you're pulling stickers off, you've got, you know, you have all of these things. Yeah.

And someone thinks that that was success. And it's like, has anybody ever read, like, what is this if you're, if there's a baby in this seat? And it's like, well, that's relevant for some people, some of the time. But it's the- It's the entire fabric. It's attached to the seat. And they love that.

That is a very particular thing that they just love. And so I would say, if I were an AI now, the one thing I would be trying to avoid, and look, we added it. Okay, FINRA for AI, we'll create that standard. I mean, look, we had to put this. When the internet was new, the big thing was to download a program and run it. And, of course, if your machine is running an administrator mode, it was download a virus and take all your files wherever.

And so with Windows XP, which was in 2000, we added this thing that prompted you and stopped. Like literally, your machine stopped, user account control. And like, it was absolute assault. And we also put it in Word. The stupid little macro to help you write your thesis also came with a warning. Every time you opened your thesis, say,

this has my... Everybody would just... And so you end up in this world where just like with GDPR, everybody is numb. Yeah. And so then they're like, well, it used to be bigger. Although Mac, I mean, Mac kind of has found it. Nobody download software. That's the thing.

It's a very usage pattern on Mac. Sorry. I don't have like 10 applications of my Mac, but. Well, that's like 10 and then you're done. Yeah, yeah, yeah. But it's a lot of people still do all this stuff. Imagine if instead it was every time you go to a new website. Yes.

Which you do now. No, that would be bad. Yeah. Yeah. I mean, to start from here, but to bring it kind of back to the macro-bite, like, I wish this was the discussion we were having, which is like I feel like we've dealt with a lot of these problems. I feel like when we talk about philosophical X-risk,

we're not solving these very problematic problems. I actually think this is actually a constructive conversation to have. Maybe props would help. I don't know. And I think part of the problem is, like, people don't remember, like, you know, how unfettered access was and how bad it was and just how relatively benign that ended up being. Like, I even remember, like, there's at Stanford during our PhD.

I remember the oscilloscope was kind of janky? I'm like, what's going on with this oscilloscope? Like, it's a little slow. And I was measuring the network traffic. And it was like, more network traffic than you would expect. And I'm like, why is their network? I didn't even know the thing had a TCP stack. And somebody, like, broke in because there was an old version of Windows CE

and was running a porn server, you know? And so like. I had no idea that. It used to be. Just for the record, no. It used to be the case that, like, any time you turned over a stone, Yeah, yeah. So many had broke into something.

And, like, was it this, like, worst case, malicious, whatever? It was actually very rarely, even though the capability was there. And so if we could just somehow tone down the rhetoric and put it in context and you still are computer systems, and yes, there's very serious stuff. And people have definitely died because networks have gone down. Yeah. There's been real issues. But, like, and then can we just quibble about GDPR?

That would be amazing. But the problem is, like, that's not the discussion. It's not about GDPR and prompts. It's about species extinction. And philosophy. And philosophy and irrefutable things. And it's just not very soon. Like I think that that's such a great point.

And I think you hear people now talk about we regulate airplanes and we regulate cars. And you forget, like, well, the first cars were at the turn of the century. And unsafe at any speed was in the mid-1960s. Yeah, totally. And, you know, people have been doing selling pharmaceuticals during the gold rush. And thalidomide happened. 50 or 75 years later, and not even in the U.S., and then there was the FDA.

And, you know, people, the first pilots were flying, obviously, at the beginning of the 20th century, and it wasn't until the 1920s that you had to get a license to be a pilot, and you literally showed up with your own plane, and you got a certificate. It was literally no different than Driver's Ed is today. And then it was 20 more years

until they had anything to do with airworthiness and looking at your plane, but it was minimal. And then it wasn't until way after World War Proof that they got involved in, like, what you think of as the modern FAA. And so you're looking at 40 years of innovation, and they were not moving slow. I mean, have you ever seen that video in black and white

of, like, all the different planes that are crashed in everything? That was 20 years after the Wright brothers. And the FAA is incredibly effective. And, right, it's incredibly effective. It's the safest form of transportation, like, you know, and so I do think that this process is... It's also the slowest, most difficult form of innovation. And so if you start the F,

if you had started, started the FAA in 1910. Yeah. You never have... Well, I was listening to a Nick Bostrom podcast just a couple of days ago. No, no, I... It was interesting.

It was interesting. And, but no, but he actually makes his point. If you regulate AI too early, you actually basically don't solve anything, and you still just kind of had the same risk, ultimately, but you don't understand the systems well enough. You willed the thing into being, but you haven't figured out how to control it.

Yeah. We don't even figure out what it actually is. Yeah, like, there are new things coming out all the time and, like, casting AI completely differently than we thought of just six or nine months ago. And I think that that, like, all the innovation that's going to happen at the application layer is going to cause things to move in and out of the models in different ways. And, like, we, you know, we thought up until last week, I think, that, you know, text prompts and text coming back was going to be the best way to interact with.

Oh, that Jeff shows. And then Jeffs. It's so good, too. And then I'll talk about that. Why you find it so remarkable, yeah. Well, okay, so the way I think about it is, so, okay, so LLMs were kind of text in text out, right? They generate text.

And they came from chat, right? It was to communicate with a human. And we've spent the last few years trying to take this thing that spits out text and cram it into a traditional program, right? But traditional programs don't really speak text, right? And so then you end up doing this janky thing where you're like, in the prompt, you're like, Here's this demon.

Here's the scheme by out. But the thing is generating text, and it kind of ignores it, and it's just been super janky. And so what Jeff basically says, listen, you know, generating the text on the outside is a very expensive thing, but it's also kind of, you know,

it's more complicated than you need. So why don't we, we'll read text, and we'll have all of that kind of knowledge to read the text, but then rather than generating text, which is very expensive, we will just, if you give us a set of options, we'll choose the best option. We can do that incredibly, we can do it

incredibly fast, incredibly cheaply, but also we can do it with much more accuracy because we can train just for this. And so for all of the use cases that are not talking to like a chat bot, but are actually trying to put it in traditional software, this is a great fit. And so this has probably been the fastest adoption

of an AI model since chat GPT. It's just been remarkable because we're all primed for this. I actually want to pile on in this one, because I can't tell you how much I love seeing this exact form of innovation. And because what it does is it does, the thing that's bugged me from the very beginning, which is there's been no user study ever that shows, like, interacting with the computer using full natural language is efficient.

It's, like, literally always the least efficient way. And it's very simple. And it's just, like, ask yourself, how many people are asked, are really, really good at asking questions? And immediately, that's, like, less than half the people can ask a good question in a meeting. And then how often do you look at the answer and get really frustrated before it's finished, but you have to pay all this money to watch the seven paragraphs come out and then apologize that it's only a little bit. And so that's one, like having a different model. And then the other, of course, is my favorite, which is the output of it is designed for probabilistic programming.

And so instead of saying, like, is this customer service question, then route to customer service, otherwise route to general health desk or whatever, it's like, well, this is 80% customer service. And that's exactly simulation. And it turns out there's like 50 years of computer science research. in literally like probabilistic if statements. And so suddenly, the coolest place to be in computer science is gonna be in probabilistic programming,

which was like all of computer science in the 1960s and 70s. So it was basically how to, because all the computers started with doing math and it was all simulation. So it was like, let's launch the missile and hit that target, but it's windy, but wind isn't constant. So like let's model the wind and decide where to put the thrusters in order to do the arc.

And so most programming through, like, say, 1970, before it got to accounting was probably. And then we ruined everything. No, accounting, there's no probability in accounting. Right. But most programming was basically this modeling kind of thing. And so most programming language design was trying to figure out how to put probability

into if statements or into while loops. Like, do this until something happens, maybe most of the time. And like, so my first CS class, like, the very second assignment, or so was a simulation about like waiting online at a store. And I didn't know it at the time. I actually looked all this up when I was reading about Jeb, which was like the whole thing was my professor

was like wrote the book called The Theory of Simulation in like 1960. And I just didn't know that because it was kind of died by the 80s because it was all replaced by hyper. And so this notion of probabilistic and that slide deck you shared about the future, what's different about language models and stuff that you said was super good. Oh, Halper Flanks. This one is phenomenal.

It was a great debt. But the part that I felt was missing was that like, oh, wait, this is not all new. Yeah, yeah, yeah, yeah. Like, all of computer science was this probabilistic stuff. And so it's going to be very interesting to dust off all of that word because it's exactly what's going on.

Like, it's not an if statement now is if X percent, not if always. And so the way that Jeff worked is just to like, you basically, it's a custom programming language almost, which is here's the prompt, come back with a percentage. Yeah, that's right. And then you put that in the if statement.

But the consequential thing is like, finally we have a way to integrate these language models into a traditional software. And I think it's kind of funny because you ask the question, why haven't the labs done this, right? And it's kind of like a, like, not an indictment, but a reflection on how they think. Like, they're trying to create beings and beings speak. If you're trying to create God speaks to natural languages

or whatever, where this is really about something that's for traditional software, which is kind of not the direction that they've been taken. But one of the reasons the uptick was been so dramatic is because a lot of us software people have been trying to integrate these models into software. It just hasn't. So even before you get to the probabilistic,

like if I want a language model to drive an if statement, like it's really hard today with this model and makes it much, much, much, much easier. And then of course, this could change the nature of software fundamentally to make it more stochastic over there. Well, I think, but absolutely. And I think that what's so cool is that it is happening outside the models, because that's what I think

is just going to happen, which is the center of innovation is just moved. Yeah. And it's just, and now people need to, like, it turns out that the, outside of the lab, right. The platform providers, you know, basically reach a critical, a point of critical mass where the innovation stops happening at the platform layer. And then, you know, Apple, there's this famous expression in the Apple community called Sherlocking, where Apple looks around and the things from the outside world become features

and people complain and it's a, but that's sort of how the innovation works, because once you're platform, you're overwhelmed, no matter how many people you add, you're overwhelmed with just keeping the thing running and compatibility and stuff like that. And so I think that this is the signal that now people have figured out that there's innovation to be done to the model,

but outside the model. Guys, thanks for coming. This is great episode. Okay, great. Thanks for listening to this episode of the A16Z podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review,

and share it with your friends and family. For more episodes, go to YouTube, Apple Podcast, and Spotify. Follow us on X, A16Z, and subscribe to our Substack at A16Z.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only.

It should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z. fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com forward slash disclosures.