a16z Podcast - The CISO Playbook for AI Agents | Datadog

Episode Date: August 11, 2026

a16z's Joel De La Garza is joined by Emilio Escobar, Chief Information Security Officer at Datadog, to discuss what it takes to secure a company where nearly every employee is using AI and more than 4,000 engineers are working with coding agents. Rather than trying to block new tools, Emilio explains why Datadog chose to embrace AI early and build the security infrastructure needed to use it safely. They unpack how AI changes traditional assumptions around data permissions, credentials, developer access, and software supply chains. Emilio shares how Datadog uses role-based MCP servers and ephemeral credentials, as well as an AI "judge" built by his security team to evaluate the intent behind code and agent skills before they enter the environment. They also discuss why security teams can't afford to wait for commercial solutions to every new AI threat, how the relationship between developers and security teams needs to change, and why Emilio is less concerned about an AI "escaping" than he is about the sheer volume of vulnerabilities AI could uncover.   Resources: Follow Emilio Escobar on LinkedIn: linkedin.com/in/emilioesc Follow Joel De La Garza on LinkedIn: https://www.linkedin.com/in/3448827723723234/ Follow Datadog on X: https://x.com/datadoghq Stay Updated:Find a16z on YouTube: YouTubeFind a16z on XFind a16z on LinkedInListen to the a16z Show on SpotifyListen to the a16z Show on Apple PodcastsFollow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

The number one story on Bloomberg right now is that AI has gone wild. He seemed remarkably calm. The way I see it is if it's not an AI model, it's going to be somebody or something with actual malicious intent doing it. I do worry about what can the agents do, what tools can they call, what binaries can they pool, and also how do they get access to credentials. If a code is meant to solve the bug,

but it gets rewarded on that, but it doesn't care if it's actually doing something else. The tree is sick, so to make it healthy, it cuts it down. Out of a necessity, my team built a job. that evaluates the intent behind a piece of code. Like, is this thing meant to do harm or not? And we actually find quite a bit of malicious skills in all these marketplaces.

Engineers now, security is very much front of mind for them. Developers have always cared about security. The problem has been that the version of security that we want them to do is just crappy. My thesis 10 years ago was that security engineers have become real engineers, so maybe now is the time. AI is changing the security landscape on both sides. attackers have more capable tools, but security teams are also figuring out how to use those same capabilities to their advantage.

In this episode, A16Z's Joel De LaGarza sits down with Datadogs Ciso Emilio Escobar to talk about what he's seeing as AI adoption spreads across the enterprise, including thousands of engineers working with coding agents. They discuss how AI changes assumptions around permissions, credentials, and software supply chains. why simply blocking new tools doesn't work, and how security teams can adapt without becoming the Department of No. And as models get better at finding vulnerabilities, Emilio explains why he's less worried about AI hackers themselves than a more practical problem.

What happens when the number of vulnerabilities we can find suddenly explodes? Thank you so much for joining us. I know you're not here to see us, you're here to go to Black Hat, and it's good to catch up, and it's been a really crazy week. So you are a C-So at a public company, tech company, one of the House of Innovation, driven a lot of really cool technology out of it. And we've been talking to everyone about these models escaping,

these models hacking. And I know you've been central to kind of the adoption of AI at Datadog. And we'd love to maybe talk a little bit about, how are you thinking through the risks? How are you deploying this stuff? And I know you're very close to the product team and you're kind of enabling it. So it would maybe start off there and around.

For all it was like, we have to do it. Like any other software company, if we're not, we're going to be in the medieval ages real quick. And I like being as ESO. I don't like having my own goat farm. And we started real small. I remember our first install, we started with cursor, and I remember getting like 50 licenses. And it was more of putting out there and see who bit. And then it's not surprising.

It bloomed from there. So now we use every other coding agent that's out there. And some of the things that were worried about as we deploy, well, first you start with the data. what are we worried about the data? And I never subscribed to the idea of, well, if I block these things, nobody's going to use them. That's never worked.

So on the non-developer side, we just got chat GPT licenses for everyone, and it was like, whoever wants it gets it, I don't care. And you sure use it to find a recipe for pot rose. I don't give a crap. And because then I avoided the all oops moment.

And it was, this was back, ironically, this wasn't that long ago. This was two years ago, and it feels like things have moved 20 years since. But I remember having a lunch with a bunch of CIOs at RSA where everyone was asking me, like, how am I not blocking these things, right? And it was like, well, A, we can't because we do that,

the company that we are, but B, luckily we can just buy away out of that problem and zero data retention and all those things. Well, probably the inverse of Haylet's block it turned out to be correct, which is the people leaning into it the earliest and the most are the ones that you actually want to reward enough. Right, exactly. Yeah, yeah, yeah, yeah. So now we have over 4,000 engineers using them.

I mean, actually, we have every employee in the company adopted in some sort of AI, whether it's coding agent or Gemini, cloud desktop, chat GPT, or what have you. And so I have two, we have two verticals. We have the engineering side, which Alexi is deeply involved, our CTO is deep involved in,

and then I'm involved on the non-engineering side for how do we get marketing sales and everyone else to use AI. And the adoption is, I think we were at like 98% adoption rate, something like that. It's one version of AI or the other, everyone is using. On that side, on the IT corporate side, the data becomes

more of the paramount issue, permissioning and all those things. One thing that AI is going to do is you think you have proper controls of who get access to what Google file or what have you. AI is going to find a way to get it. All you have to do is prompt it.

It flattens the organization. It flattens the organization quite a bit. So we actually had a moment where internally we built this business intelligence tool. It's really neat. And it was starting to show some data that I was like, It wasn't anything like a fire alarm, but it was like, should we?

And the data was probably always available to everyone. They just never took the time. So, yes. So it's data that is in a data warehouse. And we thought we had to write table and roll permissioning and all of that. But there was always a way to get access to the data if you knew the SQL well enough. Well, then what happened is that you went from only people who really know SQL are using this to now having a sales rep telling, and asking like, hey, what are, what is, I'm a commercial sales rep.

I'm just coming up with an example. I'm a commercial sales rep. How is the enterprise deal, enterprise tier team doing? Yeah. Which normally you want to keep those things separate and what have you. And the agent just took out the sequel to do it. Yeah, yeah, yeah. But he tells them what the comp plan is.

Right. So what it's not always find a way around the permissioning and all that is like we had the right permissioning for the right time. Yeah. And then the age, so we've been obviously improving all of that. And then luckily my IT group is what we've done is, Now we're at the point where we have role-based MCP servers as well. So we have an MCP server for SDRs and things like that.

So we control and govern that data. And then we're like, let them have it with whatever tool they want to use. On the engineering side, obviously, in the coding agent, like when my security team is worried about, similar what you were just talking about is on what can the agents do, what tools can they call, what binaries can they pool, what dependencies can they use, and also how do they get access to credentials. So for that, we have a few things going on where we've contributed to these open source sandbox and expanding all that where the agent doesn't actually get access to any sort of credential

file. So even if you have a file in your home directory that has your LW secret, your NPM published secret or what have you, the agent on the sandbox can't touch it. The credentials get injected into the agent the moment it needs it. And the beauty of it is we already have CLI tool that will give you these ephemeral tokens. do these things, we just have the agent not know how to call it. So it's like, oh, I'm a million, I'm an engineer on this team. I want to authenticate to GitHub, for example.

I call this tool off GitHub, and it gives me a credential that's valid for it. I don't know how long. And it gives me my access to GitHub, but now is the agent getting my access to GitHub. But not, there's a static token written in a file. Yeah, yeah, yeah. The second thing is we know the threat has expanded. Developers are now the main target. for attackers.

Absolutely. Because if I get one of these tokens, I can then build a worm that attacks packages or I can just escape or do whatever or access your production environment. The other ways that we see how these agents can do it is via skills. We track a few of these marketplaces.

How do you control the skills? That's a really hard problem. So we actually have built hooks into the agents. And then one of the things that, which we can talk about is My team built out of a necessity a judge that evaluates using AILMs, that evaluates the intent behind a piece of code. I'm not talking CVEs, vulnerabilities or anything like that.

Is this thing meant to do harm or not? Harm being a vague term on purpose. It actually does really well as scanning markdown files as well. Nice. So we built it because for the longest time, Datadog, for those who don't know, we have a Datadog agent. we take third party code contributions to it things like integrations and what have to you

so code rid of my other people we had this process for the longest time that requires a security engineer and an engineer on that agent team to revaluate the code and then approve it and then merge it so I'm like well this thing can't scale so then we build this judge by we I mean my team I don't do anything

that got really good at it and then the software supply chain hijack started happening and ID extensions started happening and then we threw this thing at those packages and it was actually able to identify the malicious piece of code that was injected or introduced during the hijack. So we're like, huh, I wonder how it would do

with markdown files. And it actually does pretty well. So then internally what that means is we have hooks into the agents, we know what skills they're getting pulled. I've never believed in a security program that just restricts everything because the pain, innovation, and all that.

It's the empire of no. It's the empire of no. But also it's like we don't want to be free willy either. So what we do is now we put this judge in front of every skill that wants to get introduced. We actually find quite a bit of malicious skills in all these marketplaces. We're working with a few of them partnering to see if they want to use this judge, but also like, hey, giving a heads up of, hey, we found this one, we found that one, we found that one. And they do a pretty good job at taking nose down.

The other thing we worry about is what dependencies he can pool, what binaries he can pull. So I think it's plenty of it out there right now. I don't think we have to justify it, but the judge does a good job at it. And then evaluating the code output of the agents against malicious intent. I think intent, and it's funny how things, how this industry works,

and this is maybe like where I can get a little cheeky here. If you allow me is... I have no control. You can say whatever. Yeah, yeah. Apparently intent now is one of the must have in any AI security type of solution. Mm-hmm. but the interesting thing is like it happened after we've had conversations about the intent anyways

but apparently that's a big deal right it's it's like the fact that these agents obviously they're trained on existing data on existing code and they have a reward structure and they have a reward structure um so if a code is meant to solve the bug but it gets rewarded on that but it doesn't care if it's actually doing something else yeah inside outside of that so we will worry Three is sick, so to make it healthy, it cuts it down. Right. So we were worried about like the typical thing that we read about of an agent producing code that either would have, oh, you know, your database isn't scaling and is paging people at 4 a.m. So the best thing I can do is just turn it off.

And therefore I solve the problem of paging people at 4 in the morning, right? So you have to be careful how you prompt these things, but also how it actually interprets your problem in execution. So we have this judge now evaluating the code output of the agents to then make sure that we're doing there. So the cool thing is that we do this internally because we have to and then the product team goes like, wait a minute, that might actually become a thing. Let's explore and then we evaluated with customers and all of that. But I mean, I get to say I'm lucky that the team can do it. one of the things that it surprises me I started at a roundtable

last week about agentic security and the sense that I got from a bunch of the security leaders who were in that call was a sense of helplessness of just waiting for a commercial solution to come in and solve it all.

Well, you're... Yes, I think that the profession is interesting in that there are a number of different CSO profiles, and I think you know, I say this as someone who was relatively technical for a C-SO, like before maybe just the generation above us, like there were no technical.

I mean, for C-So is Steve Katz, right? So that's, like, pretty recent. Yeah. You know, and I think that there was just a lot of, like, sort of like CFOs becoming C-S or C-TOs. Yeah. Yeah. So, like, I think, and then what happened,

I think is that security teams got quite big and you needed a manager, right? Not necessarily a leader or a thought leader. And so I think there's just a lot of that right now in the industry where there's very much a, you know, if you're a large industrial company and software is not your competitive advantage, you're probably going to buy most of your product. Right.

Versus like data dog. Yeah. And we're like, you better be the best of the best. And it's totally fine, right? But I still believe even in those scenarios, you may have one or two people on that team that if you give them creative license, they could do something. but the fact that that wasn't even a thought process

worries me a little bit because this thing is moving really, really fast. Naturally, like, a lot of the companies that are solving this are new companies. Yeah. You know? I think it's really hard. And this is, this is, I mean, I think the talent shortage

is still a problem. And I'm generally bullish on security jobs because of it. But, like, I think for some of the bigger programs, like, it's really hard to get technologists in there because they are shops. Well, here's my kind of argument. Do you really need a security person to do it? No, and I think some of the best security people didn't start a security people.

Yeah. Like, you know what I mean? It's like the developer that found their way into security. Right. Yeah. But also in this company, like, I'm sure there's a developer somewhere in there that they can be like, hey, how about we half you solve this problem for us in the meantime? I think that a realignment is kind of happening. So I was talking to a family member at a big, big American corporation.

And they were saying that sort of like with AI tool adoption, they don't need. to write as much code manually anymore. They still need engineers and like that. But like they're seeing, they're basically saying like, hey, look, we don't need this many tier one engineers and the security team needs them. And so I think maybe you're starting to see some of that cross-pollination where you'll see that talent find their way into security because it was always the case that if you're a developer and engineer, like you would just make more money on the software development

side. So like I think those are equalizing and maybe, I mean, my thesis 10 years. years ago is that security engineers will become real engineers. So maybe now is the time. Yeah, yeah, yeah. I mean, we pay them the same as software engineer. Now we do, yeah, yeah. I mean, in the valley especially.

Yeah, yeah, yeah. I think it's narrowing in large corporates. I think it's still software engineers have the advantage there. But that'll probably narrow over time. Yeah. So that's all we'd be thinking about. Identity security is like what are the things that we actually worry about? And then what can we build?

What can we use? And then where do we find? find more synergies there. And I know that's a very buzzy term. But what that I mean is like our devX team is also interested to understand what's happening in the agents because I want to monitor developer

experience. So why can we just all together work on it? So that's how we that's how they think has grown because it's not just a security tool for the sake of security. It's also serving all their purposes. And I've noticed with engineers especially over the last decade, like engineers now, security is very

much front of mind for them. Yeah. If only because they don't want to have to keep fixing things. And so, like, I've noticed that there is this front of mind security thing for engineers, which is great. It's like I think we've made a lot of progress in that regard. Yeah. And I, well, I actually have a little bit more of a maybe fire a take on that. I think developers have always cared about security. I think the problem has been that the version of.

security that we want them to do is just crappy. Yeah. Which is like go fix this thousand of things that none of them are actually relevant to what you're building. Yeah. But the scanner told me they're critical, so you have to fix it. Yeah, there is a lot of like the, you guys have sent me so much junk. Lost talk credibility.

So one of the things that I get to do at Datadog is actually I get to meet a lot of the engineering side of the equation talking to us about our security platform and products and how do we get their security team to use it. And they told me the same stories that I'm sure you've heard of, like I get a thousand tickets, none of them are relevant, security doesn't know what we're doing, there's no intent to even understand the systems. And the number of times I hear a security person telling me that their developers write crappy code. It's just mind-boggling because I'm like, well, then you do it.

Yeah, exactly. See how good you can do it. Probably no better. Yeah, exactly. Or maybe it's good code, but they're not fixing your, noise vulnerabilities and therefore you think is crappy code.

Yeah, I mean, oftentimes, right, with code bases, this was the learning experience for me, was that, like, you'll find these, in isolation, you'll find these issues in code, but then, like, up, there's a framework, a layer up that's actually mitigating any potential fallout, and so it's like, whatever, it doesn't matter. Yeah.

I mean, I just think, I think, yeah, security people tend to have an issue, a blind spotter on some of that stuff. Yeah. I guess, I guess sort of, like, as we sit here, you know, the, the, the number one story on Bloomberg right now is that AI has gone wild and the world is over and everything is getting hacked. You seem remarkably calm. So maybe before we go, like, we'd love to get your take just on sort of like, you know, supposedly there's this great white shark cruising out there

just eating people without the knowing. But in general, like, I just haven't noticed a sense of panic from CSOs. Like, what's your take on all this? I mean, the way I see it is, it's, it's, if it's not an AI model it's going to be somebody or something with actual malicious intent doing it. So therefore I don't panic. I do

worry about our access to those same capabilities. I do worry about the lack of regulatory framework that's deciding who gets access to these capabilities. Yeah, yeah. I feel you on that.

that's sort of like, it's really interesting who's getting access to this. Yeah, yeah. And it's sort of like, hmm, what's at play here? But then, and then it's clear that is, or, or, or, or, or, or, or, or, or, or, or, or, or, or, or, or, portrays it as clear that is outside of the frontier laps hands. And, uh, so, like, we have to put our company name on the list and then we're supposed to wait. Uh, and then when I ask, what's the criteria for how these things are evaluated? I get jazz hands response.

Yeah, but then you come to a conference and you talk to your buddies, and they've had it for nine months. Yeah, yeah. And they're making stuff that's way less critical than your product. Yeah, yeah, yeah. So our argument was like, hey, some of these companies are using our stuff.

Yeah. Wouldn't it make sense for us to get access so we can protect the stuff that you're using? Exactly. I'm sure there's a lot of government agencies running data dogs software. And vice versa, right? We have a lot of also companies

that provide services to the government. Yeah, yeah, yeah, of course. So, yeah, so I'm not worried necessarily about the models escaping Sandborder. and finding old days. Like I think, actually, I actually think inherently it makes things better for us. Mm-hmm.

100%. Guess what? We're talking about it. Yeah. Bloomberg is talking about it. You just said, right? So before, we were like, well, nobody's talking about it, so who cares? But am I worried about one of those things hitting data dog and something happening? It's like, well, it's either a model or somebody else, right?

Like, if we have a gap, we have a gap. Yeah. At some point, it's going to get discovered. So I'm not too worried about that. What I worry about is the volume, of things that are going to have discovered. And the things that are telling us how to handle that volume are not up to part. 100%.

So you must fix every CVE that's in your environment to now you're going to get 1,000 X more CVEs in your environment. Well, those two things don't necessarily align well. So I'm hoping that side also changes to the current day of things. That's what I worry. And then the, sorry, and lastly, like the only thing that I worry is the, hypersensitivity to the security of these findings because of the fact that it was found by a model

rather than a human. And I'm already seeing some of that where like, oh, Model X found this, so it must be extremely critical and true. And then you're looking at it and you're like, eh, it's actually not. But it's kind of hard to argue

that it's us arguing against this thing just because a Greek god found it doesn't mean it's actually a critical thing. But that's, I think that the third-party risk management game is going to get really crappy, even more so than it is, because of this thing. So I hope eventually those, so those things I do worry about. But I'm not always escaping and hacking companies and all that is like, well, I mean, they were going to get hacked no matter.

There's always been hackers. Right. There's two more now and whatever. Yeah, yeah, yeah. There are two really good ones, but hopefully they make us better rather than like making us work. Well, and I think the lesson that I have learned throughout my entire career is that, any attempt to gatekeep security fails. Right.

Yeah. And it only makes things worse and that an open and free discourse on these things is the only way to get better. Yeah, yeah, yeah, exactly. Thank you for joining us and having that discourse, man. That's awesome.

Yeah, thanks. Enjoy your time at the conference. Thank you. Thanks for listening to this episode of the A16Z podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating or review

and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X, A16Z, and subscribe to our substack at A16Z.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. Should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the

companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com forward slash disclosures.