AI Can Write the Code. Can It Run the System? | Grepr CEO Jad Naous
00:00The world that the AI lives in is all digital, and there's the actual world.And those things are not necessarily the same.When we started doing AI for coding, it was autocomplete.Nobody really knew to what extent you can actually build complex software using AI.And then it basically broke through the expectationsbecause we were able to model more and more complexity in the systems.I just don't know if operations is constrained in the same way.Where is even the data to learn how to build a system that understands all of the operations?
00:30My assumption is that we have all the data.The question I think is maybe about complexity.And to what extent can you take the things that you've described that are operationaland convert them into something that an AI can reason about?The dirty secret for the longest time of running like a distributed cluster was just human beings managing it.Do you think we're at a spot where you could like use AI to run one of these services?I think that there's enough to be able to...AI can now write remarkable announcements.
01:00of production code, but writing software and running it are two very different problems.In this episode, A16Z general partner Martin Casado sits down with Grepper CEO Jab Nouse,a systems engineer Martin first met when they were both at Stanford, to ask what happensas AI moves from coding into operations. Jad explains how his 10-person team uses coding agentstoday, why he thinks the strongest engineers are getting even more leverage from AI, andhow work that once took months can now take days.But the harder question is what happens after that code shifts.Production systems face unpredictable workloads, failures, and an enormous number of possible states.
01:39Martine and Jad debate whether AI can learn to troubleshoot those systemsby detecting what changed, forming hypotheses, and running experiments,or whether operating in the real world is fundamentally harder than generating code.And that leads to the bigger question behind the conversation.How far can AI go when the answer isn't already campaigned in the world it was?trained on. Jed, thanks for joining me on the podcast. For those you don't know, Jed is the CEO of Greper,but also we went to grad school in the same group. And when I first met you, we had a new student.I walked into the office and there was a motorcycle helmet and a pack of Marlboroughs. I'm like,
02:24this guy must be unbelievably cool. So my question is, what happened to all your cool?Question number one.Maybe just for a context, maybe give a quick overview of your background.So just for those that are listening,Jed is one of the best systems thinkers, I know.I love talking to them.We're going to go kind of to a deep dive of engineeringfrom a system's perspective and how AI is impacting us,
02:49especially when it comes to things like operations.But just to ground it, maybe a bit about your background.Sure.So you met me at one of the peaks in my coolness.Yeah, you're pretty cool.motorcycle and marlbrose cigarettes.I remember thinking of like, wow, are cigarettes even illegal?It was like so discordant.
03:12I remember Nick seeing me one day smoking outside of Gates and looking at me and going like,Nick Buchanan is our advisor.Gates was the building, yeah.Actually, one of my favorite thing.Do you remember Balaji Prabhakar, the professor?Of course.My favorite thing about him was that he smoked.I was like the only professor.
03:30I thought it was like great.Just to be clear, I don't smoke anymore.No, too old, too middle-aged for that now.But let's see, so I went to Stanford with you.You were always the guy I had to look up toand you were like, hey, this is the guy I want to be,you want to grow up.Because I started four years before you did.
03:49Yeah.And so you were four years ahead of me.You graduated a while ahead of me.You went and started Nysera.I stayed and stuck around, worked more on OpenFlow,helped build a bunch of stuff on the hardware sidefrom a security perspective, actually.One of the most fun projects that I worked on there
04:06was I built the first set of SHA-3 implementationsthat people were considering using,and I built them in hardwareto kind of show whether they work or notand whatever for one of my projects for my thesis.That was fun.And then I decided after a whilethat I actually don't like networking
04:25because it's just about moving bits from one placeand putting them in another way.How hard can that be?He has to either be philosophically aligned that is deep or not.Most people assume it's not.I think it's very deep.I mean, there's something interesting about it,but ultimately, to me, that's like it's not...
04:47Yeah, it doesn't do it for you.Yeah.And then I actually went and did a postdoc at MIT with Nikolai Zeldofich,who's also amazingly deep and super smart.Yeah, he's phenomenal.I did some application security work at the time, started a company based of some of the research I had worked on with Ramash Chandra over there, who was a PhD student at the time.And ultimately, technology was too early.So what we did was, well, ultimately a lot of the work was actually really Ramesh's.
05:14And his PhD was on, well, hey, if you get attacked and you get exploited, can you recover from that by rolling back the state of the whole world that's related to that application and then roll it over on top of a.a patched version of the application to remove the exploit.Yeah.And so we started doing that in the real world.And we're like, that sounds really hard.It's actually kind of relevant to what we're talking about.Yeah.To what extent can you actually bound the state for like operational systems,
05:45which I would argue is very hard.But yeah, we almost got it working.Almost.I feel like for all long tail disciplines,this is why people get into fly track for smart people.It's like you see a bit of progress, but your ass-emotting.But anyways, I digress.Yes, yes.
06:02So then I joined App Dynamics.I was there for about three and a half years.They went from about 200 people to 2,000 people, got acquired by Cisco.I was a Cisco employee for one day.And I said, and then you're like, Chad, you come work with me.Come do this VC thing with me.Yeah, yeah.I'm like, what the f f.
06:21Yeah.What is VC?What is, I'm an architect.I do engineering stuff.Yeah.He's like, great.Yeah.The less you know, the better.
06:28Yeah.Literally what you said.That's true.And then the first day I started here, you said, slow down.It's actually people that move from, like, actually in operating role, the VC, it requires a lot of reworking the brain.Yeah.Yeah, just like going from a PhD to operating, by the way.You have to undo the brain damage.
06:48This is what real world is like.I know.And then I stayed here for a couple of years, did a bunch of work with you.It was a really fun time.worked with a lot of great people like Jennifer, Mark and Ben.Satish.Satish.Matt was here when I first, almost towards the end when I was leaving.
07:07Yeah.And then joined the Pli, their chief product officer,helped them build their engineering product teams,kind of scaled from about 15 people to over 90 across product,engineering, infosec, and design.And after about three years,I kind of had a head for each one of these pieces that I owned.and I kind of was sitting around just coordinating and doing meetings all day
07:29and wanted to go do something else.So left and then started talking to a lot of engineering leadersabout the problems that they're facing and what they're trying to do.Everybody at the time, this was in 23, everybody was saying,hey, we're trying to really spend.And ask them, what are you spending money on?Everybody would say, first was infrastructure and then observability.And so I started looking at that problem trying to understand, like,
07:51why do people think it's too expensive?It turns out there's a few things.first, people were spending somewhere between five and 20% of what they spend on infrastructureon observability, but ultimately what they were really trying to spend it on was reliability.They wanted to buy reliability.And of spend, like predictability of spend.Reliability of their infrastructure and applications.I see, I see.
08:14Right?And so they were putting money into observability as a proxy for reliability.And for the longest time, the more money you spend, the more reliability you got out.But as architectures changed,turn into microservices that are much smaller,that are much more vertically integrated,the volume of data that's being collected at every layer expanded,there was much deeper market penetration for observability
08:36and kind of the volume of data that you're collectingand the cost of observability was actually starting to grow fasterthan the units of function that you're outputting.And the marginal rate or like the marginal increase in reliabilityfor every additional dollar you were spending on observabilitywas just flattening out.And so that's why people thought it was expensive.But people were still willing to pay
08:575 to 20% on reliability.And so the question was, can we actually solve that problemand bring back that increase in reliability,that marginal increase in reliability,not necessarily by spending more money on observability,but some other way.And so the thinking was like, okay, great,if we can figure out what is the signal,
09:17what is the noise, can we actually turn the observability datathat you're collecting,that's mostly fluff and mostly,and mostly insurance at this point,and do that kind of extraction of information upstream,do whatever you need upstream,and be able to ultimately massively reducethe cost of observability downstream.
09:35And if you can extract that signal in real-time upstream,then you can, one, reduce the cost,but two, you can now feed it into other systemsand use it for driving reliability operations afterwards.And the distance at the time,there was something that was called AI coming into the picture.And we're like, well, great.If we are in the data stream and we're extracting the signal,
09:58then we are in a strategic position to feed that to whatever is going to come in the futurebecause that's going to be necessary.And so that's what we built.And so Grepper helps companies.We started with helping them reduce their spend on telemetry massively.And so you put us in the data stream, plug and play, pass the data through,reduce your cost by 90%.And now we're building out that.
10:22second layer where we're feeding those signals into the AI layer that's now going introubleshooting and figuring out when things are going wrong and fixing them.I love to just talk through kind of your experience on how AI is changing dev generally,which is like a lot of people could ask this question, a lot of people answer, but I just feellike you're kind of like the engineer's engineer. Like you've been an architect, you've run product,you've run engineering teams, like, you know, like, how big is your company now? How many people?We're a ton.Yeah, so you're building a lot of software with a relatively efficient team.
10:56So maybe just a high-level sketch on like, you know, if you're a senior engineer like you, you know, that has architectural experience, like, what sort of gains do you think people are seeing?Maybe talk through like how you use AI, what gains you're actually seeing, what is the stack, what is your process, how you think about.So maybe I can give you like a historical view of like what's been happening in terms of like, right.running a team and so on.Yeah.And so, you know, when we first started AR, we're still kind of auto-complete.I remember, like, at the time, we were using co-pilot in Intelligene.We're like, oh my God, look, it's auto-completed these tests.
11:35Yeah.And we're like, yeah, but it's just autocomplete.And very rapidly, things started evolving.And what I've noticed is that really good engineers are able to somehow put it into theirworkflows in a way where they the AI starts training them oh interesting in terms of like how totalk to it yeah how to get the best out of it and what I know by the way as an aside do you thinkthat's model specific like is it do you learn how to use AI you know writ large or do you get efficientat a given model I think there's like some nuances that different models bring um I'm sure you're
12:17looking at your tweets you'veThere's like nuances, but there is a way to work with AI.And like, I think that there's also a way to work with different harnesses.And I think like, you know, we tried, for example, using open code or like different harnesses or like using codex versus Claude.Like, Claude code, like different ones have different ways of managing context or thinking through problems or when to spin up the right subagents and what to tell it.like the kind of prompts.Like, for example, with Claude code, I find that if I don't tell it,don't spin up, you know, 20 fable agents to do basic research.
13:00It'll go and spin up 20 fable agents to do research and burn all your tokens.So, like, a lot of these things, kind of you start learning about the harness and themodels and, like, how they interact with each other and then how they can interact withthe rest of the world.But I digress.I'm just going to, in my mind, like, this is kind of what I started noticing is that, like,some people were adapting and learning and kind of doing it that way,and others weren't as adaptive.
13:28And then as we started hiring more people into the team,that disparity between, you know, okay engineers and really good engineersjust multiply 10x.Can you predict who will, could you predict who would be good?Was it like good engineers got better and bad engineers didn't?Yes.Yes.It really, okay.
13:49Yeah.So, like, if you're a good engineer, you're just a better good engineer.It's not like there's some like bad engineers,all of a sudden became good engineers.No.Okay.No, no.It always ends up being like you have to really understand ultimately the setup.
14:02Because the more complexity that your brain can handle,the more you can break down the tasks and kind of assign them.I think this might actually be like non-consensus opinion.That, like, I mean, I do hear it.Sometimes, but the idea that, like, the better engineers, like, are much better.Those aren't less.And you found this to be pretty universally true?I mean, look, my team is still small.
14:30Yeah.We haven't done it across 100 people.Yeah.In my experience, for the things that we do, that is the case, because we build distributed systems and distributed algorithms.And, like, your brain has to handle the complexity.Because, like, AI doesn't do it.Right.Yeah.
14:43Yeah, yeah.And so for the problems that we solve, none of the models can actually think through the things that we do.Like there's a lot of, for example, we have a query translation layer.We were just talking about query optimization.And so you have to really understand the semantics of the query and how to get to the semantics of the query and then explain it to the model and then have the model go and build the translation layer, let's say.Yeah.So for the people that are listening that may not be so deep in computer science, what do you mean by semantic?So semantics are really the meanings of things.
15:22Like the semantic of a dog, of the word dog, is the animal that actually walks in four legs and barks.And the syntax is the fact that it's just a three-letter word and it has three different letters.The shape of the word is different.And that shape can be different between different words.but different words can actually also mean the same thingand so they might have the same semanticsbut they might have a different syntax.So a dog and a puppy might be slightly different semantics.
15:57They refer to the same.But they're so close to.So just for like computer science,the way that I describe it very simply is syntax,you know, it's like the characters you typeand semantics is what the machine actually does.Right. And so it's like the actual runtime properties of the system.So you did computer science, I did electrical engineering.
16:21You became a computer scientist.And I became a VC.That's the problem. I see.Yeah.Anyways, for people that are listening, I just want to understand.Like there's the actual syntax, which is writing the code.And then when you're talking about syntax, you're just talking about the code.syntax, you're just talking about the code itself.
16:42Then when you talk about semantics, you're literally talking about you have a machine thatinterprets that code whether it's compiled or not, and that machine does something, and semanticsis what the machine actually does.And these things are clearly very different, right?So, for example, some semantics you can't even derive by looking at the syntax.You don't know what the machine is going to do because whatever.You kind of have to test it.You have to test it, you know?
17:02And it can go all sorts of different ways when you test it.And like a very classic version of this is SQL, for example.Like, you can write, you know, some SQL query,but, like, how it actually performs,how long it performs depends entirely on, like,let's say, what the schema looks likeor what the data looks like in the database, et cetera.But that's, I would not call that semantics, though.
17:26I would call that performance characteristics.Oh, sure, yeah, I think that would be like...Like, I would say, like, the semantics part of that would be, like,depending on, like, what the shape of data isor, like, what the configuration,you might end up with different semanticbecause like you said,you want null to be treated as false.
17:45And you won't know that unless, like,you also know the configuration of the system and so on.Okay, okay.So for the purpose of this conversation,I think we should just say that there's like the actual code you writeand there's how the system behaves,and we'll call the second one semantics just so we can have a conversation.There's a...
18:03What do you want to call?We need to dive into the semantics of this and make sure they're clear.That's right.Because there's a lot of...I know, okay, okay.I know.Let's keep it at high level.Okay, so let's...
18:15Sorry, so let's go back.So, you know, actually, before we started the podcast, you and I were just shooting the shit,and you were talking about, like, how you can do products, projects now that take a weekwhere in the past would take a team in months.Can you maybe walk this through a little bit about, like, how you personally kind of...Evolved than that.We'll do projects and then kind of the gains that you see.So, like, a little bit about, like, your stuff.
18:40stack in your process.So I use plot code.I'm going to say something that's actually pretty funny,but I use Windows.What?Was that created by this company Microsoft?And I use WSL.So I use Linux for everything,
19:04like development and so on.There's no like X or UI for it.I just use the command prompt.But how do you use Windows?Because I just never really wanted to learn the keyboard shortcuts.Are you a gamer?It's like you tab over to Counterstrike or whatever.Like, why else?
19:22I'm not hitting on Windows.I mean, you know, I used it in the 90s.That's really.Like, the only reason is I never wanted to learn the shortcuts.I hate the one-button mouse.I hate the one-button stuff.I hate the number of modifiers.You have to type for shortcuts.
19:39and stuff like, I just never,I'm like, why, why? Why?Why waste time?Okay.So I like, yeah, I'll install Linux.And so WSL, I use cloud code.What is WSL?That's Windows subsystem for Linux.
19:56So it's a, they actually run the Linux kernelwithin Windows in a hypervisorand provide you access to it.It's great.They've done a great job.Great. Windows, WSL.And Cloud,code. And like my latest, the latest thing that I do is I use Opus 5.5. I tell it, which is a really
20:18good model. It's an amazing model. Really good. And it's, I've shipped in the last two weeks,these massive 20 PR things that I just dump on my team to go and review. And they're like,I'm like, don't worry about it. It sounds horrible. It's like the new reporting structure, like thepeople below review the PRs for the CEO.Is that where we've been?I was like,no, everybody's like writing the same thing.Oh, everybody's just writing.
20:46Yeah, I review my people's PRs too.And like, everybody's like just, just PR is flying around.Yeah, and, but like the thing that has changed a lot in the review cycle is that like we started much more focused on,to focus much more on like the tests.Sure.And like doing things like golden expectations and like, are they shifting in different ways and so on.But Opus 5, and like the trick that I've recently learned about is that instead of using sub-agents,you basically tell Claude to spin up command line sub-agents because those maintain their persistent session over time.
21:24And so you can have it spin-up these agents separately.You can actually run them from other sessions and so on.And that allows you to be much more efficient from a context perspective.How many agents do you run?How many?How many?How many what?agents.
21:39Typically, I work on like three or four session windows at the same time because I foundthat more than that, I just start losing context between the different things because likethey start finishing faster and then I, then they wait.And do they interact via the, do they interact via the PR?Is that how it works?The sub-agents, like together?Yeah.No, they interact directly through the command line.
22:05Oh, wow.Okay.Yeah.So like, I asked like.the main thing that does the implementation to have a tech lead reviewer.And like they basically ask it questions.Does this make sense?Does this not make sense?
22:17Review my PR, et cetera.And so they iterate on these PRs together.And then they have a PR doc that basically they work on with a design doc and everything in it.Like the first thing that I do is I ask that agent to do it, ask the tech leads to review the PR doc.I go through the PRD.I go through the design doc, make sure it makes sense.go through each piece in like detail,make sure that all the algorithms work
22:41and like all the pieces work,the division of labor works,the division between PRs and their relationships work.And then I tell it just go start doing it.Figure out if there's like paralyization that you can doand then spin up other agents to actually go do them.But ultimately you have to kind of put it all back in one stackand be able to go through it as a human one by one
23:04because otherwise it just getsmessy and bushyand you can't really tackle it.So how much time?So can you give a sense of likea project you do now versusI don't know, six years agolike time differences?
23:20So the latest projectthat I shipped was something like...By the way, this is like, to be clear, this isproduction code. Yeah, yeah, this is production code.So whatever. The statement you're about to makeis the gains that you get for like shippingproduction. So I'll give you the example of the lastthe last PR that I worked on, or like the last set of PRs,
23:39was what we call auto-promotion in a data lake.So in a data lake, you store all the data,and typically the data is column-oriented, parquet files, and iceberg, and so on.And when you get data from logs, they're semi-structured,they're arbitrary columns and so on.In order to have the best layout of the data,and to be able to query it efficiently,fast, you kind of, the ideal situation is that every field in the log event becomes a column
24:10that you can filter on independently of the rest of the rows. And so we call that auto-promotionor shredding. And doing something like that is incredibly complicated, right? Like, you have to goand figure out how does the metadata system work, how do the commits of the metadata system work,how do you coordinate across different writers and so on.And that took maybe about 10 days.In previous days, that would have taken maybe about six months for a single person.Wow.And was this just you?
24:49That was just me.It is insane.Yeah, sort of like factor of 10, factor of 20 improvement.Yeah.For production code, for when you're a production code,for new products.Yes.You know, one question I always have is like I can very much understand,
25:10you know, these models are trained on all the code ever written, let's call it, right?So I can understand why they're able to generate code, which is the syntax piece.But then, like, for, you know, you're doing core infrastructure.Let's say you're doing distributed algorithms.Like, like, the properties, like how it handles itself when it's running is very, very different.Like often you just can't even look at the syntax to know what's going to happen, right?Like a lot, let's say you're, I don't know, let's say you're writing some distributed system.And then like the way that it will perform the availability, the correctness even is going to be dependent on what the workload looks like.
25:54You know, like what are the access patterns?What is the traffic?Are you talking about like the performance aspects of it though?Or are you talking about like the semantic?availability, you may have to lock things,so it's not available for a period of time.I'm talking about correctness.Like, maybe you've decided to do eventual consistency.
26:11And so, like, if you do a read,it's not going to be consistent because there isn't to lock.And then performance, like, you know, maybe you cash,maybe you don't cash.So these are all kind of tradeoffs in the design space, right?Yeah.And, like, unless you know, like, the access patterns,you won't know how it's going to perform in these things, right?
26:28And so I just think to myself,okay, I've got an AI,I can understand how it will write.you know, like code for a generic distributed data store, for example.But I don't understand as how it will know to like make the tradeoffs needed because it doesn't even know what the access patterns are.And it's not like these things can like, like, where's even the data to learn how to like build a system that understands like all of the operations, right?Like today we rely on human beings to like manage clusters and to like, you know, optimize databases, etc.And so can you maybe talk about how you think of?
27:03Like, I get the writing the code, but like you have these deep operational systems.Like, how do you apply to that problem?And like, hey, have we made any progress at all?Or is this still an outstanding problem?The question I think is maybe about complexity.And like, to what extent can you take the things that you've described that are operational?And like that happen in production and convert them into something that an AI can reason about.And...
27:28Exactly.Like, how can an AI reason about the system?well, A, first off, like, to what extent do you see that it writing in the code that it can actually predict what's going to happen?Like, how good job does it do with that?And the second one is like, okay, now how do we make it reason about this?So let's ask the first one.Like, if I use your method, the Jad method of writing code, and I built some big distributed system, like...How often does it actually get it from performance correctness and stuff?
27:55How often is it, like, actually operationally, like, pretty sensible?And by the way, I work on this distributed Gaussian splat render, and I was saying,say it's shit. Like, it's horrible. Like, because, like, you know, you've got millions of Gaussiansplats that you've got to page in and out and stream. And, like, it just totally depends on,like, the running system. And so, like, none of the models do a good job. So in our world,what I've found is that we've defined basically really well what performance benchmarks looklike and how the system should run. And what the constraints of actually building within, like,our stream processor
28:28look like to get the right performance.But every once in a while,it'll do,like my latest project has beenall about query optimizationand query performance.And so I was like, hey.Classic example of something.
28:41I was like,we are basically generating codethat corresponds to SQLand we want to run it in a pipeline.Yeah.And we're like, okay,well, you're generating shit code.Fix it.
28:54Yeah.And it started basically,going through and what it didn't realize was that it wasn't actually doing initially any optimizationon the code. It's like, oh, it's fine. It looks like, what's the problem? And I said, well, look,there's these conjuncts over here. You're doing and like this particular term is repeated and it shouldhave been taken away. And then it's like, ah, okay, let me figure out this out and so on. And then, like,it basically went on a thread to try to figure out what are all, like, it built a bunch of hypothesesaround like what are all the things that it can go and optimize to make this better.
29:27Yeah.But ultimately it was able, because it was able to ground that in performance benchmarks,and I was like, look, here's a data set of the queries and like what the users are asking and so on.Go work on it.And so it can basically do a relatively good job that way.But when it comes to saying something around like, how much memory should I give the system?Yeah.it has no idea.
29:56And that has to come from actual experiments that run in production.And so what I think actually these systems seem to be doing well at this point in terms of operational stuff is that it knows that, hey, I probably won't be able to answer this question.Let me run an experiment somehow and come up with that information.But of course, when it comes to operations, like production systems don't behave the way that you expect.and ultimately that's why incidents happen.So have you found, are you doing any of like the AISRI stuffwhere you actually have a running system that has logsand you feed it to an AI and that kind of helps you like tweak it?
30:35Yes.And how well does that work?So we're just...Because anecdotally, my experience with other people doing thishas not been that great.Like the reports I've been getting anecdotally have not been great.So a couple of things.The first one is we're doing that for both operational pieces
30:50and for security.But the way that we're tackling itis not the way that other people are doing itfrom an AISRI perspective.A lot of those systems basically start from a lagging indicator,like, hey, this metric spiked or like this alert happened.Go back through a very complex systemand try to determine why that happened.
31:11Right.Like, yeah, sure.It's, you know, that's why those don't workbecause, like, backtracking into a root causeis actually very problematic.And so instead, what we're doingis doing it the other way around,which is can we actually start by figuring out
31:27if there's something different in your environmentthat has changed?Like can we do a large scale clustering,a large scale kind of figure out a needle in a haystack,something that is different?Oh, so you're assuming that it was working fine beforeand now it's not so something happened,but it could be just like a change in like normal traffic patterns, right?
31:49That's fine too. That's fine too.And so the thing that we're looking at is like when things go wrong, usually the system will tell you something went wrong somehow.Yeah.Right.If it's working fine and scaling as normal, nothing will happen.But sometimes you'll see some like message which says error connections ran out.Yeah.Or memory exceeded or resource buffers emptied or like some kind of indication, like some kind of really early warning, like something happened.
32:18And that's the one message that we want to capture.and then go from there and see,is that an actual problemthat we need to investigate and deal with?So how does it do?So far, it's doing pretty well.I mean, we are running it internally in our systems.We ran it in a couple of our customers' environments.
32:36And in the cases where there was a couple of caseswhere we ignored it, and those turned into production incidents.Really? Oh, wow.Yeah, it's kind of, just from an intuitive perspective,like I can understand how these models understand.stand syntax because again, like I said, they're trying on all of the code ever written.And it just turns out that it covers a lot of useful cases.It's hard for me to actually even think of a bound on the actual operations of software, right?
33:04I mean, it feels like, you know, this is a ridiculous statement, but it's actually kind of true.It's like every time you add a bit to a program, the state space of like what it can do operationallyincreases by a factor of two.So you very quickly get to like, you know, like more, more, you know, states than atoms in the universe.And arguably this is why, this is why, like, product approaches to operations have never really worked, right?Like, the chaos engineering stuff never really worked.And we've always kind of relied.Actually, the dirty secret for the longest time of running, like, a distributed cluster was just human beings managing it.
33:42And so I'm just wondering.And then in order to even train an AI to do it,you have to have enough data and enough configuration.So I'm just kind of curious, like, do you think we're at a spotwhere you could like use AI to run a distributed clusteror run one of these services or do you think that's still on the come?I think that there's enough to be able to have it start that loop.Right?
34:10I mean, ultimately, like, I think that I agree with you.Like, there's infinite space.There's a lot of different things.But ultimately, humans also operate on run books.And, like, sure, there's a lot of things that happen in productionthat you ultimately need some human to go and, like, hey, this is too complex.There's, like, some state that's hidden that I know of because I talk to this other guy,like, at the water fountain the other day.
34:36And he told me about this thing he deployed.Like there's a lot of stuff that potentially isn't visible in the state of the system.But I'll argue that like the goal of what we need to be able to do is to like get to have enough telemetry and enough understanding to like make everything really visible from a hey, here's what the system looks like, here's how it's working, here's like whatever.And then I think ultimately most of the issues that end up really showing up in operations,assuming you have the system previously running okayis due to some kind of change.And if you're able to just focus on the goal of figuring outwhat the change was and like what,
35:20and kind of going through...Can I say that's a stupid question on this,which is I never think of these generative modelsthat good at predicting stuff.And it sounds like you're talking a lot about predictive modeling,like you predict a change or you baselineand you detect an anomaly.Like, am I thinking about this wrong?
35:36No.You're not.Okay.Because the goal is not actually having the agent try to predict,but have it get a signal that something changed.Oh, that's something.That's something I see, I see.Yeah.
35:52Wait, wait.So there, sorry.So there is a separate model that does the predictive stuff,and then that will alert the AI.Yes.Okay.Yes.That makes sense.
36:00Yeah.So, like, what we do is we basically,you give us your petabyte of logs per day.Yeah.And we scan all of them.And every time we see a new kind of pattern emerge in your logsand new thing that we haven't seen before,we lift it up to the AI.
36:14I see.So that's a predictive model that actually does what predictive models are good at,which is determining out of norm things or whatever, whatever.Exactly.And it goes to the AI, and the AI kind of uses that to figure out what to do.Exactly.So the AI basically can then reason and drive that down.So I can say, oh, this message, let me actually look at the code
36:35and figure out where it comes from and understand what the code does.let me see if it's actually happened before.So it basically runs all these hypotheses and heuristicsto try to understand does it make sense,does it mean meaningful in some kind of way?And then it can determine what to do with it.So how far away do you think we are for it?Let's just go back to query optimization, not like logs or something like.
36:54So how far away do you think we are from having an AIbasically do the job that a human used to dowhen it comes to query optimization?I think very close.Really?Yes.Wow.I think...
37:05Running a distributed cluster.I just think of like, okay, you've got like James Cowling, PhD from MIT who runs a distributed cluster and like if the clusters down, like the company doesn't work.And then if it's up, it's running.I don't know.Like are we at the point where you would say that an AI can do that?No, not yet.Oh, okay.I think it's, I think, I mean, I think like, look, very, very simply, when we started doing like AI for coding, it was autocomplete.
37:34Yeah.And nobody really knew to what extent you can actually build.complex software using AI.And like when we were looking at it,we're like, ah, like, AI is never going to be able to buildX or Y or Z or whatever.And then it basically broke through the expectationsbecause we were able to model, like,
37:51more and more complexity in the systems.I just feel like I'm a very simple thinker on this stuff.And to me, these models are distribution in distribution out.And it turns out, like, you can actually train on all the code ever created.And it's actually like that manifold is pretty manageable.And I just don't know of operations is constrained in the same way.That feels to me more like kind of like the universe, like heavy-tailed and exception-basedand power laws and all the tough stuff for these models to learn.
38:20But I don't know.I mean, you're the one that's in the field.So maybe that's not the case.Maybe they're manageable.But I will tell you the history of systems, as you know, is you always want to build a smartalgorithm that does the efficient thing, the smart virtual memory management,the smart congestion control,the smart routing,
38:39and what do we do insteadevery single time?We build a...We over-provision.No, we over-provision every time.No, literally.Every single time, like,remember, like, swap memory
38:49when you, like, swap to disk?Dude, if you did that, you were fucked.Like, that's it.Your computer was dead.So, like, we literally over-provision RAM.And, like, routing.You know how many, like, adaptive routing algorithmsare, like, we will respond to traffic.
39:00But you have oscillations.Like, fuck it.We over-provision.We just use ECMP.You know, so I just feel likethe history of systems.is that you don't predict stuff, you over-provision because you can't.And I feel like, I mean, you know, so I feel like in the operation side, so either we build this,
39:16you know, we will build the systems up front that are over-provision so you don't have to, like,do kind of complex operational stuff, or maybe AI can do it the way that a human being.But I think it's very different than code because you're dealing with, like, exceptions,there's not a lot of data, et cetera.I just feel like a different domain.But I don't know.I'm a VC.I'm just like.
39:34You know, I kind of, like, the reason that I'm more optimistic here is because I live in the world where, like, what we do is actually deal with the observable world.Like, I was going to say, you're the literal expert on this.So it's silly for me to even, like, ask you these questions.No, not at all.Because, like, in my world, we, we, like, my assumption is that we have all the data.Is that, like, we know exactly when, like, a new thing happens or, like, an exception emerges or, like, a work.changes or like whatever.Like in my world, all of that has already been
40:08collected and streamed through.But the model that is not trainedon a grounded source for that stuff.No, no, you're not.That would be really hard.Right.But the story here isn't like, hey, Mr. Claude,here's a bunch of logs.
40:26Is something going on, right?And it has to be much more able to run experimentsand kind of dive in.and see, like, what are the different shapes of possible causal chainsthat actually fit in the environment that exists.I see.So, yeah, this is what you said before.Now I see what you're saying.
40:44You're saying that, like, the universe is heavy-tailed.The number of bad things that can happen is probably an exception thing.But, like, the set of steps used to debug it is a manageable thing.That's what you can teach one of these models and that's what they have.And therefore, if you give it the right signals, they can go through a number of stepsto actually constrain this problem.Exactly.Interesting.
41:05Exactly.And so it is really taking the role of the human.Because remember the whole chaos engineering wave?Yes.And the reason it didn't go anywhere is like you're very so quickly like trying to like, you know, reduce like a state space that was atoms in the universe.Like super quickly.Like it can actually make that manageable.And it was like the human being that would go through a steps that would do that on the actual debugging.
41:28And this is what you're saying the AA would do.Yes.Right.So the goal isn't.like have the model just be like, this is what happens. It's, hey, run through this workflowto determine what happened. And like we're trying that also on the security side, for example,where like, hey, in security, you have a something different changed in your like statistics,like some behavioral difference in terms of what entities exist, the cardinalities of various things
41:57that exist. And so that becomes a perturbation in the whatever, the universe. And so we takethat signal and then we feed it into an AI and we're like, hey, AI, come up with a bunch ofpossible causal chains that exist and then go through and verify if any of those fit the patternof the data that we actually see. And so that becomes a much more like, hey, here's a,here's like, I don't know, like fit this square into find the peg or like find the hole thatfits this peg.I mean, that seems even harder to mebecause the operations one,
42:35the argument is the systemwe can get into broken statesthat are kind of like very rare,and there's a lot of those rare states.Yeah.And so it's just not something that you could,you know, train a model onjust because there's so many rare states
42:50that can get into that are bad.And so maybe you can reduce that by doing the steps.The security one seems even harderbecause this isn't just likethe natural orof rare states, you actually have an adversary that is trying to, like, intentionally findrare states and intentionally find stuff that you can't do. And so it may be the case that, like,code becomes solved. It takes a little while, you know, longer due operations and it takes a lot.
43:16Yeah, but like also, like think about, I mean, I don't know, to me, like the whole world is sortof quantized to a certain next step. The plank length. Is that what the physicists say? I don't know.I'm not a physicist.I think there's like actually,I literally don't know what I'm talking about right now,but I literally actually think that like,like,like space time is discretized into a plank,
43:36like,but yes,I mean,I agree.Like,to a certain extent,what you just said,but for operations and systems in general,
43:43like there is a discretization of like,the possible states and the past through them that are possible.Yeah.Right?I just think when you're dealing with.There might be,there might be a time.ton of them. There might be a ton of them.
44:00I'm going to tell you, so remember Sundar, who was in our grad school group?Ear?Sundar Ear?Yeah, yeah, yeah, yeah.So I remember he came to me once, and he was, he was looking very sad.He's like, Martine, I just had this really sad realization that, you know, if you take any MP3, remember MP3?He's like the music first.Yes.
44:20He said, he said MP3s have a finite, let's say, you know, four minutes.It's like four megabytes, you know?and that means there's a finite number of songs in the universe.That can...No, because, you know, because it's all made of bits, right?And it's four megabytes.And that means you can just enumerate every one of those concerts.So I'm so sad in this notion of, like, the finiteness of music, right?
44:46Literally, this is like, we're walking on the dish when he had this conversation.And I'm like, motherfucker, dude, you know how quickly you've, like, overcome, like, the atoms in the universe?you haven't even gotten to a cane, right?It's like two to the two to the two to the two.Like, what are you talking about here, man?Be happy.Two to the two to the two, four megabytes.Four megam!
45:06Like, there's so much state.Like, be happy.That's effectively infinite, right?Like, what are we talking about here?So I do think in some level, like, we do underestimate, like, the number of states that, like, you know, systems can get into.And, like, maybe, I will tell you from the perspective of the VC that looks atthese companies, the ones that are trying to tackle operations are having a tough go at it.It seems much, much tougher than those that are tackling the code, because I think it's a much
45:35tougher problem. I, 100%. I agree. It is a much tougher problem. And it's not just because, like,the systems themselves are, you know, there's like external loads that are random that kind ofhappened to you or that, you know, somebody just shipped a new thing and rolled it out into production,didn't tell anybody or, like, there's some state that actually happened.when like this thing happens and this thing happensand then you end up in a deadlock.It's also because it involves humans a lot moreand not just like one human who's the coder.
46:07It actually also involves...The pools of knowledge are distributed too.Exactly.And like the interactions between like what it takesto like connect the pieces and put them together.But ultimately, I mean, I think about like where AI isand like what are the possible futuresand like the depth and the forward-looking
46:25capacity of all the things.Like, I think code is just, to me, it's a microcosm of like, maybe I'm just too fatalistic.Maybe I'm just like too pessimistic about how the world works.And I think of everything as like following some kind of probabilistic path.And so to me, like, if we can make it work in code, we can continue expanding that level ofcomplexity bit by bit and kind of adding layers and layers until we can handle those problems.of operations.A lot of people that's not fatalistic is the opposite.
46:59That's very optimistic is what people would say.For me, I don't think the universe gives up its secrets that easily, but, like, we'll see.I mean, like, this is kind of the big question.And I would say, as yet, it's not an answered question.No, yeah.No.But I don't know.A lot of things that as humans we think of as being random or impossible to tackle.
47:24Yes.And that's what we saw with coding.Like, you know, initially, like, people thought, like, the possible world of all these things that you can implement in code are kind of limited.And, like, you can't really get a system to potentially understand the full complexity of a human brain that's required.I think over time, it'll all, it's so funny that I'm actually telling you this and you're, like, the VC who's looking into the future and being like, no, well, I mean, I'll tell you my view.My view is like the study of these models are you train on data and either that,I mean, the, I was trying to say it.So you train on data and that represents a distribution,
48:10and that distribution is useful or not.And so like when you train on language, the fact that it's useful,that to me says that humans are so remarkable because they've looked at the universe,which is a very high dimensional system,and then they've reduced it to a manifold that's actually useful for a model, right?Yes.But that's different than the universe.Like, there's a reduction from this high-dimensional to the language
48:30is less dimensional.And I think that code is a reduction of all the operational states of a machineinto, like, the instructions that make it run.And so to me, there's still a disconnect between, like, the instructions or the descriptionand the actual, like, the actual running of the universe and actually running the system.I think we can make gains toward it.But what I will say is, like, you're kind of converging on simulation.Like, there are things that are computation that you're reducing.
48:56You kind of have to run it to get the answer.You can't just like, you know, predict it.And so, like, maybe we just are at an event horizonwhere, like, models will solve anythingthat you don't have to actually simulateto get the answer from.But why?Why do you have to, like, just think about it in terms of,
49:14hey, it'll have to answer the question, like, nowas opposed to being able to simulate and, sorry,being able to plan its own execution to gain the information that it needs in order to be able to.Yeah, maybe that's where we're going.Yeah, that's kind of what I'm.Fair enough.Yeah.And I think, like, there's a lower dimensional space, which is like these steps that you would take,
49:39that we can reduce a whole bunch with AI.And then whatever steps that you need that that can't do, it can create the code to do or it can simulate it.And the point that you're making, though, is that like, hey, look, the model can't actually go and figure out.out the additional dimensions of the world that we need to go and find to like model the universeinto its world.Like that's probably, I mean, that's probably true.But can it actually simulate a scientist sufficiently to be able to get to that point?Like, maybe.
50:11Maybe.Cool.Can I ask you a question?How much of a, of the VC work do you think is something?that an AI can drive.So my model for VC is it's very relationship driven.It's, I think, so it's okay.So I just have a belief that markets are actually pretty efficient, you know,
50:36and so it's not about like finding the company that nobody else finds.I think people all find, like, the right company.I think that they're efficient.And then I also have the model of VC that, you know, tech, tech is positive,and I'm kind of optimistic about all tech,and I don't worry about TAM.And so I think the key is access.It's like raising money, it's access,
51:01its ability to win deals,and there's an ability to add value after that happens.And a lot of that is like stuff that startupsjust can't naturally do, like, you know,recruit, you know, big execs or help with that.And so the way that I think about venture,you kind of have to be a human,in being to do. It's like, you know, because that's the actual job. So for that, I don't think
51:27like an AI can ever... If you gave an AI simulated universe where it could like, you know, like whateverreward hack, like building trust relationships, fine. But I think that that would have to be,to be... That said, I will say, I have seen an analysis done by AI that are as good or betterthan any, you know, genius have ever worked with. Like, so for the end, the, the, the, the,the analysis piece, they're great.But I think this is more an indictment on VCs,ability to analyze than it is, like, how powerful the models are.I just think of all a kabuki show and bullshit.
52:04I just think the right VCs are like, you know,tech is net positive.You know, tech ways always add value, like, let's find the best companyand invest in it.The markets are very, very efficient.And so I just feel like the answer to this questionis almost kind of a technical.Like, my answer is almost a technicality.
52:22So how, like, you know, one of the things, you know, I came here, I wanted to ask you this real, this question, which is like, you know, we started talking specifically about ops.Yeah.But to me, like, there's this world of AI writing code, which is kind of like this abstract simulation world that it lives in.Yeah.And then there's the rest of the world.Yeah, exactly.And the entire discussion is this.It's like, there's like this, there's like the AI, the world that the AI lives.
52:49in which is all digitaland whatever it is,the training data,the in distribution world,and there's the actual world.And, like,those things are not necessarily the same.Right.
52:56They're quite often not.And so the question is, like,to what extent do you need to train an AIto be able to survive in a world?And, like, what are the things that it actually?I don't want to answer that question.Like, what do you need to train?But, like, I'm curious about the open spaces.
53:13So we talked a little bit about operations.You said, oh, I don't know if it's going to be solved or not.I'm like, yeah, Martinez, it's going to be solved.And in my mind, like, there's so many other things.Like, for example, self-driving cars,like for a long time, we thought that's not going to happen.But it's clear that it's actually going to be done.And it's going to happen.
53:34It's just taking time because of a lot of different things.And so what are the other next stages,the next areas that you think are kind of like lined up right after code?I mean, my sense is, you know, AI is going to continue to take off in a wildfire for a constrained domain,where the solution space is in a constrained domain, right?Where the distribution of the solution space is the same as the distribution of the training set.The distribution of the syntax for code is the same as a distribution of the training set.The distribution of the generated images is the same of the distribution of the training set.
54:10In those, you know, areas, I think it's going to continue to take off as a wildfire.The domains where, like, the distribution of the solution is the natural world, I think it's going to, you know, have, you know, it's going to be much harder, take longer, and I think the economic equivalency is going to be much more difficult.Self-driving is a great example.I remember when Sebastian Thrun won the DARPA Grand Challenge when I was at Stanford.Remember that?And we're like, hooray!You know, self-driving cars is around the corner.Seriously.
54:38And it took 20 years and $17 billion of investment to get to.worst unit economics than Uber, right?So I think that like solving like, like solving real problems where you have to deal with the distribution of the real universe is just much, much harder.And so I view as AI as this general tool.It's a very effective tool.It's one where like you have a distribution of data and then hopefully you can apply that to like, you know, some problems.And we're going to apply it to a bunch of stuff.And I think economic equilibrium is going to determine what we're going to apply.
55:13works and what doesn't work, because I think it's a very, very general tools, like a universalfunction approximator. And wherever that nets out, I mean, again. You know, one of the thingsthat I said earlier was, like, how AI was training humans to, like, use it. Yeah. And I very oftenthink about, like, you know, some of the problems that seem intractable is because the world is kindof that way. And then we're trying to fit AI or the tools that we're building to manage theworld as it is. Yeah. But in some situations, we actually tend to change the world to fitto what we can actually do from...Oh, 100%.
55:45Of course.I think this is the story of robotics.Exactly.100.The automated data warehouse.And so the question, like,that always comes to my mind,like, when we were looking at these data warehouses and whatever.
55:57And, like, the only one that worked was the one where it was like,you know what?Just automate the whole fucking thing.Yeah, yeah, yeah, yeah.And I wondered to, like, what extent as we kind ofimprove or change how we...or kind of expand the domain of what we want to solve with AI tools and so on,that we actually start modeling the world or changing the world
56:20and the things that can actually happen to fit what AI needs it to look like.Yeah, I mean, that makes sense.I think that right now a lot of AI just operates in the digital world.Yeah.And for that, like literally the solution set is the same distribution as a training set.Yeah, okay, in a sense.And so that's fine.And I think that, again, like in the case of equilibrium, like I think like when you're dealing
56:42with the real world,like you can solve any problemyou want. It's just a matter of like compute and moneyand that the equilibrium pointis going to be an economic one.And I think one way you can impact thatfunction that you're doing equilibriumover is to constrain the environment.
57:01It's to constrain the system.And so I think that we're going to like we're in nowin this meta economic equilibriumflywheel wherewe're kind of optimizingthis is super general, but veryinefficient process with a solution set that people value.And I think, you know, we've got 30 years of this and a lot of values going to be created.
57:20You know, one, you want to hear a funny story?Yeah.So, you know, we built this AI thing and we're like talking to people.We're like reaching out to customers.We're like, hey, we built this AI thing.You know, we're helping customers become more reliable.Here's like what we're doing.They're like, great, we want to talk to you.
57:37And we go on the call and we're like, hey, here's what we do.You know, we built this system, helps you optimize your day.data that helps you reduce your costsand then it helps you become more reliablebecause now we can find these problems and solve thembefore they hit your incident. They're like, wait, wait, wait, wait, wait.Roll back, you said you can save 90% of ourdata dog bills?
57:55Yeah, that's what I want to buy.But the AI thing, they're like, no, no, no, no.That's what we want to buy right now.And it's interesting, like a lot of theselike for code, you know, I alwaysthink about it's like, oh, for code, people justget it. They're like, oh, shit, it justworks.
58:12Yeah, yeah, yeah.And on the ops side or like anything that's beyond that.Yeah, it's very tough to understand the actual value and put a dollar value to it and bea pay out the door to $100.And I also think from the coding standpoint, literally it's engineering productivity, whichis one of the highest line items on these things.So it's just very straightforward.Exactly.
58:28But you can also think about it, although it's SRE productivity or it's like your businessgoing down, maybe you can save an incident and what the cost of it.I don't know.It's weird.I think like how people think about it outside of the.Yeah, I also just thinklike this code was so magicalbecause it was so good and nobody thought that could happen
58:48and I think the operation stuff nowit's still like...Pretty early. It's still pretty early.So, yeah.All right.Well, listen, this is great chat.Thanks, Jeff. It should come in.Thanks.
59:00Thanks for listening to this episodeof the A16Z podcast.If you like this episode, be sure to like,comment, subscribe,leave us a rating or reviewand share it with your friends and family.For more episodes,go to YouTube, Apple Podcast.
59:13podcast in Spotify. Follow us on X at A16Z and subscribe to our substack at A16Z.com.Thanks again for listening and I'll see you in the next episode.As a reminder, the content here is for informational purposes only.Should not be taken as legal business, tax, or investment advice or be used to evaluate anyinvestment or security and is not directed at any investors or potential investors in any A16Z fund.Please note that A16Z and its affiliates may also maintain investments in the company'sdiscussed in this podcast. For more details, including a link to our investments, please seeA16Z.com forward slash disclosures.
There aren't comments yet for this episode. Click any sentence in the transcript to leave one.