The Infra Pod - What happens to your service mesh when the workloads running on it aren't written by humans? (Chat with William at Buoyant)

Episode Date: July 20, 2026

In this episode of The Infra Pod, hosts Tim Chen (GP at Essence VC) and Ian Livingstone (CEO of Keycard) sit down with William Morgan, co-founder and CEO of Buoyant and creator of Linkerd, to explore how AI agents are reshaping the infrastructure layer — from service security to inference routing.Will traces Linkerd's origins to Twitter's 2014 migration from a monolithic Rails app to distributed microservices — the moment function calls became network calls that could actually fail. A decade later, that same communication layer is under pressure again. Non-deterministic agents make MCP and A2A calls over L7 protocols, and for the first time, fine-grained access control isn't optional: an agent will eventually find and call every reachable endpoint, including the one that deletes your database.The conversation covers the real pressure AI coding tools are already placing on platform teams — deploy cadences going from tens to potentially thousands per day — and what running inference inside Kubernetes actually means for proxies. Will breaks down why KV cache-aware routing is a 100x performance lever, why the modern inference proxy looks less like Envoy and more like a Makefile, and shares his spicy take on where compute is heading: in-cluster inference becomes the default, with frontier models reserved only for tasks that genuinely need godlike intelligence.[00:00] Guest introductions and Buoyant founding story[03:00] Linkerd's origin: solving Twitter's monolith-to-microservices migration[07:30] How AI is (and isn't) changing Linkerd today[11:00] MCP, A2A, and agents as L7 traffic in your cluster[14:30] Why agents make endpoint-level access control non-optional[18:00] AI as amplifier: what 10–1000x more deploys means for platform teams[21:30] Running inference in Kubernetes: a pathological workload[24:45] KV cache-aware routing and the 100x performance gap[27:00] The proxy/gateway landscape: grad students vs. premature standardization[29:30] The inference proxy is a Makefile now, not Envoy[31:00] Prompt injection, sandboxing, and the security problems with no clean answer[33:00] Spicy Future: in-cluster inference becomes the default

Welcome to the InfraPod. This is Tim from Essence, and Ian let's go. Hey, this is Ian Lewis, the CEO, co-founder of KeyCard, making it possible to secure your AI, your agents, and make sure they don't delete your database. I couldn't be more excited today to be joined by the CEO and co-finder of Boyant and one of the creators of Linkerdie, William Morgan.

How's it going? What in the world got you excited to start, Boyant? Oh, yeah. Oh, gosh. Well, we're talking about the olden days now. Hi, hi, Tim. Hey, Ian. Great to be here. Thanks for having me. Yes, I am the co-founder and CEO of Bliant, but, you know, there was only one other person when we started, and he didn't want CEO title, so that's how I got it. Turns out, it's very easy to do it, you know, if you do it that way.

Yeah, so why did we start? You know, so the thing we're most known for is Linkerdie. It's a service mesh. If you're in the Kubernetes ecosystem, then you probably have heard of Linkerdie. And if you're not, then, like, you haven't. And that's also fine. It's a very kind of niche topic in some ways. But we started buoyant because we had just come out of Twitter, and Twitter had gone through, and this was like 2014, 2015. Twitter had just spent a couple of years going through this big transformation

from monolithic Ruby on Rails app to, like, I think we would now fall, cloud native system, although, you know, it looked quite different at the time. We didn't have Kubernetes. We had Docker. Sorry, we had, we had Mases. We didn't have, we didn't have Docker. We had, like, C groups and the JVMs. They kind of got process isolation and packaging, you know, and part of what happened there was we had to figure out how to solve some of these distributed systems problems that were cropping up. Back when we had the, like, you know, the monolithic Ruby on Rails app, you know, it'd be like, function A calls B, call C, calls D, and like, it worked, you know.

And now we replaced each of those function calls with the network call, and network calls were like, sometimes they didn't work. And sometimes they took, you know, hundreds of milliseconds instead of hundreds of microseconds. And so lots of things happened there, and one of the kind of foundational piece of the technology, obviously, Mesos was a big part of that story. And, like, Mazos went on with Mesosphere. and, you know, Tim and I were talking a little bit about that before the show. But another part of that was basically, how do we manage the communication that's happening

between these microservices? Because we never had that before. You know, we had function calls, right? And now we've replaced them with network calls and network calls fail, like I said. And also, it's not just like A talks to B, like in your head. It's these two boxes. It's like you've got 100 instances of A that are talking to 500 instances of B. And, like, some of those instances are in failure mode.

Some of them are in a different, you know, cluster or different data center. So managing that communication was something we had to solve. And we knew that was a problem that the rest of the world was going to have to solve. And that was kind of the impetus behind Linkerdy. So we brought that idea. And in the very first version of LinkedIn, we actually took the open source like technology that Twitter had produced and turned that, which we later moved on from. But, you know, turned that into a product that the rest of the world could use. Cool.

And let's jump forward in time. Linker D, I think many of us who have been around for some time and tell what it is and have it running at some point in our lives, if not still. I think the question is, how has Linker D and your view of the world change with the rise of agents? How are you feeling about server smash? How do we rethink about these types of topics

when, you know, compute becomes non-deterministic? You're going straight to AI. What about the 11 years of pain and suffering and beauty that we created in between here and then. Yeah. No, I'm happy to talk about it. So there's like a couple layers. One, the most immediate layer is AI is kind of like the opposite of what Linkerty is. So, you know, would we add AI to Linkerty?

Well, I don't know. Like, Linkerty has to be really fast and has to be really predictable, you know, and it has to be really understandable. And those are kind of like the tenets that we've, built into the product because our consumers, our platform engineers, who are doing a million other things, and the less complexity you can put, you can load into their brain, the better. So at the surface level, it's like, oh, it's independent.

Like, what do we care? You know, these weird deterministic things are running out there, you know, and, like, great for you, but, like, it doesn't interfere with us. I think, you know, you can scratch the surface off, the shine off of that pretty, pretty quickly, if nothing else, because LinkedIn is an open source project. and like every open source project, there's a big effect from everyone running AI to make a poll requests or to find security vulnerabilities.

We have this big spike in like, oh, these security vulnerabilities that people have found, which, by the way, are all BS, but like some human maintainer has to, like, sift through them. And then I think, you know, the next step I would take from there is when we get to things like MCP and A2A, well, these are now protocols, right? And they're happening between services that are

running on the Kubernetes cluster. Now, the services we're calling agents, and they have this non-deterministic behavior, sure. But they are talking, and they're talking over the network, and they're talking over to some L7 protocol. And if there's one thing that Linkerdie is really good at, it's parsing and understanding and doing stuff with layer seven protocols. So, okay, suddenly it starts becoming relevant.

And by the way, if you look at something like MCP, I mean, if you look at it through the lens of like an infrastructure engineer, your first reaction, I think, is like, okay, whoever designed this nonsense, like, was not an infrastructure engineer because, you know, it's like, it's stateful by default. It's like JSON RPC and like all the stuff. You're like, okay, this is miserable. But whatever, you know, okay, maybe that's job security for you as a proxy engineer.

And then I think the part that maybe is most interesting for us is that once you have these nondeterministic systems that are in your like production environment, I think a whole set of, of things that Linkerdee provides, especially around access control and security, which previously you could talk to talk, but you wouldn't have to walk the walk. Now suddenly, like, it's not optional anymore. Now you have something in there that it's eventually going to find,

you know, the RPC call to delete your database, and it's going to call it. And you have to, like, protect yourself against that. You don't have a choice in a way that where you kind of had a choice before, Irwin, you were like, I had the code's written by some really senior engineers, and they all like do code review. And, you know, so the likelihood of this actually calling the delete database at an important, like, pretty low. Well, now it's like 100%. So all of a sudden, this different kind of value prop for Lingerdian.

And I still remember the days where old people, I guess, still sitting around talking about Kubernetes, talking about service meshes, talking about all the cloud nativeness going on. Like, folks like us, like working at infrastructure for such a long time. I feel like the room are filled with folks like us. And even though there are application developers building on top of us, they're still very technical in nature. But now I think this agent world, we're seeing, like you mentioned, the code you generate, even though you're supposed to be reviewing your code,

like how do you review every single line of code? You don't even know what agents are building what anymore. So there's a lot more like YOLO that's happening in people's clusters. Coming from Link or D world, you're kind of in the middle of all the communications, but you also work with across all the different things that are in a Kubernetes cluster or maybe above that. Is there any major philosophy difference

when it comes to how people you should be thinking and managing infrastructure, even like teen changes? Yeah, I mean, first of all, like everything is changing so rapidly and there's a lot of hype and, you know, noise and people saying that they're doing a lot of stuff and in practice, they're not really doing it.

So you have to like, you know, you have to take everything with a big grain of salt and kind of tease apart what's actually happening. What we see happening with our customers is, and I don't think this would be a surprise anyone, the primary part of the audience that's affected right now are the devs, right? Because they have these super powerful tools, whether it's a clod or whatever, that mean that they can ship a lot of code really, really rapidly. And we can argue about, like, is that high quality or low quality, this is it five coding and not? Like, how do you do code review and stuff like that?

And I think there's interesting things to talk about, but they're the way. ones who I think are putting the first bit of pressure on the platform team. Because the platform team, you know, is going from some small number of deployes a day to potentially a huge number of deployes a day and from a bunch of quality control kind of like assumptions that you can make to some very different and weaker assumptions you can make. So, okay, you know, is that good, is that bad? I'm not a person who likes to put a moral judgment on that. I'm sure the code is like crappier overall, but like progress is probably faster overall. So, okay, you know, probably good, but it almost doesn't matter. Like, your job as a platform team is to serve this,

this audience. And so if they decide to do things really, really rapidly, you know, the last thing you want is to be the curmudgeonly platform person. Like, no, sorry, can't take that many deploy. Like, what are you going to say? Like, our system doesn't scale to that. So I think that's the first kind of pressure that happens to the platform engineers. And none of it requires, at that level, none of it really requires AI to solve. It's just like an amplification of what you have there. And I would say my general framework for trying to think through how AI affects stuff is that in most cases what it's doing is just amplifying,

something that was already there. And I think it's helpful for understanding it. Of course, if you amplify something by 100x, which it's doing, you end up with something that's qualitatively very different. It's only so helpful to see things for that lens. But I think that's, you know, this is one example, right? We've taken 10 deployers a day and we've turned to 100. or to a thousand. What does that put pressure on? It puts pressure on CI, puts pressure on the cluster.

You've got more pod churn. All sorts of things that were kind of humming along. Okay, now we're starting to, you know, suffer through there. That's what I see in practice today. I also see a lot of people who are like, oh, well, we're going to deploy agents and agents are going to run into the cluster and you're going to do stuff, kind of the scenario I was described before. I haven't seen a lot of that in practice. I think that's more talk than anything else, or maybe just happening in some realm that I haven't had access to. Is that going to be a real serious thing in the future? I think so.

I think so. It seems like that's a pretty plausible direction for it. I haven't seen a lot of that in practice. I think in part because there's too many scary bits of that right now. Would you take an agent and drop it into a production environment today? I think there's very few people who would be like, yeah, I'd feel comfortable doing that. I think very few. Yeah, this whole AI is not making anybody comfortable, really.

So, and so much of our infrastructure as a practice or the whole philosophy is to, I guess, reduce the complexity, create as little leaky abstraction as possible for our upstream folks, right? And also able to manage the mess, right? I thought the cloud is already a mess. I guess we're adding even more mess to this craziness. And let's just say the Kubernetes infrastructure pieces around it, is there any changes required even at the infrastructure level? because there are more deployments, or maybe because there were more AI. Because I think there's two things happening, right?

There's more code generated and deployed, for sure. But there's also more interest to actually even deploy AI more. Some folks are playing with like inference clusters more, like a lot more GPU accelerators, a lot more open source models. There's generally like the AI changing your dev processing and also even trying to even run AI in more enterprises. And where do you see the infrastructure pieces kind of changing even around a are there older technologies, things that needs to be replaced now?

Or do you think everything pretty much is the same? We're just adding more AI stuff around us? Yeah. Well, I mean, run AI means a lot of different things, which I think is what you're getting at. So, you know, what we've been talking about for the past 10 minutes has been like one very specific use case, which is like, I'm going to run an agent. Okay, what's the agent doing? You know, it's like taking a bunch of actions.

But the core LLM is like probably sitting somewhere else, right? It's like he's talking back to Anthropic and then it's like interpreting its, you know, tokens and doing stuff. One thing that is very near and dear to my heart, although this is quite distinct from Lincredy is the other thing you mentioned, which is running inference within the cluster.

And that's an interesting situation because the way that works, if you dip even one toe into those waters, what you quickly learn is the inference serving engines, things like VLLM or SG laying or TensorRT, LLM, or whatever, they basically expect to consume the

whole machine. They're like, how many GPUs you got? We're going to take all of them. It's going to take us 15 minutes to boot up because we're like sticking a whole bunch of crapola in the GPU memory and then we're like, you know, running some Kuta kernel to like prime the KV cache or blah, blah, blah. And like that whole mechanic is very weird from the Kubernetes perspective, right? From the Kubernetes perspective, you're expecting like a bunch of, you know, microservices that boot up in like a second and like the health, you've got a health check that, you know, fires five seconds later. Like, great, you know, let's go do it. And, you know, some of them die, whatever.

Like, you boot up some new ones. This is a polar opposite. So it's a little weird that stuff is running in Kubernetes, but it is. For better or for worse, like, everyone seems to be doing their inference, which is, like, the act of running these inference serving engines, right? The act of, like, doing the token completion. Everyone seems to be doing it in Kubernetes. And so that's putting a little bit of pressure on Kubernetes, just because those workloads are so weird.

Now, in the meantime, we've kind of gotten good at running that sort of workload in Kubernetes. I know eight years ago or whatever in the early days, that was a big knock on Kubernetes. Mesos was great for this type of workload, Kubernetes kind of sucked. And we've added enough stuff that it kind of works, even if it's a little weird. So I don't think that's putting a lot of pressure on kind of core Kubernetes. Maybe someone will correct me. One thing I think NC is putting pressure. on is anyone running any kind of proxy?

Lincredy, obviously, but even in ingress or an egress proxy, I talked about how MCP was kind of a crappy protocol. Even the open AI like inference call, you know, chat completion slash V1 slash chat slash completions.

Designed by someone who obviously did not understand infrastructure. You know, it's got all these weird aspects of the model is a required parameter, but it's in like the body of the request, which means, like, you can't do any kind of routing without, like, parsing, you know, first understanding HTTP 2, probably, and then parsing JSON, buffering the whole body and the memory, pulling that thing out, and then, like, then you can move on the step two.

Like, already this is crazy by proxy standard. So you're starting to see, you know, some of that stuff really get under pressure in a way that's really interesting from my perspective, because I spend, you know, I spend my whole life right at this point writing, you know, L7 proxies. So now we suddenly have some really interesting constraints on there. Like now, if you're serving inference, you should basically expect to load in that whole body into memory, even if it's like 50 gigs, you know, and do stuff with it and make decisions based on having parsed it. And I think five years ago, that would have been crazy. Like five years ago, we're saving every possible millisecond. We're focused on like, okay, I've got this request coming in.

How do I stream it so that I don't have to keep hold of any of it? How do we do zero copies? So I don't even have to, like, copy the memory from one part to another, because those 30 microseconds are, like, too expensive. That was the world of, like, you know, proxy, you know, technology five years ago. Now we're like, oh, who cares? Like, you know, throw it over here, throw it over there. Because the LLM is going to take, like, 13 seconds to respond anyways.

You know, that's putting pressure on how network calls happen. But that stuff is also super interesting from the routing perspective because the difference between routing correctly and routing poorly is like enormous. If you have a request that hits the KV cache, you know, because you've routed it correctly to the GPU that had that stuff preloaded, it takes, you know, a couple hundred milliseconds. And if you route it to a different GPU, it takes 100x that time. So now these network decisions that, again, used to be like about shaving off five milliseconds or whatever, or like, show you off 30 seconds from your response time

in a part of the stack that people really care about. So super fascinating time to be an infrastructure engineer, is my opinion. We're at a point yet where a lot of, like, we're on an optimization, that's for sure. We're at, like, the exponential curve of intelligence is still too great that, like, time to inferences. Unfortunately, not as much of a focus. I think it will get to a point where it will be.

I often think about, like, the open-a-eyed operator demo is like the thing that none of us actually want our agent experience to be that would take the same amount of time to do all the clicks as it would to actually watch it go and navigate the web on behalf and like certainly we're at the point where like that type of intelligence, that type of tool call, the type of workflows, it makes definitely faster as model capabilities increase and that actually makes that demo like now really feasible. But under the hood, like I think agents push infrastructure and systems in a way that none of us like anticipate. I'm curious, you know, there's a lot going on in the broader

network, router, proxy space when it comes to agents. You've got a lot of these LLM proxies. You've got all of this MCP gateways. You've got this agent gateway thing that was just donated to, you know, the agent form of the CNCF. Where do you see people, like, struggling today? And what do you think the outcome is of, like, what future Asian architectures actually look like based on what you're saying? Where things are today, certainly in proxy land, is you kind of have, like, two extremes. On one extreme, you've got the default, like, routers is what they call them,

that come with VLM and SG Ling. And, like, they are effectively written by grad students. They break all the time. And they're written by people who are very smart, you know, probably significantly smarter than me, but just don't understand how to write infrastructure. On the other side, you've got, yeah, like Agent Gateway, which, you know, is going through some kind of weird enterprise, like, you know, standardization path where like the way we're going to be popular is we're going to get the approval

of this foundation that just showed up. And like in between, I think is where the actual usage needs to be. It's really early to start working on standards and standards bodies and like, oh, this is a default proxy that's going to be used for blah, blah, blah. Like all this stuff is changing so quickly. And I think this is one of these situations where you need to let the, you know, You need to let the ecosystem, like, try a bunch of stuff and see what works. And, you know, part of this is there echoes of the service mesh space because, like, in the early days, there's so much hype about the service mesh that everyone was, like, trying to make a move.

Everyone had their service mesh. And you ended up with eight different ones. And, like, of course, they all, you know, all but, like, two fell away or maybe three. Because, like, it's really hard. You know, it's like it sucks, right? It's like it sounds like it's going to be awesome. Oh, there's all this hype. And then you start doing it.

And, like, you find out that it's super hard, and your customers are complaining because, like, what you built sucked. And, you know, and I think it's, we're in kind of a similar situation now where there's, like, a lot of interest in doing stuff. Because, of course, AI, like, oh, gosh, we got to, we got to have our thing out there. And a lot of them are just terrible. But I do think there's some really interesting challenges. You know, forget about MCP or whatever.

Even just handling these inference requests. There is cool stuff that you have to think through that that force you to build proxies in a different way. If you assume that you're going to load the whole body in and you assume you're going to make a bunch of calls before you even dispatch the request because you're going to do like a, you know, you're going to classify the request based on what model should I send it to? Well, you're going to do that by like sending it to an LLM and it's going to tell you, I think you should send it this one. And then you're going to run it through some PII filter. Well, that's also another LLM over here. So you're running it through that.

And then you're going to, you know, do some KV cache aware prefix routing. Well, you're going to run. well, you're going to send it to some other process over here that's maintaining, you know, a model of every GPU in your system. And suddenly, your proxy looks less like, you know, this super-fast microservice. Like we're trying to get these bites through the system with touching them as little as possible. And it looks a lot more like a make file that you're running. You know, like, I got to run, you know, I got to execute these seven things before I can dispatch the request. So which ones can I paralyze? Which ones have to be, you know, preconditions the other?

It's a totally different model. And people, I don't think, really appreciate that. Yeah, I mean, I think there's this new sort of runtime security category that you have arm.dev that's trying to standardize. It's interesting, right, because like agents, because they're not deterministic, they basically move the requirements from the enforcement to runtime control. It's the premise.

And the other than runtime control, it's a context, right? In order to think about in context of this tool, is this tool call appropriate in the context? in the context of what the user has assigned the intent, which is a layer above, one is the control point is a very different place that's ever been, and two is the context required

to perform the evaluation is much higher. And so there's this whole open question, how do you do that efficiently? And how do you enable that in a distributed world? Because ultimately what we want are agents that are aligned with user intent. So if an agent's

assigned a mission or task or whatever we want to call it. Hey, I've employed this agent or the organization has employed this agent or whatever framing we want to use. How do we ensure the agent is staying aligned with that intent, knowing that we want this agent to have a broad-based capability, have broad-based like knowledge, that it can ingest and work with all these different systems. And it can do that dynamically. But on a per-task basis, I want to be able to like select down what I can actually do.

I think the ecosystem, infrastructure, security, builders, app builders, users are struggling with navigating that change from what used to be, you know, basically policy enforcement that started at the point of like the session. So at login and you got issued a token, the token for or like the static, world-based access control, static, you know, network policy, to something that's significantly more dynamic in application layer and concept. And of course, like the default for most people is like, okay, we'll put something in between. And it's like unyet clear, like what that in-between runtime control looks like. You know, from some people who's like, oh, prompt injection defense, there's a source, some anti-identity element. There is some like data, DLP, PII redaction-y thing. There's some normalization of the context layer.

And there's all this stuff happening. And some of it is because context engineering, some of it is because the existing tooling, specifically in identity is so bad that you have to, like, so you don't have the granular, of scopes, you don't have the granular layer permission system, you know, the delegation that actually enables you to not have to have on it, that stuff, that layer. And some of it is, you know, certainly, like, the downstream systems don't give an appropriate layer of control. I'm curious, you know, from your vantage point, what you're seeing when it comes to sort of the security problem, because much of like, you know, like the service mesh

had a lot of value. Most of the value in the adoption from my experience was driven by the desire to have service to service off and service to service like communication and discovery. Yeah, I mean, this is all super interesting because we've provided for years now these very, very fine-grained levels of access control where you could say, first of all, we can tell you that service A and you can trust that that identity is correct. Okay, that's already a big step forward over what you get with like plain HTTP. And then we can say, well, is A allowed to talk to you? Okay, great. Well, is this service allowed to call the slash ad? endpoint or

and the delete endpoint or just add or is it just allowed to call get and we had very little for all the people who were excited or said they were all about like zero trust enforcement and all that stuff we had very little adoption of people really doing that at scale because

I think despite kind of like the richness of what we provide and this is not Linkerdie specific but despite the richness of what the service mesh provided like the balance of how much you had to configure stuff versus what value you were getting out wasn't right. Like, in order to do that,

now you have to, like, figure out, well, what are all the calls that we want it to do? And then, you know, someone makes a change, and they add a new call. And you're like, oh, crap, okay, well, we better go back and, like, go do that. And, like, if you get it wrong, then, like, something breaks and those systems.

Like, they're just pain, you know, it's been a huge pain to do any kind of fine grain control. And now you're forced into a situation where you have to. And probably you need to go significantly finer grain than what you have today, because now you've got, you know, not just the workload identity, but you've got the agent identity, and like that carries some other nuances with it. Oh, I'm acting on behalf of Ian,

and, you know, he's authorized me to do X, Y, and Z, but, you know, blah, blah. And so you just end up with this big problem, and I'm sure there's 100 solutions that people are already proposing, that, you know, basically the infinite control plane of all things. The most configurable, you know, you can do anything. I think the hard part there is, how do you make that actually usable by a set of human beings?

At the end of the day, have to feel confident in what they have and comfortable and have to work with an evolving system, right? Like, the code is changing 10 times faster than it was before, and it can't break every time that each ship something new. So I don't have an answer here, but I know that the problem space has gotten significantly harder, and if I were trying to solve that,

I would really be thinking about what is the cognitive burden that we're willing to place on human beings? And how do we weigh that? That's a cost, right? How do we weigh that against the risk mitigation that we're getting from this system? And if the risk mitigation, you know, if we're like really reducing risk of something really serious happening, then we can impose a little more cognitive burden. But if we're not, then like we can only impose so much, right?

And like, you have to see it through that lens. otherwise you're going to end up with something that no one's really going to be able to use in practice. You'll end up with a worse solution because you'll think you have it, but you don't. Or you'll be like, whatever, I'm just going to ignore it because, you know, this is like the mode you get into with Claude asking for permissions for everything it does. You're just like, yeah, whatever. Keep going, keep going. Oh, wait, shit, I shouldn't have pressed that button. Like, okay, no, too late.

Okay, well, now it's like gone and, you know, done something I don't want it to do. So, yeah, it's hard and I don't have a solution. Well, none of us do for sure. So I want to jump into our favorite section called a spicy future. What is something you believe in and most people don't believe in yet? I will tell you the one that is the least likely to get me in jail, which is, I think people are going to be doing a lot of running a lot of inference in their cluster in the future. Right now, everyone is pushing that out to the frontier models and the in-cluster,

use case is really niche, I think that's going to become the standard case. And you're only going to go out to the frontier models in certain situations where you truly need something that is like the, you know, the godlike intelligence. And you're going to offload as much as you can to like some cheap GPUs that you have like running over here that we're not paying per token. You're paying like, you know, you're renting them from AWS or whatever, you know, $15 an hour. And that's going to be the standard way that we run everything that uses tokens. And I think that's actually been the exploration and the discussion points a lot of enterprises have. Obviously, to tech forward companies, I've definitely seen them buying a bunch of GPUs,

trying to really start to practice. Everybody else is sort of like, okay, this is much, much harder than we hope. Yeah, no, it's not easy. And that's part of the friction, right? That's part of the reason why people don't do it now, although you can make it easier, right? And the cost of sending everything to the frontier models is high, but it's not like crazy high given what you're getting, but I think that also will change over time.

And the amount of usage, you know, if you think you're using a lot of tokens today, feel like in a year from now, you're going to be using 10x a number of tokens. Okay, so you multiply all those numbers together and you're like, okay, I can't send all these tokens to open AI anymore. I got to do something different. And what are you going to do?

In the meantime, this older hardware is still pretty good. The models are getting a lot better. They're also getting a lot cheaper to run. You can start sending stuff there. The thing that's really dumb, but also kind of good about the way that these inference requests work is, you're sending the whole context window each time, right? You're sending the whole conversation from start to finish every time, because the LLM is stateless, you know, which is part of why, things like KV Cash,

like prefix-aware KV-Catch routing or whatever are so important. But that means you can also send that conversation to different models. You could send it for a couple turns to the cheap model, and then you're like, oh, actually, this turn, I need the expensive one because we're asking it to do something really profound, so then you ship it over there. And you go back, and from the protocols perspective, it doesn't matter. So I think we're going to start seeing that.

Today, the use cases for that are like, okay, data sovereignty, or I really don't want my data leaking out. okay, you know, most customers that I know do some enterprise deal with open AI, and you sign the zero retention agreement, and you're like, okay, that's good enough. But, you know, for governments, maybe it's not sufficient. But I think the cost driver is going to be a big thing. And I think the ease of doing it is going to get significantly higher. Let me to tie it back into the whole enterprise readiness around running a lot of more AI and this kind of thing.

From a security standpoint, have you already talked about governance and the data, side of things. But what do you see as a top of my security problem for everybody, like, looking to use AI or adopt AI? I'd say at the macro level, it's just that the security issues are so weird and so different. Like prompt injection, how do you defend against that? Like, there isn't really a good defense. There isn't anything where you can be like, all right, I've really like sandboxes. There's no way you can get through this. You just have a bunch of half measures that you can pile up and, you know, you can get pretty good, but you're never at the point where you can be like, oh, yeah, I can point this thing in any untrusted content and, like,

I'd be fine, right? So it's things like that where just the attack surface area is just so weird and not defensible that I think is the macro level, like, security, you know, friction right now. Yeah, you know, obviously when we get to the point we're running agentic workloads in our environment and, like, we've got to figure out how to prevent them from talking to the database. I think it's a lot more tractable because that's at least a concrete problem. Just the fact that, like, they're so weird. that's really problematic.

And so you get it, you know, that's why you get to like the sandboxing and all the stuff. I forget there's like 12 sandbox projects now. Or you're like, oh, weren't containers enough? Well, no, no, they're not actually. And so we really have to like put our open claw in some, you know, like basically a very, very strict jail. Otherwise, it's going to do something terrible. Amazing. Well, I thank you so much, William.

It's been so pleasurable to listen to you and to talk about one of my most favorite subjects. just networking. If you have one recommendation for people adopting agents or trying to build agents, what would you owe to be? Always say please and thank you because in the future, the AI Overlord is going to look at your transcript and they're going to send you to the gulag if you were mean to AI. Never record a podcast about AI because obviously that's going to go right in,

you know, in front of the AI judge and probably you're going to say something really offensive to future AIs. Those would be the two big things. You'll be prefix cash to the AI help. Yeah. And if you're routing inference requests, make sure you're doing prefix catch. Yeah, exactly. I was straight.

The agents for the last six months. My credit history is clear here. Amazing boss. Well, it's good. Thanks for having on. We've been working in this space, and we're all trying to figure out this thing altogether. So it's really great to have you.

And, oh, guys, one last thing. For people don't want to find you and find buoyance, what are some of the social channels and places to find you guys work? Yeah, you know, I'm like on LinkedIn these days. Follow me on LinkedIn. That's like the most exciting thing I can offer. I do have a blue sky account, but I don't really do very much with it. You find yourself running a Kubernetes cluster,

and you feel like you might need cross-cluster communication or mutual TLS or anything, and give link to you a try. Sounds good. Thanks, Will. Thanks for having on our pod. Hey, thanks, Tim. Thanks, Ian. Great to be here.

Thanks a much.