In The Arena by TechArena - FarmGPU on the Ticking Clock Driving Neocloud Economics

Episode Date: July 17, 2026

In this episode of The AI Hedge, host Marc Austin, Founder & CEO of Hedgehog, is joined by Jonmichael (JM) Hands, Founder & CEO of FarmGPU, for a deep dive into the emergence of neoclouds—AI-first cloud providers purpose-built for modern training and inference workloads.

Hi, I'm Mark Austin, founder and CEO of Hedgehog. This is the AI Hedge, a podcast where I talk with industry leaders in AI infrastructure about the opportunity and the risk in AI and AI infrastructure in particular. And so my guest today is John Michael Hans, who is founder and CEO of FarmGPU. Welcome to the show, Jam. Hey, Mark. Yeah, Mark's my partner in crime here as we did this OCP reference architecture for the hedgehog and OCP fabrics. That was a lot of fun. Yeah, it was a lot of fun. Yeah, you and I have been on stage a few times now. We've talked a lot about technical networky things. I kind of want to change it up this time because it's just good to have a fresh conversation. And I think you've got a lot of really good market perspective. We were talking about the Open Compute Project. Instead of the reference architecture, I want to talk about the neocloud working group that I believe

you're chairing at Open Compute Project. So let's just start with what the heck is a neocloud? A lot of people probably never even heard the term. What's a neocloud? Yeah, it's interesting. When you looked at the cloud providers in the past, right, you'd have AWS, Microsoft Azure, you'd have Google Cloud. And those are really big options for hyperscalers.

And if you were to ask somebody five years ago to say, hey, this brand new, category of cloud providers are going to come up and they're just going to take half the market from the largest cloud providers in the world. You'd be like, that doesn't sound right. You wouldn't have said that even a year ago. Yeah. There's no way that that makes any sense. So if you think about a neocloud, the term is really initially to describe a cloud provider that is very specifically focused on AI workloads. And now, as we're getting farther down into the agentic era here, almost all all of workloads will be AI workloads in some sense or another.

Even simple database applications, they'll maybe do some query to an LLM or if you think about even just traditional workloads, which were web hosting, database and caching and stuff, almost all of those will touch some AI service in some way or another. So if you think about the future of clouds, most workloads will be AI workloads. And the reason why NeoCloud has made sense,

one is that it sounds crazy, but even the traditional clouds, their data centers weren't set up. basically to host AI and GPUs equipment at scale. What's the difference between an AWS data center and an AI data center? Yeah, I would say that the core difference in an AI data center are like to serve. And obviously the hypers are doing bigger giant clusters and stuff now. But we're saying in the beginning where guys like Core,

we've had a head start on basically designing data centers that are AI first. I would say the first and foremost is GPUs are the power design. are much denser than traditional compute. One, B200, HGX, say 14 to 16 KW Kowatt max. We have a customer doing a training run right now, and they're actually using 11 kilowatt per server in production right now. So that's measured today. That's one server, right?

So if you're talking about the super microreference architecture for an HGX is a liquid cool servers per rack, so you're talking about potentially 100 to 140 KW per rack. This is much higher power racks than traditional DCs had done. Yeah, so power distribution is a big thing. Cooling is another thing. I would say, yeah, the three distinct factors of an AI data center are high power, liquid cooling. Again, liquid cooling is really needed for that density.

And the density is all about scale up domains in a rack. And so, Invidia invented this NVL 72, which has NVLink switches that have a scale up domain within a rack. And when your domain is the rack, well, now you have to cool everything in that rack. You're talking about a GB300 NBL 72 rack is like $5 million is like the unit of compute now. And that is like to serve one model. Okay, when you have that much density, 72 GPUs in one rack, the only way to do that is liquid cooling. And so liquid cooling is not optional anymore.

If you want to do these high powered racks, you also need with cooling for the chip power density and the rack power density. And then, of course, the third one is high speed networks, which markets. That's a zero. That's the third. Wait, wait, why does the network matter in this? Yeah, most traditional DCs were like 100 gig. So the data centers, when they moved to these 800 gig switches and fabrics, AI has been distinctly split up into two networks.

One is the back end network or what Nvidia calls East West, which is the GPU to GPU communication. And typically on like a cluster, like a B200 cluster, you'd have eight connect X-7s to eight 400-gig nix. That's a lot of network bandwidth between host nodes. This is way more than pre-AI. Why do you need that? Why does the workload need all that? network. Yeah. So in a training run, there's a step called AllReduce and one of the benchmarks we ran as part of the cluster benchmarks is Nvidia Nickel, the collective communications library. You get you

basically stress test the entire cluster where it's in all to all communications. All the GPUs have to understand what data is in all the other GPUs at all the time. So yeah where you're talking about these different domains like a scale up domain is if one unit of the scale up domain is an HGX server, then all those GPUs can talk to each other via NVLink. EnviLink is much faster. than even PCIE Gen 6, right? MB leak, we're talking about 1.8 terabytes a second per GPU of communication within the domain. If you go outside the domain, now you still want that to be really fast. You don't want it to be 20 times slower, but 400 gigabytes a second versus 1.8 terabytes

a second. You're about four times or four to five times slower when you have to go outside the box. But that's still good enough to do these bigger training runs, right? Ideally, you want all to be super high bandwidth, but it's always just the tradeoff of complexity and cost. So that's kind of this back-end fabrics. Again, Nvidia calls East-West, but this is really how the GPUs talk to each other. And this is very distinct in AI networks where it's very sensitive to elephant slows and and Jeter and Nvidia has, they have Spectrumax for flow control and all the new stuff

going on and ultra-Eternet and stuff to solve some of these AI, very distinct problems for using Ethernet in AI workloads. And the other side of the network is really the storage and the Internet. And again, before it was like, oh, storage networks don't need very much performance, it's just 25 gig or 50 gig and it's totally fine. Well, if you're doing inference workloads now and it's their AI workloads, you're potentially talking about like a KV cash offload serving production inference where you're trying to lower the dollar per million token and you want a high cash rate. You're talking about hundreds of gigabytes a second of bandwidth to like a B200 node. You can't get that on a, you know, a nick that is a 25 gig make or something, right?

Now in production type inference use cases, you actually also need a lot of storage bandwidth, which is not typical of, again, not typical of traditional network sharing. Right. Okay. So we got this AI thing. It creates whole new requirements for AI or for just data center cloud infrastructure, right? New power distribution architecture, new cooling architecture, new network that has to be a lot faster.

New storage architecture. A new group of market entrance address that need in the market early. You mentioned core weave. they went public last year. Yeah. Oh yeah, we were talking about what defines neocloud. So yeah, I actually think this disingenuous is to say, oh, they're just neo clouds. They're not real clouds.

They're just fake clouds like they're not true at all. I think, yeah, I think serving a lot of these guys, Nebius and Corley, they serve a lot more than just GPU as a service. It's not just bare metal or GPU as a service. I think a lot of folks think that all the new clouds are just the same. They're all commodity. They all serve the same H-100s, same B. 200s and bare metal and it's all the same. And it couldn't be farther than the truth.

Most of these Neo Clouds have either some service of their specialty in or like one of the partner we work with is RunPod. They're very focused on developers. They have 500,000 developers and they've curated a really nice user experience for developers to spin up a pod really fast within a few seconds and get in with Pai Torto Tukuta. Well, I think it's if we talk about the infrastructure requirements for AI, but AI software development is a fundamentally different thing than a few seconds. Yeah, a lot of people focus on these MLOPs and fine tuning and how to orchestrate different A.O workloads

across the company. There's lots of various levels of neoclods, but I think it's disingenuous to just say, oh, they're all neoclods and all they do is the same thing. It's a different way of making software, which then drives different platform requirements and different developer user experience requirements, right? Exactly. And one of those use cases is, hey, renting a large giant cluster to a frontier lab to go do some training or production scale inference. And I think unless people have been living in a whole, chat GPT and quad have gotten pretty, pretty good. I'm using them all day every day for everything we do.

Yeah. It's not all just training anymore. Yeah, as inference drives a huge chunk of the compute demand, you know, most of these guys, I think something, Anthropics said something like the conservativeness on the compute was like they put what they thought was like pretty crazy. Like, hey, we're going to have to 10x our compute in a year as far as their plan. Their demand 80xed.

And that was like, and this is just the start. So how do you plan for something that's growing that fast and you have to invest now gigawatts worth of power to keep up with the demand for inference? Yeah. So a lot of these guys, as you've probably seen, are doing now major deals with the neoclouds at the frontier lab level where they're renting giant clusters and getting high amounts, anything that's 100 megawatts and above trying to scoop all that up because they need it. Right. There's no way they're going to scale to. Yeah. So, yeah, Neo clouds.

Yeah, NeoCloud, again, broad term, just means a cloud provider that's focused in AI workloads. Yeah, I'll use the cliche. There's a paradigm shift in how we do software that's happening right now. So that's created an opportunity for new market entrance. And what previously, I think we could call it an oligopoly, right? Three-member cartel, AWS, Azure, Google Cloud, providing the majority of the cloud infrastructure for the majority of SaaS or web apps or mobile apps. or mobile apps that preceded this AI era. So we now have what's becoming a more competitive market

for cloud infrastructure services. Like, how many neoclods are out there now? Our friends over at Semi-analysis have this thing called ClusterMax where they rate the neoclods and they also track them. But I think on their main page, I think they're tracking like net out looks like between 50 and 80 or something like that. Now, there's probably more of smaller providers that just have, you know, a few GPUs or,

small cluster or something like that. We need to keep talking to people almost every week that are planning to build neoclods that aren't on that list yet. So I think there's actually a lot of new market entrance. Yeah. And back to like our original thing where it's like if you came in, there's one camp of people that are like, well, can't just can't the, you know, Azure and AWS just lower their GPU prices and get all these guys out of business and saying, well, okay, but clearly

you don't know. Yeah. But you don't understand the market. So let's talk about that. If you actually look at the price for a B200, there's a pretty big spread in the market on prices, right? Well, there's a market price of like everybody else, and then there's a market price for the hyperscalers, meaning like just on their on demand. Like if you want to rent a B200 from AWS, you're going to get ripped off. You're going to pay three times the market price.

Why? Why do they charge 3x market price? They think that I guess their customers are big enterprises that already have some giant AWS account. Their credit cards are already in there. And their claim is that they can connect all these other. They do have service lock-in. They have- vendor lock-in is one of the core thesis of Neo-Clouds is,

hey, we have this decentralized compute, and how do we enable it to be decoupled from vendor lock-in and be able to run. I'll tell you some of the stuff that we're working on to enable this, but there's a lot of like encryption and tech that we're developing, basically to enable the ability to run in a decentralized way and not worry about your data being stolen. If I'm a CIO at a Fortune 500 company,

I've got an existing relationship. relationship with one of the big three hyperscalers. I've got a multi-million dollar, maybe even billion dollar annual spend with them. And I don't have a whole lot of governance on which development teams are using which services. So everybody starts experimenting with AI, they start running GPUs. And yes, 3X the price and it's going to take me a while to figure it out, right?

Yeah. And it's fine. I had a blog post I wrote a long time ago, I find it. But like basically it just summed up if you think about why the cloud exists, at all. It's supposed to be this on-demand scaling. So you can, you don't have to put upfront CapEx. You just pay as you go. And that offers people a lot of flexibility to only have to pay for what they use. And it was supposed to be this really cost effective and lots of scalability. It definitely is when you're developing your prototype. Oh, for sure. And yeah, there's lots of

things that the cloud does very well. I would say security. Like AWS and Azure take security very seriously. And they take data durability extremely seriously. Like you put data in AWS as three. When they say it's 11-9's durability, they mean that it's been designed so that there's a higher chance of a meteor hitting the earth in destroying it than there is, then we will see your data. It's just math. That's probability. Statistics. So that stuff is real. And that is one of the values of the big clouds.

But that doesn't mean that they're also not just completely ripping everybody off. So new entrants are competing on price and on sort of breaking service lock-in with the big three incumbents. But even within the neocloud market, there's a spread on spot price. for on-demand GPU rates, right? Yeah, I would say it's not as much as people think. Like one of the things that's interesting in the market is there's somebody that has a compute and then maybe they need some help. Maybe they're small and they just bought some data center and bought some GPUs,

but they don't have a partner. So they would go out to one of these aggregators or platforms and say, okay, can you guys just do the work for me and get these GPUs onto your platform and then we'll do some revenue share between you. And a lot of the times the data center operators will say, okay, we're off for you, for instance, will offer you $3.50 an hour for a two-year contract on that GPU. But we're going to actually sell up to the customer for $390. They get to pocket money in between. So there's brokers and aggregators and now people are selling compute futures. I guess it's not surprising

with trillions of dollars of AI. The temporary supply and demand phenomenon? Or is it sustained? Yeah. I think at some point, the middlemen will get deleted, right? At some point, if you're not adding value in the chain, then you're not going to be able to take any margin. So either you're finding an end customer, which is a real thing, right? Doing one of the jobs that a lot of these brokers do is they call it off take. You know, it's really just a end user demand of finding a customer that wants to rent the cluster. And they pair it with the person that has the cluster.

Yeah. Or a lot of these companies have their own front end where people can go rent GPUs. And then there's the gold tier, which is some of these providers are actually running inference endpoints. But it's very hard and complicated to run a inference endpoint that's profit. It's very easy to run a unprofitable inference endpoint. You just BOLM serve, here's by endpoint. It takes five seconds. But to run it profitably with lots of concurrent users

and driving down the dollar per million token cost is very hard. So it's hard. There are really big incumbents who have a lot of capital who right now are commanding price premiums. What's the case for and against Neo Clouds surviving three to five years from now? What does a neocloud need to do to be a long-term competitive player in the cloud infrastructure market? I presented at the OCP, Emia Semit in Barcelona a few weeks ago. I'm co-chairing the OCP scaling AI clusters for neocloud's workgroup,

which is basically all about how do we bring all the OCP goodness that is going on in all the other work groups and apply them to what the neoclouds need. What are the pain points of neoclodes for scaling? And how does OCP help solve those? It's kind of funny when I was talking back and forth with Claude, doing some research for my script and in the blog post. And it came up with this, it said, yes, the four getting's problems, getting facilities.

So basically I mentioned all the AI ready DCs that have high power, liquid cooling, high amounts of power capacity. And then getting electrons, the grid, how do you secure multiple megawatts? Getting hardware faster enough. Now, anybody who's in our business knows server prices are going through the roof. obviously Dell and other folks had a really good day to day in the market after earning. Server prices are going up, margins are going up.

Margins are not now just only being consumed by Nvidia and memory vendors. Now the margins are going to other players in the ad ecosystem. So getting hardware is incredibly challenging. There's long lead times. Everything's sold out. All the supply chain is impossibly hard. And then that last problem in the old clouds is getting customers.

It's like, okay, now we have all these things. How do we go actually sell it all the way through an ad user and get customers? So in those, problems. There's getting electrons, I would say the public markets are certainly favoring that as like a guiding principle for how to value neoclouds. So any of these neoclouds, they could say, oh, they have a $100 billion deal with some cloud vendor, right? Or, you know, $50 billion deal with Anthropic. But if that neocloud can't deliver those GPUs to that customer, how can they deliver those? Well, they need power. They need data centers. Well, if you don't have those, it doesn't matter if you can

buy GPUs, there's nowhere to put them. Right. So that is one way to value in Yilkla, right? I think the public markets have dialed that in for CoreWeave of knowing how much capacity they have under management and they know what the prices they have to the end customers. They know how many racks are getting delivered by what week. I work with semi-analysis on their industry and analyst reports and they know everything. They know how many GPUs are going to what customers at what date and at what price they're

paying. It's insane. Yeah, that's one way. But there's a lot of like where we play like right now. in the long tail, right? We're targeting smaller data centers, upgrades, developers. There's a ton of revenue to be made on the other side too, just like the rest of the world.

And a lot of people are going to ignore that long tail. But there's tons of opportunities and specific customers and workloads and revenue opportunities to. So I think the short answer is where you have a new market where there's a massive amount of demand and that demand is continuing to grow, that creates opportunity for more market entrance, more competition. Competitive markets are generally better for buyers. And there's no reason why that can't continue to persist. All markets face a certain amount of consolidation at some point, but like, yeah, well, the interesting thing about these aggregator type companies, or I would say

RunPod is one of our partners. They call themselves an AI platform. But they have like a distributed data center strategy where instead of just having one data center provider, they work with lots of smaller neoclots like us and they run their platform on top of them. So in these business models, now all of a sudden, then this decentralized, disaggregated thing, back to our analogy versus the hypers, right? Back five years ago, you'd say, oh, to run a cloud, oh, no, you need thousands of engineers, you need giant sales teams, you need BD, you need all this stuff. Now, you can run a small profitable neocloud with just a handful of employees. If you have folks to run a data center, in our case, we're a small company. We've got that Hedgehog AI network

that's fully out of me. Yeah, we have all. And we can, you know, not being panatic, Mark helped us build our first cluster on the networking. And that was a lot of help. We had never built an 800 gig back-end fabric, but before the first cluster. And so that was our topic of OCP Global Summit last year, is like building this first cluster and all the things we've learned, you know, along the way.

But yeah, you'd be like crazy to think you can service like hundreds of millions of dollars of revenue with five employees or 10 employees. But people were doing it right now. Right. So we've talked about semi-analysis a couple of times now. I think it's worth explaining who semi-analysis is. it's worth explaining this ClusterMax rating system that you've talked about. It's worth explaining the different rating tiers and the criteria that you got to meet to reach those tiers.

Can you talk about that a little bit and why any of that matters to you? Yeah. This ClusterMax is an interesting thing that. So Semi Analysis is one of the largest analyst firms for AI. They were used as a source for like on the Quarlyb S1 when, they went public now on the SpaceX S1 for their public. So they are the industry experts and trusted market analysts for all things AI. And some of the products that they serve are like, we're subscribers to their TCO report, which basically is like, and doing a nice plug for Dylan

and his team here. So Don't Patel's the founder. But they have like the TCO report, which is like total cost of ownership for owning a cluster and generate revenue and forecasting cash flows of TPUs. They have a data center model, which is like looking at all the different power and data centers go out of the U.S. They have a accelerator model to be out beyond the Bidia, like all the TPUs and who's shipping how many volume of what units when. Obviously, people that are trading stocks, like want this data. They want to know who is the biggest market share and much revenue of these kind of companies you're going to generate. They look at the memory market for HBM. But yeah, so they came up with this idea of like cluster max, which is, okay, if you're a customer that's renting a cluster, what's

user experience and what are the metrics that actually matter. This turns out to be the same list of like what makes a good NeoCloud. Because what this is like the Bible to all the neocloud providers. The things that they've identified are actually absolutely 100% the pain points of like managing big cluster. One of the categories is security. So like I personally did our SOC2 compliance. So I painfully I know the process and so doing stuff like that and ISO 2702,001, physical

security and standard specs. They have tenant isolation for multi-tenancy. So there's, that's a hedgehog feature. Yeah. Yeah. Back to that cloud thing. When you're comparing a neocloud security versus a hyperscaler, like, yeah, the hyperscaleor has been working on security and they have like giant security teams. And so this is the stuff where I would say probably is the biggest gap.

I would say to like a small neocloud versus like a large hyperscaler. But not to say that it's not impossible, like Mark said, if you have just good engineering and like, like us, just discipline. I have a security background for, you know, I developed the first self-encrypting drives back in Intel and did lots of work across the industry standards on data arrest security. I think it's very important. They have a life cycle, which is, you know,

basically going through this process of onboarding and spinning up a cluster and provisioning and doing rentals and transitioning for customers. There's orchestration. I know, like they're big fans of Kubernetes and so, I'm on top of Kubernetes for training workloads. I've talked to a lot of folks in the inference market, and they're not all Kubernetes based.

A lot of the stuff, we do our own KBM VM stack, and I think that's what we prefer for AI workloads. But we also do all Docker through Red Pod. But this is certainly very much focused on the orchestration, basically be able to easily have one click and be able to go deploy a workload across an entire cluster. Storage, this is our favorite category. Obviously, if you talked about, you know,

if we had to describe what's far. GPU's superpower and what's our differentiation in the market. We're absolutely, we can say it with absolute certainty, you know, we're best in the world of storage. You kind of know storage, right? I neglected to give you an opportunity to talk about your background, your bio. What were you doing before you started Farm GPU? Yeah. I used to be the product my manager for all the Intel data center and SSTs.

And then Intel sold out that business unit in 2021. I actually left Intel and went to a blockchain company, Chiya. But that was before I started FarmCupU. But those folks at Intel that were part of the NAN business unit are now at a company called Solidime. Solidime's spun off in their own by SK Hynix was the one that bought them. So we actually run Solidimes AI Lab at FarmGPU. So we have all the latest storage IS fees in the lab. We're doing brand new research on all the latest KV Cash implementations and AI workloads and training and performance.

And so we've got a top secret project that you'll see, I guess, next week is they're going to be the public blog post. But we built the reference architecture for an Exabyte scale. scale S3 cluster, and that was the collaboration between a couple partners. And so there's a bunch of just crazy fun projects we're doing on the storage. So yeah, we are absolutely at the forefront of storage in AI workloads. Just a great example of where a neocloud can differentiate by doing a really good job on something that's really important for an AI workload storage being. Absolutely.

Yeah. And storage, the way that they, some of the storage things that they care about are like cluster level performance that you're having a ability to have a persistent volume across block file and object in a data center. They like WECA and DDN because they're NCP partners, but we deploy all the major top ISPs, vast, fast data is the most popular. That's live in our data center. And then we've got S3 through Min I.O and we've got super high-performing parable file systems with WECA. We've got GPU direct storage with a company called grade. We've got all kinds of crazy, crazy stuff. But yeah, you know, we worked with that month. So you're feeling it on the on the storage cart.

criteria on the cluster max rating. What are some of the other things they evaluate? So one of the biggest categories, just in general, like back to, okay, are all these guys the same as it a commodity? It's reliability in monitoring of really, really hard. There is no like out-of-the-box thing that just does all the monitoring reliability for you. So that, I would say, we've developed a bunch of custom products for Farm GPU. Our roadmap has really been derived of like just solving the problems we had to solve to manage a production data center, which is have to deploy an observer.

probability stack, monitor and diagnosing GPU failures. We see all kinds of failures. There's NVLink failures and switch failures. There's optics failures. There's GPU XIDs. There's out of memory bandwidth. There's driver crashes, kernel crashes. So it's interesting.

Like this morning, I had a call with the analyst team at Uptime Institute. It's been around for a long time, right? They do an annual survey of data center operators. They've been doing it for probably more than a decade. And so they kind of have like a pretty traditional. definition of uptime, right? You're online or you're offline and it's divide the number of hours. And I've been working on this AI network planning tool to sort of like

quantify the impact of reliability on your profitability as a neoploud. And I'm like, those metrics don't really seem right because you look at like this meta paper that they wrote on Lama 3 and you look at the incident rate, it's like significantly higher, way higher. And the incidence types are different, right? They're more like almost like what's kind of anomalies that impact flop utilization, right? Yeah, we're feeling those very visceral. But yeah, the idea of this good put, which is if you have a GPU failure during a training run,

and now your state is all messed up and you have to go back to an old checkpoint. Well, now you have to load the checkpoint from this that they've been saving and then roll back. And so you might lose a couple minutes or tens of minutes of training. Okay, well, if that happens many, many times a day, now you've actually lost a bunch of real time. Yeah. They call it good put. The GPUs are working. They're online.

They're doing stuff. But if you had to roll back, you lost a certain amount of time and a certain amount of work that you've already done. This is, by the way, how often and frequently an AI lab will be checkpointing during a training run is very dependent on what the failure rate is. Because if there's more failures than they want to do checkpoints more often, that's not free. There's now there's more network and storage bandwidth that you have to drive to that cluster to do more frequent checkpoints. But you can try to improve the good put. So yeah, semi-analysis now has a cool little TCO calculator that kind of has this idea of good put, which is, okay, if you're renting a cluster,

but there's incident rates and you have a certain amount of good put, like how does it actually impact your TCO? If you actually include that in a TCO model, it's a very clever way to actually show the value of reliability. Because if you're talking about what is the value of reliability, it's a lot more than just the good put and availability and uptime. It's like there's a huge distraction from engineering team with things break. It is the first thing that like we would say,

like we were absolutely the smallest provider on RunPod's secure cloud platform for a long time. We're not the smallest provider anymore, but like we were for a long time. And meeting those requirements was not easy. They've designed those requirements to have redundancy on the network, redundancy at the power and the data center level, redundancy at the switch level. Yeah. To meet those uptime target because it's really disruptive and distracting for customers when things fail. Yeah.

Okay. So you were showing me earlier today. I think you call it haystack. You've done like a bunch of innovation with Grafana to come up with an observability stack. Can you talk about haystack and sort of how you're using hedgestack? Open Observability? I'll just do it, you know, shameless plug for our Neo-Cloud products here.

We have two products that we developed in this category of reliability monitoring, basically to make our job easier, right? One of those products is called Haystack. That's what we call our observability platform. And one of the things that we did in Haystack was through, like Hedgehog exposes all the switch metrics through their API and their control plane.

So we just have all those metrics going to our Prometheus database. And it was really easy for me with AI to make some really beautiful dashboards to basically look at all the switch status in real time to look at the bandwidth and utilization of all the training runs, looking at switch optic status and link status. If there are failures, we can look at the switch counters and get all the sonic Swiss level info up to our control plane. When you talk about observability, one part is just being like, oh, we have a dashboard so we can observe the status of the entire fleet in one pane of glass and see what's going on.

And then the other is getting alerts when things are down. So when a link is down, the customer's training run stops. we don't want the customer to report that to us. We want to find it before the customer finds it and fix it. I have like a two-hour SLA for our premium customers doing training. So where if we detect that a link is down, we've had all kinds of optics quality problems. I want to get there.

But where we have to go detect a link is down, send a technician out to a server, and they have to know exactly what the right optic to pull is, and then swap it or clean the fiber. And then we need to know that that's back up. So there's one part of monitoring observability that, I mentioned like it was really easy for us to go expose those from Hedgehog's side to go pipe those into the rest of our stack. But the rest of the stuff, our other's core stack is in Haystack. We have

two really unique features that nobody else has. One of them is container profiling. So we have a LLM do classification and all the containers going in and find out are they training or fine tuning or inference or image gen or video gen. And we can use that kind of data to optimize tuning on the server based on what kind of work what people are doing. And then the other is one of the we're going to be open sourcing at OCP is our NVME predictive failure engine. So obviously we know a lot about NVM SSDs. Yeah, sure. Lifetime in Intel and built the NVME spec with Intel.

But I was also the very first reviewer of the OCP NVME SSD spec. And so the cool thing about the NVM SSD spec in OCP is it was developed by Meta and Microsoft for the original authors. They just merged the actual cloud requirements into that document and then open sourced it. All of the custom firmware features an SSD vendor. The Open Compute project, we keep talking about OCP. Oh, yeah. When I might know what that is.

So that was some cool stuff we built. On Haystack, we built a predictive failure engine for SSDs. Yeah, and you were talking about scale earlier. AI scales out a lot faster, right? Just consumes a lot of resources. When you start reaching a certain scale, you start having statistically certain failure rates, right? Yeah, I had one of our agents scanned through all of our production incident response tickets that we have. Like when there's a GPU that's dead.

down and there's a customer workload on it. We have a SOC2 format for our incident response. And I have our SRE now all through agents basically generate that instant response and send it to the customer for what was the actual downtime problem and what was the root cause and what we'd do it remedy it. And I had it scan all of our production issues and our agent kind of put in categories. And there was GPUs. There was a lot of, it's funny, they were out of the storage ones.

There was like 10 failures in storage, but none of them were SSD failures. It was all like file system or mounting or something hung on some. service or and then there was optics failures and network failures that we've had where we there's actual low level five stuff that and dirty like we talked about how hard it is to build 800 gig networks a single spec of dust can literally destroy an entire link and it's just an absolute nightmare to manage a scale so when you start scaling a lot of configuration you got to do too right and if you it's funny i was just on a panel oCP invited me out to a panel in orlando to speak at this fiber connect thing i'm not a network guy i'm just a just a data server guy

But it was fun. What out there, I met the guy that from OCI, he's the lead network architect and reliability engineer for OpenAI Stargate at OCI, $300 billion GPU project, largest cluster ever built. And listening to him talk about, it's like funny, we're talking about co-package optics and all the next stuff in LPO and all these tricks that the industry is looking at to basically solve all these reliability challenges. But if you solve a power challenge, you might solve a hot swap ability or a service ability

requirement and there's all these different tradeoffs you want to make at scale. But it's funny listening to him be like, oh, we have those same problems. There's dust and fiber. He destroys things. But dealing with how do you solve that problem at like a scale of hundreds of thousands to millions of links versus like if it's us and it's like a handful, you know, that's a very different type of scale, right, where you have to start thinking about problems in a whole different way when you're talking about scale. But you talked about, okay, yeah, you can have this massive amount of revenue with a small staff that I guess implies, oh, and this stuff's super complex. It implies that everything's got to be automated, yeah?

Yeah, I guess that's the second shameless product plug we have, which is for this monitoring reliability, we develop a product we call Shepard. Of course, yeah, we're farm GPU. All of our products are named with farming memes, of course. Yeah, that's the only way. Yeah, Shepard is our autonomous SRE. So basically, SRE is a really important job for a data center. Ineoclod, it's a site reliability engineer.

So basically monitoring the fleet. And if there's an issue, you'd be able to triage, root cause, send the issue to an engineer, and then communicate that to the customer teams and stuff. Yeah. So we've basically, what I just described,

if you're going to scale the GPUs under management, and these things are much more complex than traditional CPU nodes. We have a bunch of CPU nodes in our fleet. No problems ever. Zero problems ever. GPU servers, problems all the time. Every day, we see all kinds of random stuff.

So it just won't scale. You're not going to, going to be able to scale to hundreds of thousands of GPU servers without automating all this. So one of the things we're leaning in and leveraging agents to the max is in this category of debug and monitoring predictive failure. You got haystack on the observability. It's using Hedgehog Open Observability for the network telemetry and network logs. And then you have Shepard on the sort of automated SRE, which is presumably doing some remediation of issues that you're finding.

It's reading logs. its reading system and demessage logs and kernel log. It discovered autonomously that one of the links was unplugged on one of these switches. Because it said, oh, one of these interfaces is throwing billions of errors. This is not normal. The agent was smart enough to just say, oh, this was a switch host name, and this is called like Phi error count. And it's incrementing.

So let's flag that as a bad thing. Yeah. And then presumably, I'm going to guess here. I don't know. I don't know the answer to this question. I might get an embarrassing answer. But we've sort of designed the hedgehog product where, okay, you get this open, observability and then you get this cloud native API so that you can remediate faults in sort of

full loop automation. Are you using the hedgehog API in your remediation with it? Yeah, I would say a big chunk of our problems have actually been physical problems. And again, this is just that our data center might be a unique. Like our first data center is like also a lab where we do a bunch of research and people in the data center moving stuff around and swapping cables. And these fiber connections are absurdly sensitive. Right. You bump it. The leaks go out. But like yeah, the beautiful thing about these APIs, and I love, obviously the Hedgehog is all open source. We can go have our agents read their documentation and go figure out, okay, what is the API spec? How do we implement that and use it autonomously? So anything that is agent ready is going to be in our camp.

We're not looking, having any humans code stuff by hand at an API. But with the network stuff, there is a connection to the physical world, right? And if our problems are a physical problem, there's a technician has to go to a server and pull out a optic. You can't tell that with an API. So anywhere that can be remediated without a human in the loop, we absolutely are moving towards that. You're always going to need remote hands to remove that faulty optic or whatever the component is and replace it. Yep. Okay. So we talked about a lot of things.

We mostly talked about operations and the things that semi-analysis rates you on in their cluster max rating. So that you hopefully go. Oh, I guess we forgot the last category. We forgot the last category. It's networking. Oh, okay, good. What seems like an important way? What do you care about in the network?

Yeah, I mean, I think actually the biggest change that I think that was interesting that obviously Hedgehog was like one step ahead was embracing Ethernet, right? Ethernet is now, it seems like it is one over Infiniban. NVIDIA has put Ethernet in their Spectrumax reference architecture for B300. And there was just a brand new spec that OpenAI had written and gotten industry support from Nvidia and Intel and NAPD and others called MRC multipath Bible connection. This is very clever and it's all taking all the good stuff that will eventually be in Ultra Ethernet and basically techniques that were designed to make networks more resilient

to failures and keep going and training when things go down. So yeah, I guess I'll give it to Hedgehog for being ahead of the curve on Ethernet. But yeah, at the end of the day, what the customer cares about is performance and reliability on the network side. the semi-analysis customer max tests for networking, they want to see the real network bandwidth, which is the nickel all reduced, which is how fast that the Iraqi B2 RDMA networks work in real life,

in real GPU workloads, not just quick benchmark. Yep. Okay, so operations super important, customer experience, super important. Running a neocloud business, I want to back up to that for a minute, because to get into this business, you've got to find a way to finance, buying a bunch of GPUs, which are super expensive.

And then you've kind of got this lag time from the time that you've bought the GPUs to getting them to where they're online and billable. Tell me about time to GPU value and the bullets that a typical neocloud operator is going to sweat while they're watching the clock tick. If you think about the order of operations, one example would be, because the demand is so high, customers are willing to sign a long-term agreement, like for 18 or 24 months and then potentially pay up front for a cluster that will be delivered in 60 days. And they are not going to do that with a provider that's not reputable, right?

There's zero percent chance that they're going to do that with just a random provider. But in that model, then I have to go take that money and then put it down on a down payment for GPU servers. We have to go also spend our own money to go build out co-o space. In a megawatt cluster, you'd have to go, if it's air-cooled, you have to go get higher power PDUs in there and build a cage and build the air-cool system. if it's a liquid cool, it's like getting the chillers and the loops and the CDUs all hooked up. That takes real money. So before you even start day one of putting the GPUs in, like for a couple of megawatt cluster, we might have already spent a million dollars, but day before we even start.

Now, when you get to that point, like on a typical GPU cluster, OPEX may be 10 to 20% of the total revenue per month. If you have a GPU cluster that you've installed, but it's not generating revenue, that's going to be a huge problem because you have to pay the COLO a bunch of money to get in there. And then as soon as you put the equipment and you're already paying the KOLO, one of the biggest misconceptions is that power electricity costs is one of the biggest drivers of TCO. The dominating part of TCO and OPEX is actually the data center shell lease. And there's no free lunch.

You have to build an AI ready data center. You have to buy there's transformers. There's cooling units. There's all the infrastructure, PDUs, all the infrastructure required to build out a data center. It doesn't matter if you're in Texas or Colorado or California. The equipment is the same equipment. But so if you're paying that shell lease, And right now, the going rate for Kolo is, I'll say, $150 per kilowatt per month.

So on a megawatt, this might cost me before even charge for power, just to lease the building and reserve that power for me as a customer, I'm paying $150,000 a month. So if you're three months late to a cluster, now you've missed on a megawatt cluster, I've missed $1.4 million a month times three. And then I've also lost $450,000 on OPEX plus all the money I put up front. So if you're three months later, and it's a rapidly depreciating asset and there's interest expense on it. Yeah. And it's also a high value asset.

So every day you're not renting that GPU is a day of lost revenue too, right? Yeah. And it's this time to market, time to revenue for GPUs is a very important metric for us of like, how fast can we when we get a GPU shipment? So it's coordinating the data center build with the equipment rack and stack.

It's coordinating it with having the network ready before we install the GPUs so that when we install the GPUs, we just can plug the optics and then test all the links and get everything ready to go. It's hard because in a perfect world, all those things are on different cadences for delivery. So timing the data center with the equipment shipping at the right date and you don't know if things get back ordered. I mentioned like lead times are just through the roof right now. So it's really hard. You're like a general contractor who's managing a construction project with power and cooling and a technical program manager managing an IT project, installing compute storage and networking, and you're in a supply

constrained environment where there's long lead times on a lot of components and sort of availability risk. It's a lot to pull off. Definitely. And I would say there's not like a secret sauce to that besides being like good operations is incredibly challenging and you need good people that understand supply chain and sourcing and timing and working with co-o providers. So given all of that supply chain risk, what's the value of open source and open specification computing? Does it help you hedge some of that risk? Yeah. So for folks that may not know, FarmGP was customer number one for this new OCP collaboration for the open reference architecture for training and inference. Yeah, you contribute to a lot of your purchase code. Yeah. Thank you for that. Yeah. And I had to thank

Hedgehog and Swastika for doing a bunch of obviously the heavy lifting. But the idea was that we want some reference validated solution that's been documented in obviously open source is a way to have other people verify and contribute to the work but having it seen as a real viable solution now so especially that helps with showing customers like no it isn't just some random thing we built no it's like a OCP reference architecture so it's a known good thing and now that we have built it once we're going to use that for all of our other clusters because we know it works and the TCO is good yeah the fact that it's over-sort. And it's also that openness, it should help you, like, are you looking at

disaggregated inference? Are you looking at putting not just Nvidia B-200s or B-300s into your data centers? Are you looking at sort of the whole- We- We- And on the inference, I would say we're open to any XPU that generates revenue is good for us, Neo-Cloud, whatever it is. I think the next big challenge we're working on is these storage ones I mentioned for now that people are not just running inference like for testing that are running in production.

They need KV Cash, Osload, and, high amounts of storage. It is super high bandwidth to the GPUs. And so that's one of the things we're working on. But yeah, it's exciting. Obviously, we are a big contributors to OCP. I worked very closely with OCP on the storage spec for many years doing EDSFF SSDs and the OCP and the OCPMESD spec. And I was a contributor to the sustainability project and I wrote a couple

white papers on. Yeah. So it's natural for you to get involved in this neocloud working group. I'd say probably a lot of NeoCloud founders or operators, they don't have the same background as you in most cases. So how do they get involved in this OCP work group? And what's the benefit to them if they do? My favorite thing about OCP is when they say open, it really is open. Every single media is recorded and posted on YouTube. All the media notes are available in a shared Google Doc.

Everything gets posted. All the specs, when somebody contributes to spec, all the reference code and source has to be all open source and available. So that is the spirit of openness. Obviously, open source is beneficial because if it's something is going to be better by other people contributing and collaborating to it. So imagine like you're in Hedgehog's case, Mark would probably speaking for you here. But I imagine the reason why they like open source is customers can inspect the code and know that there's nothing bad there. There's other people that can contribute to the code base that they want custom features. There's no vendor lock in.

And so they don't feel like, hey, if Hedgehog just decides to walk away right now, our giant switches in our entire AI infra is not going to become a brick. That's really important. Like for a customer like me, I'm like having no vendor lock in, I don't want to, if I stop paying my support contract for my router, are they going to delete all the features?

That wouldn't be very cool. Like I wouldn't, and just brick all my stuff. And there's lots of ways to make money in providing open source software by being the guy to help install and tune and write custom features for customers. Yeah. Great. All right.

Well, Jam, thank you so much. I love talking to you every time. And I'm super happy to have met you, to have partnered with you on this journey, and just really looking forward to your hyperscale path going forward, which for sure is happening. Likewise, sir. It's been a big fun ride here. Great. All right, ma'am. We'll take care. We'll see you soon. Thanks for a lot, Mark.