On The Brink with Castle Island - Brian Venturo and Brannin McBee (CoreWeave) on Building an Industrial Scale GPU Cloud (EP.366)

Episode Date: October 31, 2022

CoreWeave's Brian Venturo (CTO) and Brannin McBee (CSO) return to On The Brink for a discussion of their journey through the merge and beyond. In this episode:  Are they relieved that the merge finally happened? Will there be some form of non-Bitcoin PoW in the future? Could Ethereum conceivably return to PoW? The effect of PoS on ETH's censor resistance What quantity of ETH miners shifted to other PoW chains and what portion went elsewhere? What is the best use of capital for former ETH miners now? Did the merge really reduce global electricity consumption by 0.2%? Brannin's estimate for precise electricity reduction from the merge How feasible is it for former ETH miners to get into GPU clouds? How Core Weave built their cloud product What it takes to be competitive in the high performance computing sector Why the generalized providers of cloud can't just win when it comes to rendering and ML/AI Why the HPC market is exploding right now How open source communities behind text and image models contributed to the explosion in use cases The importance of Stable Diffusion versus Open AI The explosive growth of Stable Diffusion and Stability AI How the growth in infrastructure contributed to the emergence of these newer image models New directions in AI models like video creation Will generative AI be attacked by environmentalists? Can cloud data centers handle interruptible loads like Bitcoin miners can? How industrial compute will play a role in emerging metaverses Use cases Brannin and Brian are excited about Why hardware is actually a constraint to the number of metaverse users today Learn more about CoreWeave here.

Hello and welcome back to On the Brink. This is Nick Carter. This is Mining miniseries adjacent. Today we're sitting down with Brian Venturo and Brandon McBee of Corweave. They've both been on the show individually in the past. Now they're together. A lot of people wanted to know what Ethereum miners thought of the merge, what that was like, what that transition was like, how they coped with that. Corweave used to mine Ethereum proof of work. Now they've transitioned onto other kinds of workloads in the exploding space, in particular of AI and ML. other forms of rendering and high performance computing. It's a fascinating sector that they have found themselves in. So without further ado, here is the episode. Very special edition of the show today,

Brian Venturo and Brandon McBee of CoreWeave. You've each done the show in the past, and now for the first time we have you together. So welcome. Excited to be here today, Nick. Thanks for bringing us back. Thanks for having us back. You know, Brandon, did you listen to mine? I did not listen to yours. Sounds about right. Brandon had two episodes. Brandon had two?

I had two. Come on. I didn't know about either of them, to be honest. Yeah. So Brandon did one on the Texas grid. And then you've each done one on sort of like your transition from ETH mining or mining crypto to the climate. which is the topic of today's conversation.

And it's funny because each of those conversations were one year apart. So starting two years ago. And the transition is finally here. We've transitioned. You know, the merge occurred. It's over. Yeah. So your long troubled journey with Ethereum, you know, real love-hate relationship has come to an end.

Do you feel relieved? Well, I think that we're all relieved that we never have to talk about proof of stake to an investor again. Yeah. I think that there were a lot of folks that missed the point that crypto was providing us the leverage to build a massive compute resource base, kind of on a permissionless basis and how valuable that was to us. But are we sad to see you go? Yeah, we're sad to see you go. Yeah. And I mean, I think it comes back in the future as well, and we can get into that.

But in some form, there is going to be infrastructure demand on the blockchain, but it might not just be a bunch of people running around putting GPUs on old clothing racks in the middle of warehouses. I think it'll look something a lot more like what we operate today, which is infrastructure and Tier 4 data center environments. You know, when you have an argument with your girlfriend or spouse and she storms out, and you monitor yourself like, oh, she'll be back. I feel like that is the case with Ethereum. Am I crazy to believe that?

I think Ethereum is going to go back to proof of work. I think you're a little crazy to believe that the core developers on Ethereum proper would ever go back. They'll be back. They'll be back. That ship has sailed. That's not to say that there's not going to be something else that looks similar to it. that's not running proof of work. Right, but I mean that that really has to see the market come to the conclusion that maybe

proof of stake in a super centralized fashion is not the best consensus mechanism, but I think that's to be determined for sure. I mean, the early signs are not looking promising for proof of stake. Let me tell you that. A significant percentage of blocks are already being mined by OFAC compliant entities or assembled.

So already, you know, already there's effects. Before you move to proof of stake, it was a new entrance activity. You could have a GPU in your computer

and all of a sudden you're now part of the Ethereum network. And now you have to go out and buy Ethereum to be able to stake and you have to either join a validator pool or have enough money to be your own. Like you said, it's pretty centralized, but this is what they wanted. I guess we'll see if it was good or bad over time.

Yeah, that's right. I mean, it was a deliberate tradeoff, right? You win some acceptability with the Davos crowd, you know, you reduce the energy consumption. The cost is the big platforms, which custody all the coins and will do much of the staking, they are now empowered. I think that's not a controversial thing to say. and one early data point I saw was apparently 25% of blocks post-merger now being built with MEV boost by flashbots, which is OFAT compliant, and then a vast majority are being relayed by

flashbots, so not necessarily being built with MAB boost, but there are a small number of entities that now have a lot of influence over block construction. Yeah, not to mention, you know, tornado cash being censored as well. of those entities. I think what you'll quickly see is quite an amount of centralization occur within that network. Some types of transactions will be censored and what was otherwise a decentralized sensorless network previously. And it must, proof of stake must have been designed with that in mind that that was going to occur. Because there's no other, you know, solution that comes out of this. right? Like, of course, it becomes more centralized. And it must have been built with that in mind.

Maybe that makes it more enterprise adoptable, where more large entities can participate in it. We'll see where it goes. I would be surprised if, you know, they just made a decision five, six years ago, whenever it was and stuck to it and didn't take any new data points into account. But we'll see. You know, the crazy thing is that you've gone from an OFAC compliant banking system that was centralized to a now an OFAC compliant decentralized banking system that also seems to be centralized among Coinbase and other validator pool providers, right? Like that doesn't really seem very different to me. Yeah. And the validators have these enormous returns to scale. They have regulatory barriers to entry. that's you know I think the US government various regulators have an incentive to just deal with a small number of institutions and so it's difficult today to start a competitor to Coinbase right so given that there's not a lot of

churn in the set of big validators yeah but you know it can be done right and you know just because somebody's big doesn't mean that you can't build a competitor I mean we we did it to compete with AWS GCP and Azure. And to going back to that idea of proof of work mining, we had something that helped us do that, right? And the idea is that the network really needs to empower people to be able to find entry to running those validators. And Ethereum in particular right now is like the barrier to entry is so large

that unless people build their own validator pools or are able to shepherd capital, it's unlikely to happen, right? So post-merge, some people pointed out that Ethereum Classic got a bit of uptake in terms of the mining. What are the prospects for the remainder of proof of work for the GPU mineable chains? What else is there? It doesn't look like there's a lot. You know, we saw that there was Ethereum Classic got probably 16% of the hash rate. Ethereum Proof-Work

got about 4.8%. Raven coin was another one around 4% or so. But, you know, overall, about 75% of the network doesn't really have a home anymore

for the way it was used previously in proof of work efforts. And that's what? We're two weeks after the fact now. Yeah. So it does is settled. Like two days after that, though, he probably had 65% of the network still running trying to

trying to pile into these other chains Ethereum Classic had like 350 tarahash at one point which was all the ASICs and everybody that couldn't figure out how to mine or liquid at other coins going there and the same thing on the smaller coins like Raven and ergo the race to those chains was crazy and then in the past two weeks the dust has started to settle at least

where people have gotten kind of over their delusings conclusions of grandeur of what GPU mining was going to be after Eath. So, you know, we hear on a daily basis like, hey, X, Y, Z is trying to sell like 10,03090s or 20,03080s. And it seems like it's a bloodbath out there on the consumer part side. And that should continue to accelerate as well, because people are battling, you know, really high electricity costs also. So, you know, guys running through the end of the month makes sense. maybe even through the end of the year, depending on sort of power purchase agreements that they have in place, especially for the commercial guys. But, you know, any of the at-home participants in the network will likely have moved on from here.

Like that community, that huge community of at-home miners is now gone from the Ethereum network. You know, some of them, I'm sure, moved on to staking mechanisms. As I said earlier, I think there'll be opportunities and other networks in the future. I think, like, ALEO will be appropriate. pretty fantastic network and we'll draw a lot of people to it. But today, there just aren't many profitable options. If anything, these things are pretty much losing money at this point, just considering how high global energy prices have gone in the past six to 12 months.

Yeah, you know, after Ethereum mining here is done, for people that want to continue participating in crypto, that own GPUs, The economically efficient thing to do now is for them to try to liquidate those GPUs and buy the crypto outright. Right. And that's like for what the returns are today. Like that's the by far the, uh, the best use of capital, um, which is putting that pressure on the GPU market, which, um, you know, when there's eight million of these things coming to market at the same time or whatever that number is, it's in that million's magnitude. I don't think that there's going to be demand there to really say, like the satiates. supply. Have you heard from the gamers? Have you received thank you notes or anything? Like,

thanks for, you know, releasing your grip on the GPU market? Oh, we, uh, so we haven't bought consumer or mining GPUs since 2019. That was the last time that we bought a non-data center card. Right. So we were definitely not part of that problem. And, you know, we were combating with miners that moved into some of the data center cards to try to mine with that caused us some supply chain issues, right? But, you know, the principal reason for us buying cards was never mining over the last three years. It was always how do we use this to serve real clients? And then when they're not using them, we can mine with them as well. So, thankfully, I don't think we have any misplaced hate towards us from the gamers. But I think there are some people that were really angry out there

for a really long time. That didn't get talked about a lot. I mean, the flip side of the reduction and the energy consumption is potentially a lot of e-waste if those GPs are just not economically useful anymore. Right, right.

I mean, the energy consumption part of it is interesting. There's definitely a few numbers floating around in the market about various estimates. Yeah, so I think it was Justin Drake that provided this estimate of 20 basis points of global electricity consumption reduction due to the merge. And then Vitalik actually rebroadcast it.

And I have issues with this number. So I looked into it and it came from a certain Dutch central bank employee name of Alex DeVries, who if you're not familiar with his work, he is basically a climate activist masquerading as a researcher on the topic of crypto he literally works with the Dutch central bank it's a conflict of interest as far as I'm concerned and he produces these numbers on Bitcoin and I guess formerly Ethereum energy consumption they're always way too high so yeah that's

where this number derived from I'm told that you guys have an alternative estimate for the energy consumption of ETH pre-merge so right Yeah, I mean, our backgrounds are in the energy space at the end of the day. Like, this is really simple math to come up with estimates, right? Like you, the International Energy Agency publishes global energy consumption values. Their latest number is 2019 for estimated number of terawatt hours that were consumed in the electricity space. And I think that makes the most sense, just giving COVID everything long. those lines, right?

Yeah. So from there, you just need to come up with an estimate of the amount of electricity consumed by the Ethereum network. And in the most conservative way possible, you use old generation GPUs, like an Nvidia 1070. You assume that those things consumed about 70 watts at 30 megahash. So that's about 2.3 watts per megahash. extrapolate that across the entire network hash rate and you come to about 2.3 megawatts of demand, which would...

Gigawatts, no? Yeah, sorry. Ah, damn it. Yeah, it is gigawatts. Yep. I'll edit that out. Yeah. Please don't come to that part.

I want that part in there on the record. Comes out to 2,300 megawatts of demand, which is about 20.4 terawatt hours. annually. And when you relate that to that IEA estimate of 2019 electricity demand, that came in at right around 23,000 terawatt hours run the percentages and that's 0.08%. Yeah, so just under

just over a third of the estimate being promoted by the Ethereum community. for context if eth was two-ish gigawatts Bitcoin is around 10 gigawatts. Yeah, I think that's I think that's about right. Yeah. And you keep in context as well that like these GPUs have been repurposed other networks as well. So at the end of the day, you assume, you know, 20% of them are going to sit on Ethereum classic Ravencoin, ergo, a couple of these other networks. And then all of a sudden, your actual electricity reduction.

from moving to proof of stake is materially less than those numbers being kind of pushed around in market. Yeah, so you might be looking at six basis points of electricity reduction. Yeah, I think that's more accurate. Yeah, and also to support that, Kyle McDonald has an estimate of Ethereum's electricity consumption, which is virtually the same. So, you know, I think that is a much more credible estimate than the DeVries method, which is just, it's insane that he gets cited because he really is deeply conflicted. And so then the other question is those GPUs, what share of them do you think were repurposed in sort of like the core weave model? Zero.

Aside from core weave. I think that the expectation, and we've seen a lot of pitchbooks out there. Page 76, Brian, to be specific of those pitch books. Of people that were going to repurpose their GP mining operations in the clouds. And I think there are probably a couple small guys that I don't know about that have done it, or maybe you're in the process of doing it, but it's not like you're flipping a switch. There are significant technical requirements to be able to support. any type of client. You know, this idea of clients liking or being okay with interruptible workloads

like it's a great idea in theory until you actually have a client who gets interrupted. And even though they said they were okay with it, they're very much not okay with it. And the amount of infrastructure that we've built over the past three years to offer the quality and scale of services that we do are astronomically larger and more complex than I expected to be to be. So I think that this pie in the sky idea of like, hey, we can run distributed compute in the blockchain and be able to provide rendering or AIML tasks. Not like not even taking into account like the technical problems with that. I just don't think that it's really feasible to serve any type of real workload.

So you know, and it's it's terrible to see because like Brandon said, a lot of that stuff may become e-waste. A lot of those GPs are running really bad environments, and they're likely not going to be winding up in gamer rigs because the gamers have become pretty adept to the idea that they're not so great for them, even when adjusting for price. So I think that my answer of zero may not be correct, but it's definitely not a single-digit percentage point.

And just to wrap it up on the blockchain section, what interesting use cases do you think there might be for, like, infrastructure providers like yourselves in future blockchains? Like, will there be another file coin? Or, you know, are ZK roll-ups going to require, like, GP, like, heavy workloads? Like, will the blockchain come back around and be a client of CoreWeave once again? I hope so Brandon's the one who's really been running down these opportunities for us and keeping relationships in the space Yeah and I think a lot of those companies are making a ton of progress

and I believe that those new layer ones will be the entities who bring enterprise adoption onto the blockchain because they are designing their blockchains to be friendly, compliant, and scalable for enterprise adoption. And at the end of the day, they're going to need infrastructure to run those blockchains. Again, I think someone like an ALEO is a really good example of a layer one that will launch by Q1 of next year. That could be a pretty material infrastructure consumer, even just through their zero-knowledge-proof processing component.

But what exists today, it's really challenging to point to substantial demand. Yeah. And so the era of consumer GPU mining for proof of work, that is pretty much behind us. Yeah, you know, I don't know if it's behind us, but it's definitely behind us and what we've known it as, right? Where you had eGPUs tied to a milk carton with a two-core CPU that you bought for 40 bucks on eBay with four gigabytes of system RAM, right? Like those days are over, right? And what you may find is for networks like Filecoin, the systems that they need for the various tasks there are very different than what you see in an old GPU mining operation right i think as you get these specialty use cases um there may be blockchain compute demand for it right but i just don't think it's going to be coming from what we would consider today to be

cloud-like workloads right it's going to have to be something that's newly invented um like alio right um where they're bringing that computer to solve a problem. So the good news is, though, that you guys clearly had line aside on this and used that, used the existence of Ethereum and proof work mining to build out your fleet. And as we talked about in the last two episodes, anticipated this and totally diversified your revenue streams ahead of the merge. Yeah. Brandon gets upset about this all the time,

because we look back in how much Ethereum we mined and what that was on a U.S. dollar basis. And the fact that we're both still working and building a business, we feel like idiots. But we always had eyes on building something that was bigger and that wasn't just crypto-related. So we're going to use that one as the reason why we did it and why we're not both retired. Well, it's always good to build sort of like real products and services as opposed to just doing crypto stuff. Yeah, we're really excited about where CoreWeave is positioned in market today. And that's as a specialized cloud focused on GPU accelerated workloads. And we're supporting some of the highest growth sectors that exist between media and entertainment, artificial intelligence, machine learning, computational chemistry.

And hopefully what will be blockchain again in the future, not to mention more nascent sectors like the Metaverse or pixel streaming technology. technologies. And the end of the day, they all use GPU compute. They all need highly specialized infrastructure orchestration and networking components that are not really available at the generalized clouds, or certainly not to the extent that you would expect. And increasingly, this kind of compute is looking like one of the scarcest resources on the planet. Yeah, so I definitely want to drill down into this. So, I mean, first of all, like, what does it take to compete in this market for, I guess, what would you call it, HPC or industrial cloud? Like, what would be the right nomenclature?

I think it's, HBC is fine. I've been kind of brandy for using that term all the time, though. But it's accelerated compute, right? It's anything GPU accelerated. You know, we don't do anything beyond that. We don't have any specialized hardware devices that we build. We don't have any FPGAs. for things like transcoding services.

We're really around the flexibility of the GPU and the different workloads that a single GPU can go serve. So the GPU is still the sort of hardware unit that you're focused on? It is, right? And it's very infrequent for us to go after clients that are CPU only.

And while we can serve them just as well, if not better than other providers, it's just not what we've built our business around, right? And, you know, we're able to differentiate ourselves on GPU workloads because of the way that we built our infrastructure, because of our expertise running infrastructure at scale, right? The health checking that we have on GPU systems and the ability to understand what's going to cause problems

or when problems are happening, how to solve those problems. It's something that even the largest clouds today are behind on, right? And, you know, some of that is from growing up running 30,000 consumer GPUs in a warehouse, right, and having to understand exactly what's happening at all of them and how to recover from errors. But some of that's also in the way that we built our platform, right? We were able to make architectural decisions that were really based around this idea of like bursty use cases. Yeah, so tell us a bit more about that because it's clearly very different from just plugging in GPUs and running proof like Ethereum.

What does it take to handle these bursty loads? It's, so the infrastructure that we built is incredibly complex, right? And it's not just networking, it's not just storage. You know, we've built super high throughput, non-blocking networks to be able to handle the intra-datacenter bandwidth that we have. And the majority of our bandwidth is internal to our data center environments, right? Customers come here to run crazy workloads. they don't really come here to serve their websites. So, you know, the networking piece is something we've invested heavily in.

It's, that's on the kind of the physical networking side. And then we've had to build VPC networking solutions on top of that. So customers can have virtual private cloud networks that are effectively a layer two network that they can connect back to their on-prem environments or run their own virtual firewalls, etc. From there, you get into like the network attached storage piece. right, which is how do you provide access to thousands and thousands of workers with a single volume, right? How do you attach that volume to all those workers? How do you get performant metadata throughput from those volumes to be able to serve all those instances, reading tons of small files?

You know, these are not trivial tasks, right? And, you know, there's a reason why there's only like five companies in the planet that do this stuff, right? is because it's hard, right? And the technology investments you have to make are not the minimis. And it really goes back to the idea I said before that I just think there's very few people or entities in the planet that really know how to build the cloud. Lots of folks know how to work on them, but very, very few know how to build them. Yeah.

So when you, I guess this is an annoying question, but, you know, why is it that the large established providers of cloud compute don't just, win at this, given their advantages in terms of scale and then obviously just a virtually unlimited ability to spend. I'll offer the very simple solution and then Brian will do more detail, but like on an extremely high level, it's because those clouds, AWSGCP and Azure are generalized. They're trying to create infrastructure for every single use case to be able to support major companies like the forwards of the world to come onto their infrastructure. structure and move their entire computational needs from data storage and just simple workstations

into the cloud. And they do a great job of that. And that is absolutely not something that we're trying to compete with. What we identified in market was a lack of specialization. And Brian does a really good job of explaining why we excel there. Yeah. So other clouds today, they're built around like hypervisor workloads. So you go to AWS and you spin up an EC2 instance, they're going to instantiate a hypervisor and what are their bare metal nodes?

That's then going to load an operating system. That's then going to require you to set up your environment or to have a prepackaged image. The time to be able to get an EC2 instance spun up could be up to minutes, like two to three minutes. And when you're trying to respond to demand in real time, So for things like text image generation, right, if your user-based spikes and you need to serve more inference API requests, you then have to be able to go out and intelligently manage that resource base that you're serving from. So if you say, okay, I've got 10 more requests just came in.

I don't have enough resources to serve those requests. They're going to sit in a queue until I have those resources available. Like it could be two to three minutes before those requests get served. the user loses engagement or their request times out and all of a sudden you've lost them and they have a bad user experience. So what we've built here is, you know, our cloud is Kubernetes native. And Kubernetes is a container orchestration platform. And we run Kubernetes in a multi-tenant environment where all of the nodes and all the compute resources are already available for our clients to consume. And so when they schedule a workload or their their inference service auto-scaler,

says, hey, I need another replica to run, we can have that up and running within four seconds serving the requests. It's just a considerably shorter time to serve running on infrastructure like CoreWeave. And, you know, AWS, JCP, and Azure just aren't built that way today. And for folks that don't have those big engineering organizations to stand up that type of intelligent orchestration platform, right, they get all that stuff out of the box from us. The HPC market, I know there's all these estimates from Gartner about markets increasing X percent, but I mean, it subjectively seems to me it's exploding right now. Is that the correct characterization? Yes. So we are running as fast as we possibly can, right? And, you know, it feels like we're falling behind every day. You know, the onboarding cycle for clients with us, it's kind of use case dependent, right? On the VFX side is longer than it is for the, AIML side. But we can onboard a client on the AIML side today and they're generating significant revenue tomorrow. Or they're generating significant cost savings to Brian's point on the infrastructure

orchestration and real-time response rates as well. Because at the end of the day, those clients to surmount the issues with spin-up times at the generalized clouds, they just have to sit within reserved instances instead. So you might only be serving workloads 10 seconds out of 60 seconds for someone hitting an inference solution. The space is growing kind of at an unbounded rate. And it seems like for the past two years there was a lot of research done and building of open source communities, largely around the Aluthor AI group, which where we've contributed compute for to train their 20B parameter model. GPD NeoX, you know, coming out of that exercise, those researchers and community members are starting to build and deliver commercial products for the first time.

It was like there was this couple years of crazy research and people were doing things as a hobby. And then people started to find products that might actually work. And now we're in this stage where everybody is recognizing like, oh shit, people are using this. We're seeing real world use cases where this actually provides economic value, and everybody and their brother is rushing out there to build on top of it. Or build using these resources. You know, we saw an explosion in large language model startups to start 2022. That's folks like cohere or Anthropic or AI21.

And they're building proprietary or principal models. that are going to serve products for their clients, right? And whether they have the business model that they're building products directly or they're serving APIs to their clients, everybody's super, super interested in the space because of the progress that's been made. And with that and the scaling of those models

and the performance of those models, the compute needs have gone up exponentially. And that's just in the large language model side, which, you know, as they're building products for that, That has implications for things like writing assessment or writing help or crafting blog posts or text adventure games. And where we're seeing like this crazy spike in the past two months is really around the open sourcing of text image generation models. And this really started after OpenAI launched Dolly 2, which was a closed source model and you had to request access to. You know, when closed source models are released, I think it's kind of a, everybody looks at it and goes, wow, that'd be really cool.

I'd like to have that until somebody goes out and recreates it. And that's really where all the creativity comes from is when it's been recreated and open source and people can build on top of it. And, you know, I know that you've had a ton of fun with Stable Diffusion. But the amount of demand generator around the release of stable diffusion instability AI's product Dream Studio has, been insane. Yeah. Or we've just been lucky enough to be working with stability AI and that launch. But the quality of the images that this model can create is, for me, is like eye opening.

It's like, wow, this is the first time I've seen AI do something that I'm like, okay, this actually has the ability to change the creative process. Yeah, I've enjoyed playing with both the language models and more recently with the image models. I mean, I was absolutely blown away by what was achievable with stable diffusion in particular. And I think I'm like the number one client for a trim studio. I can promise you you're very much not. Yeah, we've generated on the order of hundreds of millions of images since launch. And I think it's worth keeping in context.

I mean, this is four or five weeks old. Yeah. I mean, this is, this is brand new technology that I think EMOD and the Stability AI team had the foresight to release his open source to bring the technology to everyone and immediately democratize it. I mean, it's just. And the crazy thing about democratizing this stuff is like, you know, why would somebody open source this? And the open source community is like a crazy positive feedback group. Right. And open sourcing technology like this is going to lead to like an unmeasurable amount of impact on businesses and developers and engineers down the road.

You know, when a company is starting up, one of the things that they have to make the assessment of is like, do I like go find something that's off the shelf that already does this or do I build my own thing? And, you know, for a small engineering team building a new product, if they decide to build their own thing instead of using an open source solution, like they're likely. not going to be competitive unless they have something super differentiated. So for people that can leverage open source tools, right, having something like stable diffusion available to them allows them to extend it and to do so much more, right, and add value around it, like, they're light years ahead of somebody that would have to design that from scratch. And this goes back to like, like, why is this all exploding right now?

And, you know, one of the things that Nvidia did incredibly well over the past like six or seven years is they invested so heavily in their Kuta developer ecosystem, right? And they realized like, okay, like, we have this hardware now, but for anybody to actually use it, like, we have to go out and like build all the tools for it. So they can build with it. They can port other things that they have over to it. And like, they got that they really had to make it easy for everybody to use. And now that all those tools exist and all this prior open source work has been done on top of it, right? Like, they're, they have a huge moat around their product because everybody's going to default to them because they can save hundreds,

if not thousands of engineering hours work. And it's really easy to scale compute. It's really hard to scale engineering teams. The thing that shocks me is you guys remember a couple of years ago, those, there's a very simple model, well, not simple, but it did one thing, which was just produce images of faces that were, and that was that, like, typically. with like was it the GAN models GAN? Yeah, yeah, it was GAN models. So it was, you know, pretty low resolution images of human faces and that's the one thing it did.

And that was actually kind of shocking to people because it's like, oh, no, the person I'm interacting with on social media may not be real human and have no way of knowing that and we're going to be inundated in bots, et cetera. And then I feel like that was maybe 18 months ago or two years ago. Now fast forward to today with stable diffusion, you can basically generate any arbitrary image at all that you want at an incredible level of resolution. And of course, you have to get the prompt right, but it can do human faces at a completely photorealistic level, like portrait photography style. I mean, what explains that increase in complexity? just so much more complex and sophisticated in such a short period of time.

I mean, for my seat, I think it's access to scale compute infrastructure to be able to do that. We consistently get demand of thousands of latest generation AIML chipsets, and that looks pretty different than what you observed even a year ago. And I think a component of that is going back to my point earlier, these chipsets, specifically the A100 series, what will be the H100 series as it's released next year, they're the scarcest commodity resource that exists on the planet. Hard stop.

It is sold out across every single cloud service provider that there is, and it's sold out in two-year-plus contracts. Wow. It's used for, training these next generation models. And I think what comes with the H-100 series will unlock an even more complex set of models to be built. Ones that we haven't even contemplated today, to your point. I mean, the rate of acceleration of technology in this space is driven linearly by access to compute infrastructure.

And that's what we've positioned ourselves as a provider of strictly this infrastructure. infrastructure within the market. Yeah. So, you know, what's driven is large change in quality, right? I think that the, the Dali 2 launch from open AI, like showed everybody was possible. And when people saw that, right, I mean, the ConfViz department at University of Munich, right, they came out with a latent diffusion model that stability AI then extended and improved. Right.

But like this idea of like these diffusion models and it's so cool what it does, where it basically takes a full image and and adds noise every step of the way and then trains the model starting from a pure noise image back to what the original image was. And using captions from that as the descriptors, right? They're able to create these images from pure noise, which, like, that's pretty mind-blowing that they were able to take these millions of images in this lay-on dataset and to train it to be so accurate. you know, while folks are playing around with this stuff, and I'm sure that people are going to try it after this call, and I'm just going to plug Stability's product here at Dream Studio.AI, you very quickly realize that it's not like,

hey, I want to see a cow riding a bus driving down the street. And while you may get some great initial kind of images from that, there's some art to this idea of prompt generation. right and that speaks a little bit to like how early this stuff is is that folks haven't been able to build the models to be really good at accepting simple prompts yeah they're incredibly powerful but folks have to try to folks really have to learn how to use them but you know the big step there was like okay we're going from some of the the vQ gang and clip models that existed before this where it was like yeah that kind of looks like what i thought it was going to be to like stable diffusion words like oh my god like i can't tell the difference between not and the real image yeah i i will say from experience the prompt generation is incredibly challenging um and the model does have constraints i mean it's by no means perfect like if you introduce more than sort of two characters into the image it'll get confused and it's not that

good at human body parts because it's very easy to create something that breaks the illusion and it's not that good yet yeah right and I think that there's probably going to be some features that are coming out from stability AI in the next couple weeks that will improve that quality in the things you talked about there yeah you know I think that what what was released here was like okay this is the first step and we see projects and companies all over the place building on top of that, whether that's unlimited amounts of upscaling or it's people that are doing text to animation generations. I think that you saw Armand Van Buren did an entire music video. It was go for the song computers take over the world. All of it was

generated via stable diffusion animation. Yeah, and for listeners who actually want to give it a shot, it's quite easy, I would say, actually, even to use the model really efficiently. So my recommendation would be to use Dream Studio, get the API, and use Christian Cantrell's plugin to Photoshop to use it. And at that point, it becomes very easy to, you know, to generate an image and then mask it or generate on different layers. To do image to image, which is astounding to me personally, is to do your own, like, kind of crappy illustrations and then attach a prompt to it and then have the model upscale it. It also supports impainting now.

So basically you mask a certain part of the layer and then have the model fill it in with context and with a prompt. I mean, the functionality for like digital artists, just in terms of time savings and efficiency is just unbelievable. And I've been having so much fun with it even as not an artist. So, yeah, I mean, I can't even imagine what the next version is going to look like. I mean, the culmination of a few technology stacks here are going to get interesting pretty quickly, right? You have natural language processing models that allow user to offer a short prompt, and then for the model to fill in paragraphs worth or more of context around it.

You have text-to-voice models that can synthesize any kind of voice that you'd like to be represented, to represent that text, and combine those two software stacks with text-to-image as well, and what will ultimately be text-to-animation, all of a sudden you have this capability to produce unique content from a simple prompt that you've never been able to do before. Like unique video content full with voice animation, all from a simple text prompt. And the applications of it are just going to be wildly expansive. We're particularly excited about it because of the infrastructure implications around it. because as it exists today, there isn't enough infrastructure on the planet to serve this.

And I wish we could say that you can go, you know, cobble together a bunch of retail-grade GPUs to serve it. But at the end of the day, this needs to be enterprise-grade infrastructure within data center environments, using some of the modern technologies that we've developed at CoreWeave to orchestrate that infrastructure. So here's a provocative question. I was just reading this White House report on like crypto mining. And it was pretty tough, I would say. They don't like it. And their big worry is that it's going to consume, you know, a large percentage of the U.S. grid energy, whether or not that's true.

You know, that seems to be the concern is that, okay, well, this is a sector that's new, it's growing. and it's competing with households and an industry for energy. Now, and there's the laden with the assumption that it's sort of not a valid, you know, use of those scarce resources. Now, do you guys expect anything similar to emerge in terms of pushback from the sort of AI space? Like, okay, this industry is growing super fast. It's scooping up a ton of energy. these data centers are appearing everywhere.

Is that something that you're concerned about? Is the sort of like the crypto mining critique getting transposed over to sort of like the AIML space? I don't think so. And a lot of the people that have issues with crypto mining is they look at it as like it's an endless pursuit, right, or a pointless pursuit. And they don't necessarily understand like the security or the trust implications of what that proof of work mining offers. I think it would be pretty hard for somebody to make a case that became very popular that these AI models and the pursuit of AI isn't providing some type of creative benefit.

And, you know, there is going to be other societal implications that have to be discussed around that. But I don't think that the compute to serve that stuff is going to be considered or will be treated adversarially. and I'm sure there's always going to be some small minority that does treat it that way, but I don't have any concern that us serving infrastructure for AI or ML services is going to be considered that way. Yeah, I think the scale kind of really matters here as well. Going back to those numbers we used earlier on the Ethereum network, that's an implied amount of over 30 million GPUs. That's an implied amount of about 2,300 megawatts or 2.3 gigawatts.

or 2.3 gigawatts of installed capacity. We're probably the fourth or fifth largest operator of GPU accelerated compute in North America, and we're about 15 megawatts in total. Like, it's night and day difference in terms of scale, right? And then put this in comparison to Bitcoin mining, right? It's just an utter fraction, and especially relative to what's being,

produced as well. Yeah, 15 megawatts is like one small Bitcoin mine. Like you routinely have individual Bitcoin mines that are over 100 megawatts. Yeah. And you know, just to go further on that, like that the number that brand on reference there is, is like the capacity of our three regions right now. That's not actually what our IT load is in those environments. Like that's what we're built to consume there. Right. And while we have systems in those facilities to consume that amount of power, we're very much not consuming 15 megawatts around the clock. Right. It's a fraction. Right. Because unlike crypto mining, it's not an always on load. You're performing the jobs as you got them. Yeah. Yeah. And different client workloads have

different power profiles. And we do a lot of portfolioing of workloads to really understand, a lot of portfolio to understand, like, how power is consumed across our fleet. We do some intelligent balancing to make sure that we're keeping workload spread pretty evenly so we don't overload any areas. You know, there's a lot of analytics that happens there just in our scheduling layer to make sure that, you know, we're keeping that stuff balanced and understanding that stuff's going to fluctuate pretty dramatically. Yeah, so on the same topic, actually, you know, one thing I will give Bitcoin minus credit for is, you know, they're definitely more interruptible and so they can participate in these various like demand response programs and then

they're more location agnostic than you know most industrial uh loads so they can you know find relatively cheap or underutilized energy and plonk themselves down there now on the hPC side i feel like it's probably less imminently interruptible because you if you have a big load come through, you maybe don't want to delay that. And it's less location agnostic because latency really matters. I feel like you want to be closer to major, you know, the major metropolitan areas where maybe these queries are coming from. Do you think that HPC could lend itself to being a location agnostic and be interruptible? Does that comport with sort of the model? Yeah. In A, canopy location agnostic parts of it can right and the parts that can is really on the batch

training or batch inference side right where there's no big component to it um that's really where you're going to go for um that that's going to be on a cost management basis right is like where do i go find the cheapest real estate cheapest power um the best environment so that we don't have to run our chillers all the time we can we can run evaporative cooling and be kind of economizers in those plants um like that's that that's the driver there like you'd always like to be as close close to eyeballs as possible, even if you don't need the low latency. For workloads that are actually serving a client or an end user, and this is specific or most specific to things like virtual desktops in the VFX space for artists workstations where

they're doing high fidelity graphics or they're doing episodic TV work inside of a virtual workstation and they need to be able to run 4K playback. or like a pixel streaming event to serve Metaverse experiences where somebody's in this real world experience in their browser and they're doing it off of the compute in our cloud. That's where the latency really matters. Do I think that this stuff can be run on an interruptible basis? I don't, right?

And I'll talk about that separately too. On the batch training and batch inference side, there are real costs to interrupting training workloads. and you know while people run checkpoints and stuff to make sure that if things get interrupted they don't lose all their work these are complex sensitive systems that it's not that easy to go from 100% power to turning everything off because you have to interrupt your entire load but turning it back on again like it's just not at easy and anybody that it is hasn't done it i guess you could schedule it you know around what the the grid flexibility might demand yeah but But the real value in that interruptability is like is the panic scenarios when the grid really needs it. And it's like, hey, I need this in 10 minutes. Yeah.

And like those scenarios, they just don't work for that type of work. Right. And you're talking about potentially losing like hundreds of thousands of dollars of compute time because all of a sudden your provider needs to go save $10 a megawatt. Right. Like I just don't think that people want that. And I think it's really important to understand that, you know, for these EIML, work. workloads especially, power prices roughly 10% of the cost of operating that GPU from a fully amortized basis. It's a fraction of it. It's like the hardware cost is where the real cost of those

components come into play. So, you know, lowering your electricity price by a few cents a kilowatt hour is really not impactful at the end of the day and certainly not at the expense of the latency. or the interruptability that comes with that. So it's just very different than what you would see within the crypto space, I believe. And, you know, there's some folks that would make the argument that, like, oh, hey, if I'm doing this at a site with stranded power, it's more environmentally friendly, right? Like our HPC, or specifically our distributed training clusters are all built

in our Las Vegas data center right now. And that's 100% renewable power. Is that nuclear? So it's a combination. It's run by, we're a big client of a company called Switch, and they're an incredible partner of ours. They run awesome infrastructure. You know, I would just refer you to their website to find out what that mix is. But, you know, they have an incredible commitment to environmental stewardship, right?

And, you know, we've made it easy on the distributed training side to run all that stuff there. But, you know, when we get back to the real-time workloads, right, and the idea of interrupting, like, somebody's pixel streaming session or their virtual desktop in the middle of a workday, right? Like, we have hardware failures. Like, that happens to every cloud. And, like, when people get interrupted, like, they get pissed. Right. And my support staff, like, if we ever have issues or when we have issues, because everybody has issues, like, they hate those days. Right.

And even the clients that said, oh, yeah, like, I'll run spot or I'll run interruptible. Like, nobody actually means that. You guys have such a diverse mix of clients and use cases. I mean, I feel like you've mentioned over a dozen different types of things. People use your infrastructure for. What aside from the awesome growth in the image models, what's really exciting to you right now? So this idea of the metaverse is like really loosely defined. And to me, the metaverse is really just like the rendering and delivery of real-time experiences.

Right. And I don't know if that's going to be for industrial simulation where you're running digital twins or it's going to be like product configurators like you see on websites or if it's actually doing immersive experiences in the browser from afar. Right. You know, there are so many different ways that that can go. And I don't think any of us really know what the, the true direction of the metaverse is going to be. Right. But when we as an infrastructure provider, when we start thinking about like, okay, we have clients that need 1,000 concurrent users or 10,000 concurrent users or 100,000 concurrent users. And every single one of those users requires a discrete GPU to serve the experience, right? Like, that's where I'm like, okay, like, this is cool. How do we solve this problem? How do we scale this? solve this problem globally. Right. And, you know, back to Branden's point before, like, that infrastructure does not exist if you combine all of the clouds today.

100,000 concurrent. Let's say, if you get to the 500,000 concurrent user basis globally, can't do it. Yeah. Across all clients and everyone building in the Metaverse, like, that's a pretty small number. Yeah, that is the largest inhibitor of growth to that Metaverse space. It's just simply access. to scale infrastructure because you need one GPU per client. Maybe you can fit a couple of clients on a GPU using some of Vendidio's technology, but at the end of the day, like that's when

that is what will stop growth and that is why it's the scarcest compute resource on the planet. I watched tech demo for the other side Metaverse, which I'll give them credit. It actually looked very smooth. I think there were thousands of concurrent users just in for the tech demo. I hate to give the, you know, the ape crew credit for anything. But, you know, I think the other, you know, the sandbox and the central end, you never really get more than a couple hundred users on. Yeah. So, you know, we've spoken to the group that runs the infrastructure for the board ape guys, right?

So it's a company called Improbable. And, you know, their history and their product history is really in running game servers. I think that those guys are probably like top three experts on the planet at doing that stuff. Right. So my understanding was that that was largely CPU based, right? And they relied on a lot of the processing power on a local client side, right, to be able to handle that, which I think is a lot easier and that's more proven. Right. And, you know, where everybody gets freaked out about this stuff is like when you get into like, hey, this is a really high end experience, we have to do crazy graphics rendering in real time somewhere and we can't do it on a fun. We have to do it in the cloud or we have to do it. It can't be on the user's device. That's where it gets really dicey. And I think one of the big missed narratives today is like everybody's like throwing money at these Metaverse companies.

Right. And like with absolutely no understanding that like, hey, the physical infrastructure here is going to be the blocker. And unless you have that at scale, like no matter how much you invest in these smaller like platform developers or game developers, like if it's not there, they're going to fail. the growth of that space doesn't happen without exponential growth of the specialized clouds serving that space. That's so interesting. I hadn't thought about the infrastructure constraint at all. Yeah, and that's where I get super excited about that. I'm like, okay, like bring on the million users concurrently.

Like, we know that even to get there for one, for one event or one experience, like, this whole thing has to get a lot bigger. And then when you go from one million users and 10 million users, right, that's, work is really interesting. So for those listening and want to investigate Corrieve and learn more about the product suite, where would you direct them? So our website CoreWeave.com has a lot of pretty good resources about what we do and how we do it. One of the things that we do really aggressively is we make sure that our clients are getting what they need to be successful on CoreWeave. So we have DevOps and engineering teams that will dig in with clients to help them understand like how do I run this best, how do I architecturally think about these problems?

How do I solve them? Not just on core weave, but kind of in a grander scheme too. So, you know, I feel like as an engineer, people are never willing to click talk to sales. But after you sign up, reach out because there's a whole team of people here that are really excited to help with interesting use cases and scaling crazy infrastructure problems. So, you know, we're excited to hear always about new up-and-coming things,

no matter how crazy the ideas are. And we've got the people to support it. Awesome. Well, I think you guys are officially the number one most frequent guest on the brink. So thank you for that as well. Always a great conversation, Nick. Thanks for having us.