On The Brink with Castle Island - Chase Lochmiller (Crusoe) on Powering the AI Revolution (EP.506)

Episode Date: February 26, 2024

Crusoe CEO and Co-Founder Chase Lochmiller joins us for an episode sharing an update on improving the efficiency of power production with on-site digital infrastructure. In this episode: Solving the broken links between supply (flaring, renewables) and demand for energy (BTC, high performance compute) Engineering a portfolio of applications to pair with different sources of energy Aligning economic incentives with climate solutions New hardware coming online and the impact to AI inference and training Decentralized Airbnb for GPU networks – what works and what doesn't Halvening impact on BTC Miners Crusoe's climate impact

Welcome to On the Brink. My name is Sean Judge. Today on the podcast, I sat down with Chase Lockmiller, CEO and co-founder of Crusoe. Cruceau provides scalable, climate-aligned digital infrastructure optimized for high-performance computing and artificial intelligence. The company is a market-leading Bitcoin miner and has been early in servicing the AI revolution. Without further ado, here's my conversation with Chase Lockmiller. Matt Walsh and Nick Carter are partners at Castle Island Ventures. All of these expressed by them where the guests on this podcast are solely their opinions and do not reflect the opinions of Castle Island Ventures.

Guest and hosts may maintain positions in the assets discussed in this podcast. You should not treat any opinion expressed by anyone on this podcast as a specific inducement to make a particular investment or follow a particular strategy, but only as an expression of their personal opinion. This podcast is for informational purposes only. Brought down by bad mortgage investments, Lehman, which has 25,000 employees, will be liquidated.

The federal government loans American International Group, AIG, $85 billion. This is a different kind of market, and the Fed is a sling. The federal government is stepping it to stabilize Fannie Mae and Freddie Mac, the two mortgage giants that have been threatened by the housing crisis. The Bank of England has pumped 75 billion pounds more into Britain's ailing economy with a new round of quantitative easing. You print a couple trillion dollars and all of a sudden people start to worry. So out of this worry, we have something called a Bitcoin. Bitcoin.

Welcome to On the Brink. I'm Sean Judge. Today I'm joined by Chase Lockmiller, CEO and co-founder of Crusoe. Chase, welcome back to On the Brink. Happy to be back. Yeah, it's been a little while. Yeah, it has. And I think along that time, we have some new listeners. Maybe to start, it would be helpful for folks to just understand an overview of Crusoe and the mission and kind of history of the firm. Yeah, so Crusoe started about six years ago. Our core mission as a business is aligning the future of computing with

the future of the climate. So we build large-scale digital infrastructure and computing infrastructure to tackle the most energy-intensive computing problems. Chief among me, those is Bitcoin mining and digital currency mining. We've built a fairly large business, primarily powered off of waste methane that's produced from oil and gas production, where currently it's being wasted or burned off as an associated byproduct of oil production. We're able to sort of co-locate data centers on site with that waste gas production. Instead of it being flared or vented into the atmosphere, creating a big greenhouse gas emission footprint, we're able to actually capture it on site and put it to some beneficial

use to produce power and power that digital infrastructure. We've also sort of been very focused on energy-intensive computing applications outside of the crypto ecosystem as well, and have been very focused on building an accelerated computing cloud platform that's focused on AI and other high-performance computing workloads. So that's been a very exciting area of growth for the business over the last 24 months as things have evolved pretty rapidly in the AI landscape. Yeah, it certainly has. And you know, In the past, you've talked about these broken links between the supply of energy and the demand for it. I think you touched on a couple key use cases for the demand for this energy.

I mean, more mechanically, how do you capture the supply side of this? Who are you talking to from the energy producer side? I think part of our mission, this whole notion of aligning the future of computing with the future of the climate, we really try to go as far upstream as possible to the source of energy production. And when you look at sort of the demand profile for computing as an entire asset class and consumer of power, it's one of the fastest growing power consumers in the world. Today, global data centers consume somewhere around 2% of global power consumption that is rapidly going to be rising above 10% over the next decade with AI playing a major role in that

and things like digital currencies playing a big role as well. From that perspective, really our philosophy is that it's not going to happen on accident in terms of like how that infrastructure gets powered. If you just kind of leave things to play out as they may, could have resulted in sort of a very negative outcome from a mission standpoint in a global sustainability perspective. Our goal is to really be intentional and thoughtful in terms of how we partner with upstream energy producers to try to drive down both the environmental impact and the cost of this future digital infrastructure. So talking through some of the issues that we can help resolve as a computing provider. I think when you look at the energy landscape,

there's tremendous waste out there. Our first big business, which was focused on mitigating methane emissions from flaring, flaring is this global problem that persists in the oil field, where you have about 14 billion cubic feet a day of gas that is being burned off and completely wasted. Putting that in perspective, you could power about 65 gigawatts worth of power, which is, call it six times the total power footprint of the entire Bitcoin network. So it's a really, really meaningful amount of waste energy that's out there causing harm and not benefiting anyone in a positive capacity. And by sort of partnering directly with people that, you know, are engaged in flaring and wasting that gas, we can provide a value proposition in that we can bring a revenue stream to, you know, an otherwise

wasted asset, as well as drive down the overall emissions footprint and drive down the cost for computing infrastructure that has energy as its largest operating expense. Another big area of partnership for us is actually in the renewable energy space. When you look at the way renewable energy gets produced and consumed, the best places to build renewable energy assets are not necessarily in the same places where people are actually demanding and consuming power and not necessarily producing power at the right time. the solar farm produces power when the sun is shining, the wind farm produces power when wind is blowing. And where the sun is shining, where the wind is blowing most consistently

aren't necessarily in the exact spots where people are demanding power. There's an interesting market dynamic that's sort of played out here where renewable energy producers actually underwrite their projects against two different revenue streams. So there's the obvious one of power sales. They're in the business of producing power and selling that power. But maybe the less obvious one to most people is actually they underrate the projects against production tax credits, where they actually get paid these production tax credits if they produce renewable energy power and then sell it. It doesn't have any constraint around what price they sell it or who they sell it to, but so long as they sell it. So this resulted in kind of a weird market dynamic where people are building wind farms in places where it's very consistent. windy and where they're going to produce wind power sort of the highest percentage of the time,

independent of if there's actually a buyer there. So you look at a market like West Texas, where the wind is very consistent, plentiful, but there's not a lot in West Texas to actually demand that power. And so people have built these large 100 plus megawatt wind farms. These wind farms are actually generating power very, very consistently. But the rate at which they sell that power is often negative. So they're actually paying someone to basically get rid of the power so that they can collect that production tax credit. Sort of created this weird market inefficiency. We can help solve that problem by actually partnering directly with those upstream wind energy and solar producers by co-locating digital infrastructure on site and again, bringing a market to an otherwise stranded

energy resource. So it's honestly not too dissimilar from our flare mitigation business. It's sort of a a different problem, different segment of the energy market, but it's very much aligned with our core value proposition of wanting to deliver climate aligned large-scale computing infrastructure that can also be cost-effective. Yep, fascinating. So you're capturing energy that's not being used, you're putting it to use. I guess what are the differences in downtime tolerance between cloud computing product and the Bitcoin mining?

And can cloud kind of be adapted to tolerate? downtime? Yeah, it's a good question. So we've tried to actually build out a portfolio of different computing applications that have different reliability constraints. And so our cloud business today is primarily we consider to be mission critical. We try to have a very, very high runtime, whether it's running up flare gas power or on-site renewables. We've had to sort of engineer some solutions that can deliver five-nines of power reliability. Now, Bitcoin mining is obviously a lot more flexible. So when you sort of pair this together with cloud computing, you end up with sort of this stack of interruptibility. And ultimately, we're going to be adding some preemptible

instances on our cloud platform that will be able to manage to lower reliability standards, but probably still you'd want to target higher reliability than something like Bitcoin mining. And that's one of the beauties of Bitcoin mining, frankly, is it's like the most flexible, interruptible, curtailable computing workload in the world. And it's sort of like a power sponge. It can soak up any excess kilowatt hours that are being produced and it can turn waste energy resources into the hardest, soundest digital money that's ever existed. I think what's cool is when you look at it from a campus perspective, you can actually

orchestrate the way power gets produced and utilize on-site. to actually manage multiple different computing workloads in a way that actually benefits all of them and actually drives down the cost for all of them. Because in the case of your cloud computing workloads, you're typically having to engineer backup power generation resources and other things to manage a high degree of uptime. If you can now monetize those backup generation resources with Bitcoin mining, it actually makes the overall cost of the whole campus go down quite significantly. I think one of the interesting things here is that this isn't necessarily like you guys set out to just mine Bitcoin to have it be this positive climate impact story. It's very economical because this power is super cheap.

And that's the same kind of concept on the cloud side. So if you're talking with an AI company that needs compute, sure, they feel good about potentially it being using what would have been wasted kind of energy. but this is also very economical. Yeah, that's right. I think we've tried to align economic incentives with environmental impact outcomes. The tradeoff that we've really made here, any business I feel like is,

you're just making a set of different tradeoffs. But the tradeoff we've made here is that we're actually willing to build data centers in places that people historically haven't, which means maybe some challenges on building out the networking architecture and footprint. Maybe it means having a little bit more latency to get to the location.

But because of that, we've actually targeted workloads and applications that are more tolerant of a bit more latency. And that's really what's led us to Bitcoin mining is a great use case. It's a globally decentralized network. Having hundreds of milliseconds of latency is actually tolerant within Bitcoin. You can actually calculate what your cost is per extra millisecond of latency. But it affords you quite a bit of flexibility there. In the AI landscape, when you're looking at training large language models or running these big training workloads, those are very latency agnostic as well. Even on the

inference side, if you're feeding tens or hundreds or thousands of tokens into a big transformer-based model, the inference time actually way exceeds any sort of additional latency that's required to get to a data center that might be in a place like West Texas, where it's like you're adding 50 milliseconds of latency compared to a data center that's in northern Virginia. But ultimately, that doesn't matter because the inference time is two seconds. It becomes kind of a rounding error. And that characteristic of AI has actually opened up new possibilities of how the data center world architects, the infrastructure needed to support its growth. Got it. I guess on the AI side of things, how do you think about new hardware coming online? How are you planning for the H-200?

Yeah, there's a lot of interesting hardware solutions that are coming out. This boom has unfolded. It's created a very attractive prize for, you know, many. And I think many chipmakers are sort of watching Nvidia stock go up and up and up as their data center sales unit is, you know, not getting it out of the park. You know, it is a very fast-moving landscape. When you look at the H-200, which is the next generation from the H-100, ultimately, it's a very similar architecture to the H-100. It's the H-100 with a little bit more H-PM high-band

with memory. What we're seeing from a benchmarking standpoint is it performs slightly better on certain inference workloads, but it's fairly similar performance on most training workloads. So we are excited about that, but the next generation of Nvidia architecture, we're also very excited about as well. You know, we're sort of expecting towards the end of the year and into early next year. And we're also excited about kind of other things that are happening in the silicon, you know, the AI silicon ecosystem between AMD recently launching the MI300X, which is sort of their competing product to the H100, which we've seen, especially from a price performance standpoint, be able to get some gains on the H100, especially at inference workloads.

That ship actually has a lot more high band with memory compared to the H100. It's 192 gigs of HPM. We think it's sort of an interesting landscape in terms of how things are going to play out. Intel is obviously not sitting idle as well. He's been sort of investing in what's called the Gaudi platform. Constantly sort of watching that and excited to kind of see what they build. And there's a whole suite of independent startups that are focused on this as well, between Sambanova, Cerebris, Graphcore, Grock, that are all sort of having their own solutions

that may be competitive in the overall landscape. But, you know, ultimately, Nvidia is still sort of the king here and really the dominant player in the overall AI chip space. For your customers in the AI world, is hardware become less of a bottleneck? Or what are the key bottlenecks for compute? Is that eased at all?

Like, you see the headlines, but what is it from speaking directly to them? So, you know, I think the numbers I saw were that video was going to produce 500,000, H-100s in 2023, or that was the estimate, and they were going to roughly produce around of 1.5 million

here in 2024. So there was a mad scramble to get H-100s in 2020, and they were very high in demand. But I don't feel like it's like eased up at all because a lot of the things that are working are actually demanding and requiring a lot more capacity.

And then you have a lot of big players that have made major moves in terms of building out large clusters to try to compete in the AGI race. So Facebook announced last week that they're going to be building out a 600,000 H-100, well, no, H-100 equivalent cluster, I think, of which was about 350,000 were actually H-100s. But when you look at that, I mean, that's a staggering capital investment. I mean, it's like $20 billion week CAPEX investment in terms of building out this cluster. and that's really just coming from one company. There's a lot of different other folks that are out there competing in this landscape.

And then the applications that are working, driving a lot of adoption between chat GPT and many of these other chat bots, things like Character AI. They're just demanding tremendous amounts of resources to support the inference workload. So from our perspective in our seat, the supply crunch hasn't loosened at all. In fact, maybe it's got worse. Yeah, that makes sense. In our kind of crypto world, I mean, how do you think about some of these decentralized Airbnb for GPU networks like Render? What do you think about these? And why do these probably not work for AI? So Render, I always thought it was like very interesting and cool founding story.

And O'Toy was like very sort of a leader in sort of the rendering world. So today, I would say a lot of the workloads in big GPU clouds are still very focused on training. And when you look at training. Decentralized architecture actually just doesn't work that well because you're very focused on not just, I think there's been a lot of focus around like the GPU itself of like, okay, I need an H-100. What most training workloads really need isn't just one H-100. They need a big cluster of H-100s that are networked together and the unit of a compute has become the cluster, not necessarily just the individual GPU. And so you need a design that you need to be thoughtful about how the individual GPUs are networked together within a single server, so a single node,

as well as how all of those nodes are interconnected on the same networking fabric. So what we've mostly focused on in terms of Crusoe and Crusoe Cloud is actually building out this high-performance non-blocking fabric within Finneband in what's called a rail optimized architecture, which means the pathway from getting from one GPU on one server to another GPU on another server is optimized in terms of the number of network hops that it has to actually go through. And so when you try to apply the lens of decentralization to this, it becomes really, really challenging because the clusters themselves are somewhat non-uniform. It's not like you can just easily swap one in out for the other.

So that's one big thing. And then things like data locality and like high-performance storage and having your data required to run your big workload on. a stored local and some sort of amountable volume also becomes important. And it's sort of just a challenging problem to solve. Where I do think that decentralization could play a big role is actually in the inference space. It's kind of like a subset of the inference space, though, because you look at some inference workloads and they're actually multi-GPU or multi-node inference workloads, which actually require that same

networking stack that I was talking about earlier. But in some of the smaller models, so maybe say like a seven, billion parameter model that can run on a single GPU, so you can fit the whole model and memory, those actually become more appealing from a decentralized architecture because GPU computes suddenly becomes more commoditized and easy to sort of manage and run. And you're talking about moving the model to a specific location and then the actual tokens that would feed into the model. It's not a huge data footprint. So you're ending up in a situation where these small, text tokens that are feeding into a model. It's not a huge networking constraint. And you could probably

work through something in a very decentralized fashion with that. Fascinating. I guess back thinking about the Bitcoin side of the business, the mining side of the business, what is it like to run a miner ahead of the having? Not our first having, not our first revenue here. For maybe newer listeners, the reality is your revenue just gets almost cut in half in one single day. Can be intimidating. I think ultimately, like our thesis around Bitcoin mining is, it's sort of this commodity market for computing. And, you know, like other commodity businesses, if you can be a low-cost provider, you're going to have bull markets and you're going to have bear markets. You're going to have moments where you have outsized margins and EBITDA, EBDA margins and operating margins.

And then you have moments where those margins are going to be compressed dramatically and people are going bankrupt. And you can see this across many different commodity production businesses, the swings in Bitcoin are particularly volatile. But I think our philosophy around it is that if you can be a low-cost operating player in the space and you can actually just survive the downturns, cover your costs and not be over-levered, you can actually make meaningful gains and sort of the upswings and sort of the bull markets that unfold. There's a bit of like a baked-in convexity with the space where it's like as Bitcoin economics get worse, sort of the lower efficiency miners drop out, which actually drives difficulty down. And the more efficient providers

can essentially get a bigger share of a smaller pie balances out some of the negative price dynamics, whether it's from having or price action in Bitcoin or difficulty movements in terms of people adding more capacity to the network. Got it. Great. Yeah. But overall, I mean, look, I think what's interesting right now, too, is you have a next generation of hardware. People are building out quite a bit with S-21, which is the new Bitmain chip that was just released and starting to ship. You're seeing hash rate sort of grow quite a bit, and people are circulating on what this is going to mean, and there's quite a bit of hash rate that's out there that's from previous generations.

I think a lot of that will fall off with kind of the halving, if we don't see a significant price movement. But the having is always one of these interesting moments where it's been pretty consistent. We've had a pretty good bull market after the halving every single time that's happened. There is kind of a structural reason for that. It will become less relevant with each future having. But the structural reason is essentially miners oftentimes will sell the newly minted Bitcoin that they produce. And that creates sort of a persistent sell pressure on the market with the amount of new Bitcoin being produced being cut in half, or the rate that is being produced being cut in half, that pressure lightens and assuming lying demand is fixed, ends up with a massive upward

pressure in the market. So that actually improves Bitcoin economics, which I think is what a lot of miners right now are speculating on, is that we're going to see a price rally. But it's tough to sort of bank on that as a miner. Yeah, certainly. On the climate side, I mean, you guys are obviously doing a lot, can you share anything around what impact you guys are having in terms of if you have that kind of audited outside of the company and just kind of how you think about improving that? Yeah. So broadly speaking, from an energy perspective, we're very focused on the way we source energy

to power our operations. We sort of have this broad off-grid mining or off-grid computing business. And then we have an on-grid computing business. on the off-grid side, methane emissions have been identified as one of the key components by any global thought leaders in terms of sustainability as one of the key components to really resolve if we're going to meet any of our goals, any of our climate goals as a society. I think the one and a half degrees target was set Paris Climate Accord, I think, is probably we're going to miss that one, just being pragmatic about this. I don't really see a mechanism for us

to get there. But I do think the world's going to adapt, and it's a complex problem to solve, frankly. We fundamentally believe that methane emissions is one of the most important pieces to solve in that problem. And the reason for that is that methane traps 84 times more heat in the atmosphere than our equivalent of CO2. This has sort of been the reason that the Biden administration set up this methane pledge. If you look at something like the IRA, the Inflation Reduction Act, Of course, has nothing to do with inflation at all and has to do with climate and energy. If you look at the Inflation Reduction Act, you know, it's filled with a lot of incentives for a next generation cleaning up the energy production side of the economy.

It's filled with a ton of incentives for wind, solar, geothermal. One of the key aspects of the IRA is it has a penalty for methane emissions. So it's actually a lot of carrots and a big stick for methane emissions. So providing a solution that can help reduce methane emissions for oil and gas companies and landfills is like a great solution to actually avoid a penalty, provides further financial incentives for people to work with solutions like us that can help them cut their emission footprint. When we look at our business as a whole, we actually have tried to be fairly transparent with the world in terms of kind of what we're doing operationally. We published

a ESG report in May of last year that covered our 2022 operations. teams actively working on our 2023 report. But in that report, it sort of demonstrated from an emission standpoint, we actually reduced 60% more emissions than we created as a business, which was a pretty cool statistic that I know the team was incredibly proud of. Carbon accounting is sort of this weird, new accounting science that's been deployed. And it's kind of weird to sort of think through, but from a carbon accounting standpoint, that actually put us as a net negative company, which means we reduced more machines than we actually created. That's great.

Obviously, when I met you a number of years ago, I think pretty much 100% of the business, the revenue you're generating was on the mining side, but you always kind of saw this AI wave coming. Sounds like you guys are benefiting tremendously from that. Put it on you. Are there other pockets of emerging technologies that you could seek breaking out in the next three to five years,

probably on that cloud computing side, that Crusoe could start to capture? Yeah, our cloud computing business is focused on AI. We also do do quite a bit of work with other high-performance computing applications, so call it computational physics, computational biology, drug discovery, some of these things that have AI elements to them, but are also very, very computation and complex. And then also things like VFX and rendering as well. But when I'm looking at the landscape, to me, I think there's going to be plenty for us to do within AI to sort support growth and demand. We're sort of seeing there as AI applications get embedded into

tons of enterprise workflows and consumer workflows. But that being said, my forecast on what is like the next big wave of technology, to me, one of the big trends unfolding is actually in the privacy space. I think there's some really interesting trends unfolding in terms of private computing architectures that actually enable more permissionless, decentralized computing to take place. So to give you an example, Intel has this private computing platform called SGX, that even if I have access to the server, if I've access to the network, if I've asked to anything, I can't access anything that's taking place within the CPU.

And I think trends like this unfolding, I think, are a positive for decentralized, centralization and actually open up new computing paradigms in terms of the amount of decentralization that can take place in terms of supporting growth. That's very exciting. One more thing I'll say here too is I've always been a long, long term believer in terms of full homomorphic encryption space. We sort of like are able to solve that in a computation and feasible way. I think it's going to be a game changer for everyone.

And what's been interesting about the unlock and AI is that it wasn't like we had a new model. I guess the transformer paper was like a big milestone. moment, but multilater neural networks have existed for many, many decades. The unlock that actually made AI feasible for running these big nonlinear modeling techniques, I was actually more computing, more specialized computing, and more data, this massive paralyzation that took place with GPUs, and then massive data sets that have been sort of built and stored that people can actually train on. I think you're probably going to see something very similar take place in the

encryption space, where it's like people actually building specialized hardware to solve these problems in an accelerated fashion. So like a homomorphic encryption ASIC that can dramatically improve the computational performance of encryption to make private computing a more feasible technology. Sounds like an exciting future. Well, Chase, I appreciate you coming on. It's truly one of the most fascinating and important companies, I think, being built today. So thanks for joining and where can people learn more about you and Crusoe? You can find us on Twitter, on LinkedIn, also our website, crusoe.a.i. And we'd love to hear from you, whether you're interested in computing resources or

interested in jobs. We have one of the most unique employee bases in terms of a mix of electricians and mechanics and technicians to machine learning engineers, system engineers, people, mechanical engineers, designing and building data centers and cooling solutions, all sorts of different roles and jobs that we're hiring for a crusoe. Awesome. Well, thanks so much for joining Chase. This has been great. Cool.

Thank you, Sean. Thanks for listening to another episode of On the Brink with Castle Island. To find out more about Castle Island, visit castle island. Visit castle island. To listen to all of our podcast episodes, please go to On thebrink dashpodcast.com or just click on the tab in our website. Thanks for listening.