Grey Beards on Systems - 143: GreyBeards talk Chia cypto with Jonmichael Hands, VP Storage at Chia Project

Episode Date: March 2, 2023

Today we interview Jonmichael Hands (@LebanonJon, LinkedIn), VP Storage at Chia Project , who has been in and around the storage business forever, mostly with Intel and their SSD team, before it was sold. He was technical marketing for NVMe. He also ran the security and crypto track at FMS2022. He recently worked on sustainability, … Continue reading "143: GreyBeards talk Chia cypto with Jonmichael Hands, VP Storage at Chia Project"

This is another of our occasional video podcasts, so bear with us as we try to fine-tune the process. As our co-host today, we have Mike Harsh, a friend and overall good techie that spends his life in edge IoT and sometimes on crypto. We have with us here today John Michael Hand, VP of Storage at Chia Project. So, John Michael, why don't you tell us a little bit about yourself and what's new at the Chia Project? Yeah, so I joined Chia in May of 2021. Prior to that, I had been at Intel for 10 years in the non-volatile memory solutions group doing SSD development, product management, product marketing, kind of you name it for the SSD product line at Intel. As you guys probably are aware, in 2021, that whole business unit ended up getting sold off to SK Hynix. That's

now SolidIme and after that. I went over to Chia around early 2021 during this whole crypto craze. If you guys remember Chia in 2021, it was pretty interesting to the pretty interesting i don't know if you guys remember remember kind of what happened but uh yeah we like sold out uh all of hard drives in china and then pretty much shortly after us and you know hard drive prices got away up to like i think four or five cents per gigabyte which is like now if people that know the story just kind of like what like cheap nvme drives are today if like consumer drives you can buy for the same price so it was pretty wild times um and obviously the crypto markets uh was pretty pretty nuts back then but uh yeah chia chia is still chugging along we've actually made a tremendous amount of progress in the last two years it's been um so really just actually bringing on

all the actual functional capability of the blockchain. Yeah, Chia's a bit of a bright spot in the crypto space these days. I mean, everything else is kind of having a real serious problem, but Chia seems to be chugging along, I guess. Wouldn't you say, John-Michael? Yeah, so if you go to the website, you'll see secure, sustainable, and compliant is kind of our positioning of the blockchain. And those are really important things, right? So people like Bitcoin, but obviously consumes a tremendous amount of energy and that's not good. So that was part of the reason why we started Chia

was how can we make a blockchain that's as we're more decentralized than Bitcoin, completely decentralized peer-to-peer permissionless networks so anybody can join when they want to and leave. This is something we call Nakamoto consensus, which is you don't really trust anybody, you just trust whatever the longest blockchain is. And so we said, okay, how can we do this without using a tremendous amount of energy like Bitcoin? And that's kind of how we stumbled on storage. And then the compliance thing, which has been obviously top of mind for everybody in the crypto space,

is a lot of these companies that sold unregistered security offerings and were just outright Ponzi schemes are now getting cracked down on by the SEC. This was kind of our thesis in 2021. As we're, you know, if you read our public white paper and strategy, we've always wanted to file for an IPO and be a regulatory compliant offering and be publicly traded. And it would be the first crypto company to be publicly traded in the sense that we have a native crypto asset on the balance sheet. So that's obviously a pretty big undertaking, right? So in order to be compliant, you have to do regular accounting and all this other stuff. I mean, you have to have actual organization behind it, some sort of, I don't know, C-Corp or S-Corp or L-Corp or whatever, don't you?

I mean, it's not an easy process to be compliant to that sort of stuff, right? No, it certainly isn't. And yeah, thankfully, our CEO and CFO have done this before with other public companies. Our new CEO, Gene Hoffman, is very experienced in this space. So he started eMusic and other popular things. So he's done this stuff before so so can we dive in a little bit to the value proposition of chia in terms of like energy so um is it fair to say that the way chia works is by with a different proof of work that uses a lot less energy and has kind of a longer tail of... So the goal of Chia is to have the same or better security, but with much less embedded energy in the network. Is that kind of fair to say?

Yeah, so there's this, Chia uses a consensus called proof of space and time. Our founder, Bram Cohen, is most famous for inventing BitTorrent, right? This is this peer-to-peer file sharing network. And a lot of these same principles that made BitTorrent really successful are some of the core ideas that are in crypto, this fully decentralized peer.

And it's actually not a secret. Bitcoin is an homage to BitTorrent, this whole idea of the naming of Bitcoin. So the idea was, so Bram had this hypothesis that one, he's like, so it seems like there's a lot of storage out there. The whole idea with Bitcoin was supposed to be this one CPU, one vote. And obviously that devolved into the single purpose ASICs that all they do is shot to 56 to mine Bitcoin. They consume a lot of energy, create a lot of e-waste. That's bad.

But storage is, you know, a terabyte is a terabyte. Obviously there's different latency and everybody in the storage world knows these memory pyramids, which is, you know, you pay more money for storage media that has faster, you know, lower latency, like DRAM and SSDs, hard drives. But the actual space is actually a really nice commodity because the way that we do our proof of space in Chia is you don't get any benefit from extra latency. So if you have a terabyte of SSD or a terabyte of hard drive, you're just earning rewards for one terabyte of storage. And IDC has this nice report called the age of the data sphere, which basically says, here's all the data in the world that's created. Here's all the data in the world that's stored. And then they have this graph of how much of that is underutilized. And I think in 2020, maybe it was 2022, it was like eight

zettabytes of storage. It was like over three zettabytes of underutilized space. So that's one good thing. Like, okay, there's a lot of storage out there. The other thing we're doing is focusing on the circular economy, which is like getting used storage back into the crypto market. So today, unfortunately, what happens is if a hyperscaler like, say, Microsoft or Meta or something uses a hard drive, they use it for their five-year lifecycle. And then at the end of that, when they decommission it, they do this encryption, full media sanitization on it, and then they throw it into a shredder because of data privacy concerns. And so one of the big ideas for Chia was like, hey, can you guys stop doing that? Let's stop destroying all the storage, and then we can get it back and recycle it back into

this market. And crypto is a really nice, interesting use case for these used hard drives where you don't really care. It doesn't store user data. We just store cryptographic caches on these hard drives. So you can use any any quality of media like used hard drives are perfect so you referenced uh in one of the emails to me john michael the ieee paper that you did i think with tom uh on on the uh your repurposing of storage and things of that nature um i i didn't have to admit I didn't quite finish the paper, but it seemed like it's an interesting approach to doing things differently than shredding all your magnetic drives and things of that nature. I've been spending a tremendous

amount of time over the last year and a half in the sustainability space, a lot around circularity. So we have, many of you guys maybe know the Open Compute Project. So there's a new sustainability project, which I'm heavily involved in. We released a white paper last July on media sanitization of the circular economy. I've been working with folks like Meta and Microsoft and Google, the chair of those work groups to kind of get these standards. We started a new nonprofit called the Circular Drive Initiative, where we have Chia, McDonough Innovation, Micron, Western Digital, Seagate, and a bunch of other storage vendors in this work group, basically trying to solve circularity and storage. And then, as you mentioned, I'm part of this IEEE

security and storage working group, where we actually created the brand new specification on media sanitization. It's called IEEE 2883-2022. And this basically describes some methods to basically be able to securely remove all the data on a drive, completely enabling reuse. The ones that people are common, people used to call this like secure erase or people call it wiping or erasing and shredding. Well, we have a specific scientific word for it. It's called sanitization.

And this is the process of rendering the target data on a target device inaccessible for a given level of effort. So these purge media sanitizations are actually secure versus nation states. And which is surprising to people, you don't need to buy fancy software to do this. The firmware in every single modern hard drive and SSD actually supports sanitized command by default. So that's amazing. Yeah. But how can you trust it?

Yeah. And this is the million dollar question. Yeah, this is yeah. So, you know, there's, you know, for, for the majority of people, you basically can write, there's something called the verification of sanitization. So there's two camps here, right? for like a FIPS 140-3 certification, they actually look at the firmware implementation for sanitization of like, is the key wrappings and encryption methods the right standards? Or is the firmware operating properly? But realistic verification is you write a known data pattern to the drive, issue these erase commands. And in a crypto erase, for instance, you're just actually changing the media encryption key on the SSD or hard drive. Some of the hard drive vendors call this like sanitized instant erase or, you know, ice or whatever, you know, it's just, it's called crypto erase and sanitize is what the official name of it is. But you're basically just changing out the media encryption key. And then when you

read all that data back, it's just going to be random data because you've changed the encryption key and all the data is still on the drive. It's just, you've thrown away the key. So if you write a known pattern to the drive, do a crypto erase and then read the data back you're not going to get the same data and so yeah yeah i mean it works well for encrypted drives but i don't know if it works well for non-encrypted drives i guess most of the hyperscalers probably use encrypted you know for hard drives uh they basically segment these out and in the future they will be right so if you guys have paid attention paid attention to what's going on in the future, they will be right. So if you guys have paid attention to what's going on in the storage world, obviously, as you're well aware, the consumer hard drive market is kind of dying in favor of SSDs. And most of the hard drive business is going to these hyperscale type drives.

And it's very, you know, there won's something like trusted computing group or something like that. They do this actually to scramble the data before it hits the NAND because NAND doesn't like data that's patterned. So, yeah, any SSD used to be like NVMe format or something like that for some of the very first generation drive. But basically any drive after 2017 that supports NVMe 1.3 or greater will support this sanitized command. Amazing. Yeah. But yeah, these little things of getting this data out to the world is why we're running these white papers and starting these initiatives. And that trust thing is like the really hard part because, you know, trust at a hyperscaler that has zero tolerance for data leaking, right? Like Meta or Microsoft,

they're not going to allow somebody to go to eBay and buy a drive that, you know, used to have customer data on there. It just doesn't happen, right? So that's a little bit more challenging, right? Where they need to have hardware roots of trust and firmware attestation and all this encryption for data in flight and rest and lots of other things added to that stack to where they feel comfortable. Versus like, you know, if I'm selling a drive on Craigslist or something, I can just pop the drive in the machine, run the sanitize command two seconds later, and I'm fairly sure all the data is gone. Right, right, right, right. seconds later, and I'm fairly sure all the data is gone. Right. Right. Right. Right. So if you're successful in implementing these initiatives, what's the vision of the life cycle of a drive?

Like between now and like when hard drives are obsolete, is it just a matter of extending the use for life? Well, hard drives never go obsolete, Mike. I mean, that's a whole different question. I'm sorry. Go ahead. Yeah. It's just, you know drives ever go obsolete, Mike? I mean, that's a whole different question. I'm sorry, go ahead. It's just, you know, they go down the storage stack, right? To where, and we're already kind of seen that

where, you know, hard drives are still used for what like meta or other folks call warm storage, which is like frequently accessed blob storage. That's the majority of their data, which is like stuff accessed within a couple of months, you know, pictures, videos, whatever. Yeah, the ideal life cycle of these drives, right? You know, the reliability of hard drives and SSDs

is actually extremely well understood. You know, it always funny, you know, I worked in the corporate reliability department at Intel for a long time as well, and early in my career in the validation. And, you know, you have these Pareto's and you know the annual failure rate and the annual return rate of drives.

And I mean, for some of the most popular Intel SSDs that had like, you know, five to six million units shipped in their lifetime, you know, after like two to three firmware updates, because a lot of SSD failures are actually around firmware. The AFR was actually like one half of the rated publicly listed AFR. So it was like 0.02%, like it was like 0.2% AFR was actually like one half of the rated publicly listed AFR. So it was like 0.02%.

Like it was like 0.2% AFR. And actually some, some models got down to like 0.1%. And most customers are not using the full endurance of the drive. We found NetApp has this really nice study where they, they took like 1.6 million drives in the field and they had their telemetry running on the NetApp software that, you know, give all the feedback and the smart of the drives. And they found out that the overwhelming majority, 95% of the drives were less than 5% used after a full five-year lifecycle. So for one, for SSDs that can last 10, even longer, right? There's AWS instances that

are running some of the very first Intel SSDs and spot instances right now. That's 11 years, 12 years after. I mean, so the life cycle of these drives is well, well above five years. And everybody knows that, right? You can buy a drive on eBay, right? There's Intel drives that I worked on 10, 11 years ago on eBay. They're selling right now and they work just fine.

Which, so yeah, the idea around the circular economy is you cycle them from one value chain to another. So Hyperscale will use them three to five years, and they may have a benefit to go to a higher capacity drive or denser storage or better TCO. But once they're done with that, the idea is instead of shredding them, they can securely sanitize them and then get them to one of these other use cases, whether they're selling these to a distributor or to a crypto farm or anything else. The idea is that now these can get another four to five years of useful life before they finally break. And then because they didn't have any user data on them in this crypto use case, then they can safely send them back to C8 or West Digital or whoever the vendor is for recycling all the components, the magnetic. Like hard drives especially have these rare earth materials

in the magnets. You know, so you definitely don't want to be using Windows and Apple. And SSDs is even worse because obviously it takes a tremendous amount of electricity and water in these giant fabs to create the NAND wafers. And so the idea of just taking a perfectly good drive and hucking in a shredder is just, it's very sad to me, right?

When there's a ton of use cases for that storage elsewhere. So Chia is one of the main use cases for this sort of storage. I mean, from a crypto perspective. You know, right now we have like 24 exabytes of storage on the Chia NetSpace. At the peak, I think it was around 48 exabytes, which was like, you know, in 2021 was like 3% of global hard drive sales. Right. So it was a pretty big chunk. You know, if Chia's if I was to say when when Chia is a top 10 coin. OK, OK. If we're being wishful thinking here. But if we're a top project and being used to secure global finance, it is very realistic that we'll see 300 or 400 exabytes of storage in the Chia network.

And I'm working very hard in my job to make sure that we ethically and sustainably source that storage. One, because I just told you, it already exists. We know that this storage is out there and it's just getting destroyed. And so that's the big unpleasant part of this whole thing is we want to be able to sustainably grow and do this in a way, you know, we're going to be completely impervious to any nation state attacks. You know, nobody's going to be able to just buy 300 exabytes. You know what I mean? So, you know, and we, again, we, so we know the storage is out there either being underutilized or being destroyed.

So that's, that's the end goal. So, so. Isn't Chia already, you know, a very distributed environment? I mean, it seems like it's got, you know, more, I don't know if the right term is farmers or harvesters than any other, you know, equivalent crypto coins or that sort of thing. I mean, like you said, Bitcoins are all ASIC. You know, the big guys are all ASIC driven and, you know, and that's about it. And it's almost to the point where if you, you know, if you control the ASICs, you could control Bitcoin to a large extent. Yeah. So in the crypto world, in Chia, we have farmers, not miners. Thank you,

Ray, you perfectly touched on. And again, we called it farming because you're not putting a bunch of continuous energy on there. You basically make these plot files. These are like cryptographic 100 gigabyte files. You copy them to the hard drive and then they basically just sit there idle. You know, a drive farming to you today is 99.7% idle. So, you know, it's like basically the easiest workload. It's a, yeah, right for you storage guys, it's a 0.5 IOPS is the crypto,

is the read-only workload. So it's a, it's about the easiest workload and drive we'll ever seen in flight. So a half, a half IO a second effectively is what you're saying. That's it. Yeah. Modern hard drive. It's a couple of kilobytes a second. It's nothing. Yeah. And the, yeah. So we have a, if you go to dashboard.chia.net, this is our public dashboard. I think we have over about 130,000 Chia farmers today all over the globe. And the reason that we have so many, and we believe that we are the most decentralized cryptocurrency in the world. I don't have any evidence that any other crypto is bigger. You know, Bitcoin has at its peak had like 60,000 or something like that.

At our peak, we had like 450,000 nodes or something like that in 2021. Yeah, so the whole reason why this is not just an accident, this is one good engineering on the full node part. We have really dialed in the database and IO requirements to actually running this fully peer-to-peer node. It runs on something like a Raspberry Pi or a single CPU VM. It's like nothing, right? And the only thing you need to join the Chia network is 100 gigabytes of space for a single plot file, and you can start farming. So if anybody has any spare storage space,

they can turn that into earning money by utilizing that spare storage space and putting it up to the Chia network. So the reason why we have so many farmers is there is actually a financial incentive for them to run these nodes. They actually get paid today about 50 to 60 cents per terabyte per month, which may not sound like a lot, but when you can buy a used hard drive for five or six bucks a terabyte, you're effectively paying that storage off in like 10 to 12 months. That's actually a decent deal. Yeah, yeah, yeah, yeah. You mentioned earlier about the IDC reports that show underutilization being a significant component of storage that exists out there today.

And I'll take that as a given. I mean, it's certainly feasible. I mean, do you see something like Chia being a way to take advantage of that underutilization in the enterprise, let's say? I mean, if you have, I don't know, a petabyte or two sitting in your enterprise in which you're only using maybe,

let's say three-fourths of that, and you've got like 250 terabytes sitting around someplace. I mean, do you expect somebody to go fill that with the chief plots? You know, the traditional enterprises, it may be a little bit tougher, but there are some companies that are taking advantage of this. Like, you know, and I was just just at CES on a panel for the Blockchain Association and there's a fellow there from Crusoe Energy Systems in Denver. And they do this flared gas Bitcoin mining, and then they have this GPU cluster for rendering.

But when they're not using that cluster for AI use cases, they're doing crypto stuff in the background. And so we've seen that with Chia, where people have these assets that are used for AI and they have a bunch of storage. And then when they're not using it, the customer goes away, they can flip that over to these other crypto. And Chia is a great opportunity for that, where they can say, okay, we have extra storage. we can put it up to the network for a little bit and then a couple months later okay we have to erase a little bit of it for you to discuss for workloads they can reclaim that space as needed uh but this is a really tremendous way to basically uh leverage that extra um space and basically pay for itself over time i had a friend of mine that, you know, as he was,

they would, they would buy storage ahead of when they actually needed it. And, and they would plot on the storage until they actually started to use the storage. And I'm using that as sort of a Chia framework and things of that nature. Exactly. Yeah. With the way we have it set up,

you know, there's no, there's no user data. You're not storing any, anybody else's files or anything weird, you know, so there's not really, once you understand the protocol, you'd understand there's not really any security concerns by doing this. Like it's completely, completely fine. kind of got some traction where Backblaze did an analysis to decide whether they wanted to do that, basically plot Chia ahead of user data on their system. And I was surprised at their conclusion to not do it. And I'm wondering if they're going to revisit that

in light of recent developments, being able to create plots a lot faster. Yeah, I would say it was much more challenging. The software in 2021 wasn't very mature for these generating plots. And that's something that's, you know, two years of innovation has come, you know, used to, I know people were worried about like, you know, wearing out their consumer type SSDs by plotting. And there was all this, like, you know, there was like this gold rush in 2021, where everybody's trying to get in and nobody really knew what they were doing. You know, now you can do this completely in memory and super fast. And so it's much the barrier of entry, especially for somebody who has a lot of resources like a larger company.

It's much, much easier than it was in 2021. I don't know if you guys remember this, but I did actually speak with Backblaze after that. They interviewed me. Yeah. And we had a great interview. It was just like, you know, their scenario was like other companies. There was a lot of uncertainty in just the crypto market. And at that point, the Chia NetSpace was just growing exponentially like crazy.

And we're like, okay, when will this ever stop? But the biggest challenge is that in these... And this is the same reason why you can't farm Chia on AWS S3 or Dropbox or any of these services is because they replicate the data. And in some of the more sophisticated storage systems where they're using colder storage, they're erasure coding it. Or in many of these systems, it's just straight up a mirror replication. And when you're doing something like Chia, that storage now has to compete with somebody else who is not doing that, not using any redundancy, not using RAID, not using erasure code,

and they have a better TCO. And so the economics are never going to work out unless you actually, you know, strategically orient the storage software, you know, to basically not use all this replication and overhead. And there's ways to do that where you could have a, you know, shim file system layer or something where a fuse layer where, you know, the plots are underneath the actual real user data and some other stuff. But, you know, we, we've seen a couple of interesting, I mean, somebody wrote their own file system for

plots. I mean, like, I mean, there's all kinds of crazy stuff the community has written. I mean, it's like, okay, you've nerd sniped me now. What's the, what is this shim file system that I can use of like a, a nice to have data class that you can then just overwrite as needed, you know, it's like, so, you know, storage since its invention has always been about like trying to keeping bits, like trying not to overwrite data as hard as possible. It's a totally new paradigm. And that's funny for a lot of like real storage guys that get into Chia. They're like, you know, I tell them like, no, you know, you like throw reliability out

the door. Like it doesn't matter. I can untrackable bit error rate of 10 to the minus 12, like perfectly fine for Chia. Like it's just random data. They're like, like, and I'm like, you know, don't raid your drives together. They're like, but what if one fails? You're like, don't worry about it. You're not losing any user data. They're like, and I'm like, don't raid your drives together. They're like, but what if one fails? You're like, don't worry about it. You're not losing any user data. You just recycle the drive and move along your merry way. Yeah. So it is a different mindset of thinking about this new type of data, but it's also tremendously fun for storage folks because

it is like somebody mentioned something about Chia farming, like being analogous to like Sudoku for a sysadmin. You're like, you know, you're moving, you're creating these data files and moving them around. Like I have scripts that are moving hundreds of terabytes of storage a day right now for my Chia plots. And it's just super fun, like architecting these bottlenecks and interface, you know, buses across network and storage and like actually seeing how fast you can

move these plots across the systems. It's, it's super fun. Yeah. I understand to some extent why you wouldn't, you know, do something like Chia plots and then, and then archive them off the tape or something like that. And then bring them in, you know, if you had a tape library or something like that, bring them in on demand as needed or something like that. Yeah. You know, had a tape library or something like that bring them in on demand as needed or something like that yeah that's a question you know we specifically set the protocol up so the tape wouldn't work

yeah you know if if somebody can do a random read on tape under 30 seconds then then they're then they're golden but uh yeah with the whole lto tape and stuff like that you can do it quite a lot under 30 seconds but it has to be mounted yeah that's the challenge yeah that's um yeah it's funny again this the workload looks very much like a like it's like an archive type drive you know it's similar to what an archive hard drive would be at like a hyperscaler but yeah like half an iop kind of thing yeah it's yeah it's an archive type of world right yeah the only difference is access. The drive is getting accessed every 30 seconds or something,

which on an SSD is totally fine. The drive just goes to sleep and wakes up to do the IOs, goes right back to sleep. And modern SSDs are like five milliwatt active idle in this state. Hard drives, it's a little bit too frequent for the hard drive to spin down, but they can do some of the things where the hard drive goes into this state called idle B where it parks the head. You can do stuff like that, but you got to tune some of this. But it's pretty well understood. You're still moving ladders and stuff. You're still doing a lot energy-wise, even though heads are parked. Yeah. The big difference is, you know, you have a hard drive that's five Watts,

you know, comparing that to like a Bitcoin miner at like 3,500 Watts. Oh yeah. Yeah. That's true. That's a little bit different. I think there must be an opportunity here for trying to figure out how to use tape with Shia, but that's, that's a different discussion. Well, there, there's a great financial incentive. Anybody who wants to try can get some farming rewards. So, you know, this is the beauty of these open decentralized systems, which is, you know, if you have the right incentives and then the network automatically figures out what the right

market economics are and the right optimal solution, you solution. So you mentioned earlier that over the course of the last couple of years, Chia has matured as far as a crypto capabilities perspective. When you're talking like smart contracts and things of that nature, is that what you're saying? Yeah. So going back to all the stuff, I live in the world of storage because I'm our storage guy, right? This is my job. I work with our main partners.

You can imagine this is like Western digital TGA micron, but you know, there's my kind of friends. These are all my friends. Yeah. And you know, I'm a big part of SNEA and OCP and IEEE and all that stuff. But like the real use of the blockchain is, is the actual use case. And so in Chia, we have these things called smart coins. And it basically allows us to do anything you can think of to program into the coins. And so what we've done for Chia is actually release these as primitives so that other companies can actually bake these into their software suites. So the first one we have is called offers and Chia asset tokens. So basically you can

tokenize anything in the world. Now you could tokenize, one of the things we're working on is tokenizing real world assets and commodities like carbon credits, but you could do a stock, anything you can think of, you can turn into a token. And then we have these offers, which are completely decentralized peer-to-peer trading of those assets. And so what I could do is make an offer for this decentralized asset, post it on an exchange. And it's done in a way where there's no counterparty risk. Like there is no risk of me losing that asset. I'm just signing half the transaction saying, hey, I'm willing to trade this asset for this other asset.

And somebody on the other side of the world can say, oh, I like this. I'm going to take that trade and then they can complete that on chain. So this is full decentralized peer to peer trading. And this has never existed before. And it's almost mind blowing to think about. any markets adding real liquidity and depth and the ability to do this trading, it creates these market makers and all kinds of decentralized exchange, all kinds of really cool tech.

Yeah. That's kind of like the, yeah, that that's the first one. And we have on top of that, we have an NFT standard and there's a lot of people that do this type of trading for NFTs. And it's just like fun art,

you know, where it's nothing like super high value. The one we did as a reference platform for the NFTs was called Chia Friends. And we're donating all the proceeds of that to charity, to the Marmot Recovery Foundation, which is like a kind of funny thing because we have like marmots as the, as the meme in Chia. So that's nice.

That's nice. That's nice. Yeah. And that's kind of the first big area of blockchain. I mean, a lot of the other blockchains have, I don't know, almost languages for their smart contracts and things of that nature. You're not offering that in Chia or you have that as well? Yeah. So yeah, our smart coins are in this language that we invented from the grounds up called Chia Lisp. And it's a Lisp-like program language. So yeah, it turns out this functional language works incredibly well for smart contracts, because what we care about more than anything is security, right? You can't have banks and financial organizations and, you know, using a smart contract that's going to be hacked and

not secure. And so you have to be able to audit these things and know that they're working properly. Yeah. So we have this, yeah, like again, I'm our storage guy, but Chia will likely be most famous for this Chialisp and programming environment. It's so much so that the Bitcoin folks have already been starting to take parts of it and include it into some of the Bitcoin protocol because it's quite frankly, it's much better than what they have. And so one of the, there's all kinds of interesting stuff you can do in this Chia. I mentioned it's Turing complete language. So anything you can think of that governs the money, any single possible thing you can think of, you can do. Now in Chiaia, you know, like in Linux, everything's a file.

In Chia, everything's a coin. And the rules that you program into the coins are like, who can spend this coin? Is there a weight limit before you can spend it? We have something called a clawback, right? Which is like, okay, if you spend this coin, it goes into this escrow period for a certain amount of time. And if a wallet wants to claw back and reverse that transaction within the escrow period, you can. And so this prevents stuff like sending money to the wrong address. Or if somebody hacks you, you can try to go rekey this, your wallet, and then do a clawback of the money so that now the attacker can't have your money because it goes. So there's this whole suite of applications of coin security stuff that we call custody. Now, custody is just like, OK, you're in crypto.

You are your own bank. You are the bank. You are self-custodying the money. And so in Chia, we're allowing now a bunch of better functionality. Our CEO likes to say, you know, we want to be crypto needs to be easier to use than cash and much harder to steal. And smart coins and contracts have this ability to do that. You can program all these really cool things into the money that allow you to have much more security, much more control. So we're going to have these custody vaults for Chia that you can have different classes of wallets, like a cold wallet that you have lots of money in that's super secure, warmer wallets that you're using for more day-to-day transactions and stuff, hot wallet that you maybe use on your phone with state

channels or small payments. And you have all these different classes of wallets and custody with different security locks. The other thing you can do with these things is tie them to dids or digital identifiers where you can say you have uh you you want to delegate your trust out to five different friends and say say you lost your wallet like today if you lose your bitcoin seed like there's no getting that back you just lost all your money forever like there is no there's no call the bitcoin guy and get your money back but in chia we could actually have a way where you could delegate out to a certain number of people and say, okay, if four or five of those people got together and put their keys together, they could recover my wallet and create a new key and claw back that money. Oh, that's nice. That's a nice feature.

So, I mean, reading, you know, watching the video that you had and stuff like that, There's these things called harvesters and then there's nodes and there's farmers. I understand farmers are plotting and saving, saving the plot files on their storage. I understand the node is actually, I'll say, maintaining the state of the bit, the blockchain, if that's correct. I'm not sure that's correct, but it's as close as I can come to. Go ahead. Oh, yeah. As I say, before we go there, just while we're on that topic of the blockchain use cases, just before I forget, we have one more major product called DataLayer, which is basically a federated database, but we track it on chain.

And so you can allow for a decentralized database to have like auditability and transparency of a public blockchain. So the first real world use case of this data layer product that we have is our largest today. It's at the World Bank for something called the Climate Action Data Trust, where they're actually using this technology on the blockchain to track the registry of all the carbon credits in the world. And there's countries like Singapore and others that are actually using this for real registries to track these carbon credits. But this is just another application where blockchains can kind of give this transparency, auditability, visibility into the real world where now you can have a bunch of parties that don't

really trust each other, have data and know that it's coming from the right place. So if I'm like an industry and I'm burning carbon left and right and stuff like that, and I want to actually spend some money and buy some carbon credits to offset that carbon activity, I can use this data layer? That's where they'll all be tracked now. If you go to this Climate Action Data Trust website, you'll see that they're going to have this registry of data and any project that's a real carbon project will be in that registry. And so today there's a lot of shady stuff or somebody's like, oh, they're like, it turns out that they weren't real or from a company that's

already retired them or whatever. Now you can go on the blockchain and be like, oh, this came from this registry and it went to this country and this is the person that bought it. And here's when they were retired. All this happens on chain. And so it's, you can trace it back and actually replay it back. The best part about this is it's anybody, any one of these 130,000 public Fonchia farmers that are running Node can just turn this on and invalidate the data themselves too. They don't have to trust, you know, that the data's coming from,

this whole point is it's all on chains you can see uh you know who signed what commits and uh or monitoring the stuff in the database no let's let's get back so and they're what they're effectively using chia's federated database data later to do this but anybody could use this chia federated database data layer to do other stuff, right? If I want to do Napster or something like that. It's just an amazing, yeah. And remember, this data isn't stored on-chain. The data is stored off-chain, but the hash of the data is stored on-chain and tracked.

And so you could publish data layers for all kinds of really interesting stuff. And yeah, we're going to see some really cool use cases. But think about it. It's just a distributed key value store that now you can audit and transparently look at the records of who updated what, when by anybody on the chain, whenever you want. The applications for this are just mind blowing. Like people have already come up with just crazy stuff. Yeah. This is bizarre. I mean, so the storage is actually off-chain, but the key and the hash are sitting on-chain is what you're saying. You got it, yeah. It's all tracked through this data layer thing that we actually have built into the actual whole client and everything. So yeah, it's very, very slick. Anybody who wants to make a public blockchain app, this is something that you can easily integrate in. So that was, I was going to kind of steer us there. So is that an example of Chia actually going towards storage, decentralized storage as a service or, or enabling decentralized storage?

I know that there's other projects out there that they want to be the, you know, the decentralized Amazon S3 or whatever. Yeah. But is Data Layer something that could in the future enable those kinds of services? Yeah. Well, it's even like somebody could make their own service and then tie it back to Data Layer. So Chia doesn't have to like host this themselves. They somebody could make their own service and then tie it back to data layer. So Chia doesn't have to host this themselves. They could just make their own product and use this as that reporting. Taking Merkle tree hashes of data to making, if you're doing a backup or something, that they could validate that it is the right data and that nobody tampered with it.

But we're not going to go compete with Filecoin or anything. We think that these... Actually, I work with a lot of these guys like Filecoin and storage and all these decentralized storage companies that are actually providing a real storage service. But you're competing with AWS. It's going to be a hard guy. Backblaze, Wasabi, those kinds of guys. Yeah. The interesting thing, which I think is you know the file coin right now is more of like an archive use case but like the price is if i'm reading the prices correctly it's like 10 times less than s3 like so for storing archive data it's well seems reasonable for especially you know now you're going to trust your company on on this stuff for like these crypto decentralized storage stuff maybe Maybe not. Performance is the other side.

Reliability. Certainly at 10x lower price than AWS, now you start thinking, we should go test it or look at it. We can start playing with it and stuff like that. It seems to me that if you're someone who's responsible for a lot of storage and it's connected to

the internet, you can weigh these different options. And, you know, Chia sets the floor on what the value of online disk storage space should be, right? And then anyone willing to pay an incremental amount higher than that, like you would delete your Chia plots and then it would be cool to to have like a

fluid uh system that could arbitrage that market i know that you know your your real sensitive workloads aren't gonna aren't gonna trust their data to um uh to these five plus year old Chia drives or whatever. But I think it's an interesting thought or I think since Chia's like already attracting all of these storage minded folks to the ecosystem, anything that combines the opportunity to leverage that storage in other ways is gonna be interesting to them.

Yeah. You know, just like the, you know, we're starting to finally see, like, you know, 10 years ago, people talked about AI, but now we're seeing it, like, for real, you're seeing stable diffusion, you know, art, like, being generated on the fly from a scribble that looks like a beautiful picture. You're seeing chat TBT. In that same sense of like, people had this idea for a long time, which is there's all this underutilized resources just sitting out there and doing nothing. And there hasn't been the right financial incentives and capability to actually use those for real customer workloads. We're starting to see that now. I'm seeing a bunch of new services that are able to take idle GPUs. And instead of

using them for some stupid crypto mining, use them to for AI workloads. This is a perfect thing to offload for training because it isn't mission critical data. It is ephemeral data. You can just, you know, get a bunch of GPUs from a bunch of different people and throw out a bunch

of different distributed workloads. These workloads are distributed by nature. We're finally starting to see some of this come together and it's been just amazing to see how much progress the industry has made just in, just in the last like couple of months, I've seen a bunch of new projects doing, doing this.

And it's all this, this AI has sparked all this interest again, getting this idea of like using underutilized assets, earning revenue from them, you know, for instead of having to pay AWS and all these guys that are hoarding all the a one hundreds. Right. Right.

Yeah. I think for those reasons that you'll find that there's a lot, a very long tail of value to be exploited in all these resources that are floating around out there. And if you can be the one to, to, to unlock it, there's a lot of potential.

Yeah. All right. So Mike, any last questions for John Michael before we leave? I do, but I'm going to take them offline. Cause they're really specific and, and ultra. All right, John Michael, anything you'd like to say to our listening audience before we close? Oh yeah. I tried to stay high level. Cause I know, you know, I know a lot of the storage folks are not like super into crypto. But I just tell you, like, take a look.

Chia is a really awesome project. And, you know, it's not like I know people hear crypto and think like scams and Ponzi's and all this stuff. But it really couldn't be any farther than the truth. And that's just we have so much real value in unlocking and this decentralized money and financial use cases. And there's so much cool stuff going on in the ecosystem. But mostly if you're a storage nerd, like you should check out GeoFarm. We all are.

It's super fun. Like there's, you know, just making these plot files, moving them around and seeing how this whole system works. You know, you had a question about how all this harvesters and farmers, I mean, we have all this stuff in the protocol about how all whole system works. You had a question about how all this harvesters and farmers, we have all this stuff in the protocol about how this stuff works. And yeah, it's just cool to geek out on and play with. So if you're into storage, just check it out. I know many people probably took a look at Chia like two years ago

during this crazy run up and like the software has matured a lot since then. It's been two years of hardening in this open source system. So yeah, take a look. It's much better, much easier than it was back then. It's been two years of hardening in this open source system. Take a look. It's much better. Much easier than it was back then. That's it for now. Bye, John Michael. Bye. Thank you guys so much.

Bye, Ray. Thanks again for being on our show today, John. John Michael.