Grey Beards on Systems - 39: Greybeards talk deep storage/archive with Matt Starr, CTO Spectra Logic

Episode Date: December 16, 2016

In this episode, we talk with Matt Starr (@StarrFiles),  CTO of Spectra Logic, the deep storage experts. Matt has been around a long time and Ray’s shared many a meal with Matt as we’re both in NW Denver. Howard has a minor quibble with Spectra Logic over the use of his company’s name (DeepStorage) in their product … Continue reading "39: Greybeards talk deep storage/archive with Matt Starr, CTO Spectra Logic"

Hey everybody, Ray Lucchese here with Howard Marks here. Welcome to the next episode of the Greybeards on Storage monthly podcast, a show where we get Greybeards storage and system bloggers to talk with storage and system vendors to discuss upcoming products, technologies, and trends affecting the data center today. This is our 39th episode of Greybeards on Storage, which was recorded on December 8, 2016. We have with us here today Matt Starr, CTO of Spectralogic. So, Matt, why don't you tell us a little bit about yourself and your company? Matt Starr, I've been with Spectralogic for, it's my 27th year.

We're based in Boulder, Colorado. We're privately held. We build our company. That explains 27 years because nobody in Silicon Valley lasts that long. Yeah. It's kind of funny. Boulder is kind of a pocket for storage. We do have a lot of startups, a lot of disc manufacturers here.

But you see people bounce around here too. I just like Spectralogic. It's a great company to work for. It still has that kind of startup feel because our founder and CEO is still with the company. He's been with the company. This is our 38th year coming up, so it's a fun place to work. What do you guys do at Spectralogic, Matt? Our primary focus is back up and archive, and I would say really it's archive. It's data retention, long-term data retention. So our customers end up being the guys with the massive amounts of data. I always say if you have 10 petabytes, I'd like to talk to you.

If you have 50 petabytes, I'm already talking to you. And if you're going to an exabyte, you need to talk to me. Because that's the kind of stuff we end up storing is those massive amounts of data. And it can be either on tape or spin down, power down disk. But it tends to be targeted towards that, spin-down, power-down disk, but it tends to be targeted towards that, lack of a better way to say it, low, slow, and inexpensive storage. God, 10 petabytes. It's going to take me a while to get there,

but slowly but surely I'm running out of space to my backup disks and stuff like that. It's terrible here. Yeah, 10 petabytes. When I try to explain petabytes to somebody, I start out with a USB key, and then I start talking about how many USB keys I store. And I basically can say, you know, fill your house with USB keys, and now you're starting to get to where I'm at. So what's this Black Pearl stuff I keep hearing about? So Black Pearl, we released that about three years ago, and it's kind of an interesting, I always think of it as at least an interesting technology. It takes the kind of the new construct of object storage and RESTful interfaces and then fronts it or puts behind it a tape and spin down disk storage. So you get the object storage front end, the easy put and get nomenclature, very easy to

set up and run, and you get the low cost and storage benefit of tape or disk behind it. I always say S3 to LTFS tape is the way to think of Black Pearl. Yeah, well, despite the fact that you guys stole deep storage for it, I love the idea of Black Pearl. Yes, it's funny, Howard. I think when they dug that name up, they had to send you a note, didn't they? No. You're not getting any royalties, Howard?

What's the problem here? No, I'm not getting any royalties hard what's the problem here no i'm not getting any royalties oh god however the statute of limitations has not yet run out no i guess but but i was sitting in your analyst day when you introduced black pearl yeah i was there and molly said deep storage and my head snapped up and and i realized she wasn't talking to me i thought i really did think she had already talked to you that's that's funny that's interesting because it was the the code name for the product was uh bluestorm and uh nate our ceo when he came up with the name black pearl i asked him if he'd seen the movie and he said what movie and i was like okay we're we're good you know because i go out to see

disney every so often as you know and i talk to them about what's happening in the storage market and and they always tease me about when do they get their royalty check yeah same problem so the other thing about black pearl i thought was interesting was that you kind of open sourced a lot of it yeah you know i would say I've been in storage for a long time, and I would say if you asked me 10 years ago, would I ever open source my code, boy, I would be very scared of that. But I look back at it now,

and I say this is the right thing to do, and it goes back to something I heard, and maybe this was from you or someone else, but name how many Linux or Unix operating systems you see in the world today, and which one has won? Well, the open source one won. You know, when I worked on our Alexandria product, I think I wrote drivers for 24 different Unix machines,

and that's a lot of different flavors of Unix to be playing on. And now, really, there's two or three available, and one's the winner, and it's Linux. So open sourcing to me is a, is a good way to allow your customer aid to help you. It gives your customer the ability that if they have to get off your stuff for whatever reason,

they usually can very easily and it keeps you on your toes. You know, you have to be always out there in front of your competitors because what you've done is, is more or less in the open source. So is it like a community associated with it? I mean, is there like other people actually developing code for it and stuff like that? Yeah.

So we just had, I think it was Liber Brain Institute, I think they're in Oklahoma. They just did a Ceph backup client that they released to open source. So they basically take their Ceph storage system and they make a second copy to Black Pearl. And then they watch the changelog on Ceph, and any of the data that comes into the Ceph system, the changelog will fire off a job, and they'll make a copy of that data off to Black Pearl. So they get basically two sets of object store, but one of them is truly a backup of that object store. Cool.

Did you guys open source the code for the object store itself or just your extensions to the S3 API? So a little bit of that is open source. So if you go up and I would not say the actual S3 interface is open source, but most of the meat behind that, including our port of ZFS to BSD, a lot of people pulled that down, including I think University of Pittsburgh and a few others have pulled down our ZFS port on BSD. Now, what's interesting is you see people running ZFS on Linux, but if you read those two license agreements, those are definitely a contamination. In other words, you shouldn't legally be able to sell a product, at least our assessment is. Let's not go down that path because then we end up at Stallman, i don't want to talk about stallman you know because because the concept that you know we have two open source licenses and the fact that one is slightly more restrictive than the other is now stopping users from having choice is just

users who do it you know users who are sophisticated enough can do it. This is... I'm going to stop this soon, but this is one of my problems with the open source community because they keep going, well, of course anyone can do it when they mean anyone who can compile this from source themselves. And that doesn't include me. And so I generally assume that anything that is too technical for me to do isn't allowed to use the word everyone.

Well, that's why I like the BSD license. That's what we're out under. I've compiled stuff on Linux from source. It was a couple of years back when I was actually smarter and stuff like that. But, yeah, I'm certainly able to do it at one time. Yeah, but I would run ZFS in Linux if it just came with CentOS. And that would be great, right? And so this whole thing about there's a conflict between the licenses. You can't distribute anything that's more restrictive along with something

that's less restrictive. Well, great. You guys have your religious argument, but all the people like me who want to use it are just screwed. So you guys do a lot with supercomputing and stuff like that. What's going on in the supercomputing end of the world? It's kind of a different... Howard and I are both kind of enterprise geeks. Yeah. And I would say it's one of our larger verticals, M&E Media and Entertainment is another one, HBC being the one that I actually, as an executive, I sponsor that vertical. So we just finished up SC2016 in mid-November. And it's kind of an interesting, definitely an interesting bit of customers. You talk to customers that are talking about a data rate of 700 terabytes a second.

700 terabytes a second? Where the hell do they put all that stuff? Exactly. Well, that'll generate an exabyte. If they keep it, it'll generate an exabyte if they keep it it'll it'll generate an exabyte in one year so that's coming off something called ska square kilometer array yeah yeah yeah the yeah in europe right it's well it's it's kind of worldwide we're not fully part of it but it's uh south africa australia cambridge university and a few others

that are kind of the three spearheads of it but you look at that that and you say in 2022 that when they want to do this, what are they going to do? So they're sitting down with vendors now and talking about how do they move it, how do they store it, how do they pare it down, what do they need to keep, all those pieces. And it's always interesting conversations. And that's a huge networking problem, let alone a storage problem. It's, you know, how many hundred gigabit Ethernet links do you need

to be able to carry 7 petabits per second? Exactly. So it's the same thing. I mean, you talk about there's another one going up on a mountain in Chile. It's a 60 petabyte per year system, and they're going to move that 60 petabytes back to the U.S. as a second copy. So think about the cable they're burying for that one. Well, this is where it's hard to beat the bandwidth of a truck rolling down the highway, as Amazon just showed us.

Yeah, and we do see that in some of the military applications, which is you can't, you know, you have so many airborne platforms flying around the world. They're collecting data 24 hours a day. They got massive WAMI sensors on them, which are multiple 4K sensors. You can't watch all that, right? But you want to keep a copy of it, so what do you do? And those are some of the problems that we're trying to solve. And sometimes the network can't solve it.

Sometimes the network can, sometimes it can't. I would say that tape and removable storage, though, as a sneaker net, we don't see it that common. We see it probably five or 10% of our customers would use it for that level of data movement. Right. And that's like offshore oil rigs, right? So Discovery Channel is a good example of they take a helicopter out to every boat, and they used to pull P2 flashes off. think they're now pulling lto tape off but same with median entertainment where they're doing a post house is going in and doing either color correction or taking out all the logos you know they'll send a copy of that movie on lto tape because lto and ltfs is one as the de facto standard in median entertainment

so yeah you see it you see this data interchange for tape. We see universities where, you know, a researcher will come in, and depending on the university, they'll own their research. And so at the end of their tenure or their end of their stint at that university, they can come back and say, I need my data. Well, when you talk about HPC data, that's a lot of data. So they give them, you know, a stack of 10 LTO tapes with LTFS on it. Yeah. Let's, let's talk for a minute about LTFS because, you know, I find that most of the people I talk to and I'm, and therefore assuming our listeners think of tape as, you know, what they dealt with so many years ago that it was problematic, you know, DLT and DAT primarily. And a tape was an indecipherable thing that had to be read by the same software that created it. So I'm a big fan of LTFS. So why don't you give us the nickel tour of what LTFS is and why we think it's a good idea? Sure. First, it's not a product. LTFS is a technology or a format.

It's very much like thinking about TAR. And since we're on greybeards tar would be the you know the basically the old tape archive system uh it's a known format it's a published format well ltfs is is very similar except it has a few advantages and the biggest one is the index partition so you take a tape it's partitioned there's a small partition which is the index partition a large partition which is the data partition and then what happens is I write all of my, basically like your inode table or your fat table of a file system, that data goes into the index partition, and then the bulk of the data goes into the data partition. What makes this valuable to the customer is no matter the size of the tape, no matter the size of a 10 terabyte

cartridge, a six terabyte cartridge, it doesn't matter. You load it up, you do the read of the index partition only, and you know the entire contents of that tape. The other thing is, is that LTFS is open source, so you can run it on Mac, PC, Linux, wherever you want to run it, you can run it. Yeah, I was told you can almost take an LTFS tape with a proper drive and plug it into a USB port on your Mac, and it's like a USB drive. You can, and that's part of that driver stack. So, you know, LTFS itself, it's like a USB drive. You can, and that's part of that driver stack. So LTFS itself is just a format. And then what they did is they used Fuse file system and user space, and they basically tied a Fuse driver to it so you could plug it in.

And it's pretty fantastic when you think about it. You can come from a 20-petabyte tape archive, take one tape out of it, plug it into a desktop, and you have a 6-terabyte USB key sitting there. You know, I think I want one of these. So Black Pearl is a front end to a tape library. Yep. That lets me write to it via S3, so it's not some low-level driver crap that I have to deal with,

like tape drives of old. Yep. But it stores the data in LTFS, so theoretically I can pull a tape out and send it to Ray and he can plug it into his Mac. Yep. Yeah, so that's the... If I only had the drive. I'm sure we can arrange something, Ray.

And Howard, there is one thing that's kind of interesting. We're using object storage technology in Black Pearl. So when you're putting data to us, you can be adding metadata, but it is an object-based interface. But when we go to LTFS, we kind of go back to file-based storage because LTFS is a file-based system. So you're just like Swift and you're storing an object store in file systems. And everybody, actually, if you look at almost every object store out there, the underlying storage system is a amalgamation in file systems. And everybody, actually, if you look at almost every object store out there, the underlying storage system is a amalgamation of file systems

usually. And so what we're doing there is we're putting those files out there, but we're also sending all that metadata that you gave us in the object store, we're writing that out there in XML. So if you were to pass this tape between you and Ray, Ray could actually... Wait, it's in the index partition? You're writing the... The XML metadata, yeah.

No shit. Excuse my French. So I will say, if you used a native LTFS driver, yeah, would you be able to read it? Absolutely. You'd have to know a little bit about the metadata that Ray put on it

or you put on it, but it is a transportable format, and it's pretty cool. From a customer value perspective, the other thing is that, again, if they for some reason don't like us, point somebody else at the archive and they could read the archive just as well. That's the other thing, that I'm going to live and die by my service and my product investment in R&D.

And so if I'm not doing well, they can always, you know, 10 petabytes, it takes a long time to move. But with LTFS, you just have a new system point at it. This is very interesting. I would never realize that the metadata was actually going to make its way all the way to the tape device itself. That's interesting. Yeah, we can trade a tape between Black Pearls and pull the metadata in. Just like today, we can replicate from Black Pearl to Black Pearl.

So you can have a Black Pearl in New Mexico and one in Colorado, and you could replicate those two. But if you pulled a tape in, we'd actually be able to read it and pull all that metadata in. And when the customer does the search of that, that's coming out flash, right? They're not going out to the tape library. The metadata is actually stored in flash. Right, from a Black Pearl management unit perspective, right? Yeah, so if I'm smart, I can write a digital asset management system that stored everything I'd ever search on in the metadata so that it's on the appliance. You call it a DAM? Is that what you asked?

Yeah, Digital Asset Manager or MAM, Media Asset Manager. Yeah, so I built one back in the day at BBDO. Maybe it's time to dust off that code and get it updated to S3 and stuff like that. It was very special purpose, and they own the code, unfortunately. I understand that. God, if it was only under an open source license, we'd have this problem solved. Yeah, so we stored all the metadata in SQL Server. Ah, yeah. And then could put the archive any place that was, at the time, accessible via SMB.

But now I would be using an S3 interface. Yeah, I think that's what makes – I think that you brought something up about DAMs and MAMs. You know, M&E Vertical, these guys love it because it gets rid of that HSM system, which is A, expensive and complex to use. And in media and entertainment and a lot of these other workflows, they don't need an HSM. They don't need an active archive because they know exactly what they're going to get. The system, the ERP system, the edit station is driving the push and pull of data. And if you're dealing with large enough objects, then the, the access time becomes difference becomes relatively small. Yeah.

Talk to us a little bit about what's going on in the LTO tape. I mean, I don't know, is it LTO six or seven now it's out available or. Yeah. Seven's out today. So tape wise, you know,

there's, I always use the term, I blindly stole this from IBM. It's kind of a tick tock. I mean, from Intel TikTok. And so with IBM tape, as Aaron Ogis says, the root of all tape is IBM. At least the root of all tape drive is IBM.

So when you look at tape, they developed their TS drive, their enterprise drive, and that's the tick. And then their tock is the LTO drive. So they build up the head, the chipset, and things like that. They put it into their enterprise drive, and then they commoditize that onto the LTO platform. And so today we're shipping 10 terabyte TS-1150 and 6 terabyte LTO-7, 360 megabyte a second on TS and 300 megabyte a second on LTO.

So that's each drive. What's coming, though, is kind of interesting is we're kind of in the heyday. I would say at least the heyday for tape heads. And the reason being is because if you go back to the mid-90s to early 2000s when the disk drive guys were benefiting from this year-over-year capacity doubling, that's the head technologies we're at in tape right now. So we're finishing up GMR, giant magneto-resistive heads, and we're going to TMR. And what you're going to see here shortly is the next generation TS drive, the 1155.

That should be coming out early to mid next year. That's a drive that uses current JD tape, the current tape that you buy for the 1150 drive, but it's a 15 terabyte cartridge all of a sudden. So voila, you have a whole bunch of tape, you put in an 1155 drive, and you get a 15 terabyte cartridge all of a sudden so voila you have a whole bunch of tape you put in 1155 drive and you get a 15 terabyte cartridge with your old media and then lto8 lto8 will be right on the heels of that i'm hoping for a later 2017 releases is our best guess and what we're hearing from analysts and that'll be a that'll be a 12 terabyte tape. Okay. So good stuff. Yeah. One of the, one of the stupidities I've seen with people with tape is an organization that says, well, when we get the LTO five drives,

we have to transfer all the data off the LTO four tapes to LTO five tapes. Oh, that'd be a bad idea. Well, yeah. Um, but I've seen, I have seen organizations that have that rule that all data must be on the latest tape because at some point they changed from 8mm to DLT and got rid of all the 8mm drives and couldn't read them on the DLTs anymore.

And they haven't changed their policies to say, well, you know, the tape drive will read two old generations. So it's every third generation we need to upgrade. Yeah, or keep the old drives in the system for even another generation. I have customers doing that often. Yeah, but eventually one would have to migrate data. Absolutely. When we first talked about Black Pearl, there were some data integrity, data migration features that hadn't yet been baked in that I thought were really important. Since it's LTFS, I should be able to take two old small tapes and merge them into one big new tape without having the front end dam or backup system know about it.

Are you guys doing that yet? Yeah. So I think it's in the release that's just coming out right now, which is a migration. And migration is basically going from one version to the next. And then the checksumming along with that, which is as you move data across from a client machine up to Black Pearl, that checksum is going to travel with that hunk of data, that file or that hunk of data. We want to validate that all the way through the system.

And on recovery or any time we do a migration or any time you ask us to, pull that data back up and validate it. Now, our tape libraries have always done a CRC check, and tape drives themselves have done CRC checking on the tape. So in the tape library itself, we can do data integrity verification where we actually read the entire tape and make sure all the CRCs are correct. And that's just basically doing a SCSI read and throwing the data on the floor. But it does validate that the data is there because the CRC check is validating.

So you sort of scrub the tape over time to make sure it's still readable and stuff like that? Yeah, and that's built into the tape library. And then also Black Pearl will do that, and it will pull the files back up, and it can actually read the files back up and then validate the checksums on it. And we'll do that both on migration and recall. You have multiple layers of protection. And the other thing is you have proof, somewhat providence, of the data that left your client is the data that's coming back off tape. Right, because it's an end-to-end checksum.

Exactly. So have you gotten smart enough that every time I write an object to S3, you write it to two tapes and do the CRC check every six months, and if one fails, you make a copy from the one that survived? So I don't have the auto-remirror yet, but I do report to you that I've lost one of your copies and you need to start up a remirror yet but um i do report to you that i've lost one of your copies and you need to start up a remirror on that system but uh auto remirror is certainly something on our list it's one of those

a longer list than we have developers right now yeah of course you know that you know i'm going through my you know what what do i actually want for 50 year retention because it seems like every six or nine months i get an email from some pr firm that's pitching some new storage medium that you know it'll last 100 years and while that's fine for like the library congress where i want them to have you know the last copy after the nuclear apocalypse for my near line even 10 years from now I will be able to buy technology that's eight times denser than my current technology. And my data center floor space is going to be so expensive that I need to migrate just because I can put data in an eighth as much space. So you're not concerned about the storage capacity per se as much as you're concerned about the cost of the space.

I'm going to migrate every 10 years because of technologies, technology drivers and business drivers outside. Can I still read that? Yeah. And I think you pick your date, right? It could be 10 years, it could be 12, but you pick it. And the reason I would say that is I have a pdp 1134 what running your winery what are you doing with a pdp 1134 i don't think i can boot it anymore because i don't remember how yeah yeah yeah and that's where close to is who can who can run that pdp 11 for me and so the idea is that you've encapsulated your data in a digital format even though it may started out as digital data it's still re-encaps data in a digital format, even though it may have started out as digital data. It's still re-encapsulated in a digital format, and you need to always be migrating that format forward. It goes to one other thing, and you were bringing this up, 8-millimeter tape, 4-millimeter tape.

I would tell you that I preach this to more and more people, which is backup is not archive, but archive can be backup. And what I mean by that is when you take your backup tapes or backup sets and store them for five years or 10 years, there is no system in a backup environment that says, oh, you need to migrate this data to a newer technology. That doesn't exist in backup systems. But when you think about archive, archives are almost all built with the idea of moving technology forward. And so you can make two or three copies in an archive of data that's not changing, reduce your primary backup set, and have a backup copy in the archive. before something happened. And the time horizon on that is 90 days or less. That if it's more than 90 days since you made this copy of the data, it is highly unlikely that you're going to have to restore it to its previous use. You may need to access it for some other purpose.

That would be court litigation, those sorts of things. Right, but that's not putting it back. So backup systems index their data based on where it came from so I can put it back. And they only keep that catalog like 90 days. I mean, if you look at NetBackup, it doesn't keep that database of everything you backed up for 20 years because the SQL server gets too big. Yeah, exactly. Geez, I've keeping my backup up for 20 years because the SQL server gets too big. Yeah, exactly.

Jeez, I've keeping my backup index for 10 years now. I've not gotten rid of it at all. But, Ray, then yours is of relatively modest size. You know, if you look at, I have a Mac here and I know one of you guys, I look at Time Machine and I look at that as kind of as much of an archive as it is a backup. But if you've ever tried to recover off Time Machine from a disaster, it's a four or five day recovery.

I do a lot recovering files from Time Machine all the time. But I've never had a complete disaster where I had to recover from. That's a different issue. Yeah, I decided I should get the backup of my file server off-site and got a 15-day evaluation of CloudBerry.

But the backup didn't complete within the 15 days because I live someplace where I can't get 100 meg internet connections. So you've mentioned the tapehead stuff. I've always been of the impression that tapehead's 10, 15, 20, 25 years behind disc technology. But there's always the next 25 years disc is going to run out of any technology that's going to be able to increase the density. So that would mean sometime 25 years after that, let's say, tape would run out. Do you foresee any point in the future where density on tape and or disc would stop to increase you know i can't i wish i could stay out 25 years that that's where it's going to be but i'm pretty sure hammer isn't going to work for tape though yeah i don't think so either a bit

pattern media is you know again the disc drive guys have a massive technical wall in front of them right now and that's why you're seeing some of the problems in the density jumps and the layoffs and things like that. What I would say, though, is when you look at tape, you know, tape is really – somebody used the term 3D storage, right? So the real question is, how do I make my cartridge bigger? How do I make my reel bigger? Where when I look at a disk drive, I could physically make it taller, but that's just taking the room of two disk drives. But a cartridge, a physical cartridge, I can keep wrapping the bands around that cartridge. I can make the media thinner and things like that. Is there an end to tape?

There's certainly possibly an end to tape. But there's always, always, always a place in the storage business for the fastest storage and the least expensive storage. And almost always, the least expensive storage has a deficit of some kind. So I use the term cotton candy. If I could store data in cotton candy and it was cheap enough, it would be the replacement for tape. But today, we can't do that. So to me, there'll always be some media out there, be it tape or something else that has low cost, but usually has a deficit over the high speed systems that are available today. Yeah.

And this is the problem with the disk drive guys. They're stuck between flash being the fastest and tape being the cheapest. I once worked for a guy that always said that everybody wants their data in memory, but they want to store it on tape. Well, they want to store it in the cheapest possible location, right, or something like that. Right. Well, I mean, that's the whole concept of flash and trash. I wouldn't call tape trash.

That was not an unsigned word, Howard. Well, I mean, you know, tape is what it is. It's a high bandwidth, high latency, cheap storage medium. And the more data you have and the less IO frequency you have to that data, the better it looks. It's not today a good primary backup medium. Trying to get a backup stream to keep up with an LTO7 tape drive is really difficult. But it's a great place to put your archive data. Ray, we've talked about this before, which is the model for the large

data user is more of a warehouse than a factory floor, where his data is stored in this archive repository, and the database guy stole the word data warehouse, but it's a warehouse where I store my data. And then when I need to work on my data, I bring it to the factory floor. My factory floor happens to be a flash or scratch file system. Those systems are the ones that we're seeing being deployed more and more where you go into HBC, you go into media and entertainment, you go into bioinformatics, any of these guys that are generating, you know, petabytes of data, and they really have to go to that ERP-based enterprise resource planning model

where somebody knows what data or most of the data that's going to be used tomorrow, so they're pulling it into the scratch file system already. Well, it's funny. Those particular markets look like old-fashioned batch. It's like we're going to be doing the special editors cut of captain America, civil wars next week. So let's pull all of that footage back up into the spinning disc system so we can edit it next week and be ready for it.

Yep. But it's the same. I mean, if you look at some of the satellite imagery guys, you know, a world event happens. They'll, they'll most likely have the most recent imagery online. And what they need to do is just go geographically and go find, you know, a world event happens, they'll most likely have the most recent

imagery online. And what they need to do is just go geographically and go find, you know, the last two years or three years of that imagery. And that can come up within, you know, if you think about the imagery of the entire world and how many copies of that we have stored in certain locations, I may have a satellite image from every day over, you know, my house in Colorado. And if I could see what I want to look at every day? No, I need to see every month maybe. Yeah, and when the nuclear plant explodes, then, you know, I need to find the picture,

the Landsat photo of Chernobyl before the explosion. But I don't need it till 6.30 because the news doesn't come on till 7. Yeah, well, Colorado had a landslide, and so they went back and looked at the satellite photos and looked for what they believed caused it, and it was a clear cut. So they had allowed a lumber company to go and do a clear cut. No kidding. Do a clear cut, and basically within two or three weeks of this clear cut, rain came down that clear cut, and that's what caused the damage.

So it was kind of interesting that they— Oh, the lawyers are going to have fun with that the lawyers are gonna have fun with that well yeah well that's interesting yeah i think the the thing about the data that we're seeing out there it's just amazing you know usgs is uh now starting to scan all of their old 1950s imagery so you know today if you can go on google maps or google earth and you can kind of rewind time and you can see the the same photo from a long time ago or the same location from a long time ago. Well, when the USGS gets all their stuff scanned, that'll be really cool because you'll see stuff back to the 50s and 40s where they went over with airplanes instead of satellites. It'll be good to see, look at places like Phoenix. Oh, God.

Las Vegas. Think about that one. Oh, yeah. Denver. Las Vegas. Think about that one. Oh, yeah. Denver. Denver's grown considerably. Yeah, but places like Vegas and Phoenix were unlivable before air conditioning. And so, you know, post-World War II, they go from being podunks to major cities. Yep.

So you guys do a lot with, like, you mentioned telescopes. I mean, are you guys involved with CERN and some of the other national labs and stuff like that? I think we're in six of the top ten labs in the U.S., so they're certainly a customer of ours. And, Matt, you come to New Mexico and you never visit. There is that place. What's it called? Los Alamos? Los Alamos.

Yeah, Los Alamos National Labs. Thank you. Just down the road. Thank you, Howard. You know, Howard, it's funny. Nate, our CEO, has an airplane, and it can fly a long ways, but he likes to fly to New Mexico a lot. So those are the only two labs in the U.S. that – well, three. He actually – NCAR is the other lab that he Mexico a lot. So those are the only two labs in the U S that, uh, well,

three, he actually, uh, and cars, the other lab that he visits a lot. Um, so he's kind of picked those off my list as his, as his lab.

So next time he's down there, I'll let him know that, that you guys need to go out to dinner or chat. Yeah. I go down there once every two or three years. He goes down there once every six to nine months. Right. Well,

yeah, but if, if he's flying himself, he can fly right into Los Alamos. Exactly. Yeah, exactly. So those guys, so Ray, to answer your question,

those guys, yeah, they're large tape users. Argon has some big tape subsystems. Argon also is using Black Pearl, and it's kind of one of those clients is called Globus. I know Globus. I know DataGrid.

Okay, so GridFTP is the engine below it. And it's kind of a cool way to share data in an HPC environment because they can use the high-speed networks that DoD and DoE own. Even though they're talking over authenticating over a public Ethernet, that system can actually move data over some really broadband private networks. I did a paper, it must have been almost 10 years ago, on Globus and the data grid and stuff like that. And I ended up talking to the guys at NCAR that were using it quite a lot at the time. Again, they've got the high-speed networking capabilities to move the data around in massive

quantities. Yeah, so now those guys have gone to the university base where they do, you know, publications. So if I wrote a paper on, you know, let's call it erasure coding and a new system for that, and I want peer review, I can post it on Globus to the six people I want to review it, and that Globus system will allow them to pull my paper and all the background data and stuff like that. So kind of a cool system that these HPC guys are using for moving data around. So for like PLOS publication or something like that,

you would put the data and the paper over on Globus. So one of the rules now is you have to post your algorithm, your data, and all these other things to really get your paper reviewed and approved. So if you think about some scientific models for weather, if that's what your paper was written on, you'd have to provide all that weather modeling data, which could be big data sets. So Globus works well to move that around. You live in an interesting world, man.

It's very specific as far as the customers we talk to. It's the guys that have a data problem that most people don't run into or at least haven't run into yet for another 10 years. It's funny how as tape fell off in the enterprise for backup, it's just still exactly the right solution for these people who have just enormous data sets. Yeah, yeah. It's kind of funny because everybody, you know, we hear cloud and everybody's going to cloud. And I always say disk drives, switches, network cables, servers, CPUs, all those pieces, including tape libraries. The cloud vendors used all of that hardware. And so the value of cloud is when you're aggregating data or moving compute, you know, and moving your entire application up to the cloud. But when it comes to petabytes, you are the cloud.

Yeah, you're your own cloud to some extent. But you guys also do work with hyperscaler kinds of customers, right? I mean, you don't have to name anybody, but these guys also use tape libraries and big active users of these sorts of things, aren't they? They are. Yeah, and I can't say their names, but if you searched their names and searched our name, you would probably find them together. A couple years back, I don't know, it was Gmail or something like that had a failure and they lost some data. And all of a sudden, they found all the data. It was on backup tapes.

Rumor was it was on backup tapes. Absolutely. No, no. The guys from Google were quoted. They said it was tape. And it took them a couple weeks. I mean, it was a massive loss. And it took them a couple weeks to, I mean, it was a massive loss, and it took them a couple weeks to get all that back.

It does show that, you know, even something like that does backup and does, you know, non-rewritable or immutable data sets. Well, God, with all the data that's going to the cloud nowadays, how could you not do it? You couldn't afford to do it any other way, really. Yeah. Well, you know, for services like S3, they replicate the crap out of it and then say, well, that's the SLA. Yeah, and they've replicated zeros a few times, which is the value of the error gap on tape, which is usually, you know, to get to tape, it's a different mechanism than just open, write, close. Yes. Yeah, well, I mean nowadays we're hearing a lot about ransomware and how this senior executive got phished and ransomware and not only did it encrypt his laptop but put it in the entire file server and all the backups that are stored on that other file server. That just demonstrates the, no, no, no, your last resort backup media has to be completely inaccessible from everything but the process that makes the backup. Agreed. I mean, and that's the

same with, I would say with, you know, what we have archived a lot of archive customers, Imperial War Museum in the UK is one, and the assets he's protecting, they consider them, you know, some are, you know know things for the world for example one of the projects they're working on is a is a copy of all the nato archives nato korea archives so the korean war nato has come in and said we'd like these curated these are radio broadcasts not like radio broadcast military but radio broadcasts like announcements uh video and things like that they want to curate that for you the rest of time. And so he has three copies of that data. One of them is in Scotland, completely off-site. So he makes a tape locally in Duxford

and makes a copy on disk and makes a second copy on LTO, and one of the tapes goes to Scotland. It's stored there in a vault. So you have to get it offline, air-gapped, and you have to get it away, is the other thing. How many other challenges, of course, is that the formats of these things will change over time? You have to have some way of identifying the format and being able to extract that format. You look at a JPEG file today, in 20 years, this will probably be JPEG 27 or something like that. And it may or may not be compatible.

Yeah, and I would say that one of the rules that we, you know, NASA Ames is a good example. 30% of their resources are dedicated to migrating forward. So they're always in migration. They have a massive amount of data that they're moving forward, but they're always migrating that data forward. So it's the same kind of thing that when you think about this, you've encapsulated your data in a digital format, be it JPEG or be it a tape format, but it's still encapsulated. You have to keep that encapsulation, not only that encapsulation correct, you have to move the OS forward, you have to move the application forward. And if you looked at some of the tenets of archiving that I write up, it really is the, you know, it's not just the ones and zeros that you stored. It's the operating system, it's the application, all those pieces

have to be every so often looked at to say, do I have to migrate this forward? Yeah, and we have to look at formats like PDFA and XML that, you know, they may be less dense, but at least they're designed for long-term access. Yeah. Well, you know, somebody go open a D- base file for me. I'd like to see that. Right. Yeah. Yeah.

Or, or, or word perfect or something like that would be my, you know, I've still got word perfect files floating around. Well, word 2013 will still open a word perfect 4.1 file. Yeah.

I don't know if I was at 4.1. So it goes, so it goes back that far. Um, but, but I did when I was working at U.S. Tobacco, end up running into things like Multimate files. Wow. And they were, you know, and I asked them, you know, can anybody open these?

They say no. So, well, then why are we keeping them? Yeah, so we can send them in response to subpoena and make it the other guy's problem. Yeah. Yeah, best effort. That's a terrible legal thing. Best effort. a guy's problem yeah yeah best effort that's a terrible legal thing best effort well i mean u.s u.s tobacco was at that by that point run by the lawyers they they had been sued so frequently and

their philosophy was we keep everything forever because all of our sins have already been exposed yeah good point and the last thing we want is to be the guy who goes oh yeah that 20 year old document we threw it away and have to explain it to yet another judge so matt you guys do like nas devices as well don't you yeah and again targeted towards archives so uh you know deep storage um we use both uh sas drives and smr shingled magnetic recording and it's kind of interesting i think shingled magnetic recording. And it's kind of interesting. I think shingled magnetic recording, when I watched those first come out, this is where you kind of lay the bits closer together, almost basically track on top of track.

Those drives are really targeted towards archiving. And when you put them in a normal system and try to operate them in a normal system, bad things happen. Yeah. Well, we get them. You know, we're primarily a tape library company we write a lot of tape software our black pearl knew how to talk to tape when we put it in our nas system and our black pearl system we put it behind black pearl it was called arctic blue um it you

know we treated it more or less like tape on the arctic blue side it worked great when we treated it under nfs and sifts uh it fell down pretty quickly the nice thing for us was Seagate's just up the road, so we were able to work with them. But it is an interesting technology for archive because if you need your data accessible within less than a second, that SMR technology, if it's a write once, read maybe, is some pretty good stuff for that. Yeah, those drives read just as fast as ordinary drives. The problem is that you have to write a whole zone at a time, and that zone is multiple gigabytes.

Yeah, yeah, yeah. You lay down a slab and that's it. Yeah. So I'm looking forward to when you get the version that just has a log structure bigger than a zone. It's like, yeah, we use 50 gigabytes of memory to store the log, and then we write a RAID stripe of 30 gigabytes to all these drives. Yeah, the analogy that – so I have a company we're working with in Australia. They're a broadcast company, and the way Black Pearl works is it'll pull a chunk of data back.

Even if you ask for one piece of the file, you ask for one byte, I pull more than one byte back inside of Black Pearl because there's no reason to pull one byte off tape. Now, it's so much work finding that one byte, you might as well take all of its neighbors with you. Exactly. So I think right now on LTO7, I pull back 64 gigabytes, which takes a couple of minutes to pull back. But it was interesting that this customer, when they asked, they said, oh, well, can you tune that uh this customer when they asked they said oh well can you tune that down a little bit and i said well what do you want me to tune it down to and the tape read time would be a half a second and he goes that will give me three to five minutes

of my clip in other words the bit rate and the data that they're that they're using because it's such old stuff is so small it's ridiculous in in today's tape world. And that goes to the same thing with the things you're talking about, which is IOP disk and random access disk, that's totally different than SMR, and I think they're going to have a tough time unless they put a bunch of flash in there to make SMR handle well. So the world seems to be going to all flash storage at the top end, and hybrid storage for tier 2 and Tier 3 kinds of things, and then obviously tape and archive solutions beyond that.

Do you think the all-flash thing is a flash in the pan? I'm a BLSI engineer, so that's what I came out of school for. I don't think so. I think that, again, there's always going to be a place for that high-end memory. Will it be flash or will it be… 3D cross-point and all that other stuff. Exactly. Yeah, yeah.

But the real question is, will it be solid state? And my assumption is for the next five years, ten years, it's going to be solid state. But then you get into, could you do things in optical and other things? Because those guys are down to the atoms. Don't go there, Matt. You're going to end up at holographic storage. At this speed? At this speed?

Optical? And holographic storage has been five years away for the past 20 years. I was going to say the same thing. And no, what I was talking about is really the idea of racetrack memory and other pieces where could you do that same thing if you look at a sas cable today today's 12 gigabit sas cable i want to say there's 14 bits in flight on a one meter sas cable oh yeah yeah i understand right yeah one nanosecond per foot that's all you get yeah exactly more or less that's the law so yes some interesting things i think when you think about flashing and where that technology is going to go. There's an interesting paper by Dr. Fontana.

I think it's David Fontana out of IBM. He wrote the square inches of storage. Basically, he takes every technology, flash, RAM, disk, and tape, and he basically talks about how much it costs to build a square inch of storage and how much that square inch stores. And then he goes through how much it costs. And the big problem with flash right now is, of course, fab. Until I can bring more gigafabs online, I can't solve the disk drive problem. They can't beat out the disk drive guys yet until they get more fabs online.

At $6 billion a pop or $16 billion a pop, something like that? Yeah, in a $50 billion business, it's going to take you a while to recover that money. Then we get back into – and Crider's Law says that disk drive guys are going to find something. They'll make it to 25 gig. Whether they get substantially past that, I don't know. Yeah, I't think there's any any doubt in my mind that they'll get you know they're already got 10 gig right 12 12 and 16 are certainly in the labs working the real question i think though howard goes to some of the guys said earlier which is they've lost on the top end to flash they've lost on the low end the the small form factor drive to flash they're certainly

losing the low end of low and slow cheap storage to take tape archive so the question is is profit will they be able to make enough profit to sustain that r&d budget right the problem the problem the hard drive from where i sit the problem the hard drive guys face and the part that makes me really wonder why toshiba stay in the business is things like hammer and bit pattern media are going to require enormous fab outlays. I mean, bit pattern media is like making semiconductors and the fab costs are going to be similar. Yeah. Five picosecond clock, by the way. A stable five picosecond clock to make that bit pattern media work.

A stable five picosecond clock to make that bit pattern media work. A stable 5 picosecond clock? And they have to amortize that cost over a decreasing market because the flash guys are eating them up. Yep. A disk drive costs $50. Regardless of capacity, the spindle motor, the casting, all that stuff is MSRP, $50. And when 100 gig SSD is $50, well, 100 gig is big enough for half the laptops. And I'll pay $80 for the 200 gig SSD, and that's big enough for most people. So the number of drives, you know, the disk drives are becoming like tape did 10 years ago. It's, you know, the only ones that make sense are 4 gigabyte and bigger 7200 RPM and slower.

Yep, yep. And that's not the profitable side of the market. Which means that as Seagate and Western Digital shift more of their income into those products, they have to increase the margins of those products. So the rate of dollars per gigabyte falling starts to slow. And it's at the point where Flash is available and three times as much as spinning disk, the market for spinning disk starts to become really questionable. Yep.

Agreed. Agreed. You know, and here's the thing, is you're spot on there, which is, you know, I just, my little Mac here, it's old, it's a 2011 Mac. I put a one terabyte flash drive in there,

I think it was 600 bucks. This machine is like new. And so, you know, would I have gone and bought a four terabyte drive for this machine? Not at the cost of what flash is for that one machine. But I can't do that for petabytes.

And that's the thing that's going to happen is that that middle market for disk is really getting squeezed down now because the bottom's gone, the top's gone. Now the middle's being squeezed. And now C8 and to a smaller extent, WD, because they're more heavily into Flash, they have to run the company on the income from those remaining products. Yeah. Yeah. Here's the question though.

Is it valuable to continue? Is a 16 terabyte drive enough for the next five years, 10 years? No, it's not. From a disk drive perspective, no. Okay. So, so you have to get to 25, right? Yeah. 32.

I'm thinking. But to Howard's point, to do do that i'm going to put a laser in there that laser is first it's a wearout component um it's the first part i got a heat problem and then i have a cost problem so all of these things add up to being that that that cost of goods on that that that case and spindle and platters becomes instead of 50 bucks the cost on that is now 75 or 80 and the m MTBF has gone through the floor. There's a lot of things in a disk drive that have wear-out problems, besides the laser. I mean, so, yeah, I understand that.

Yes, it's new technology. Can they do it? I think they can. It's just a question of, is the market there? Can they do it at a cheap enough price where they can sell enough to make that R&D investment, that FAB investment pay out? That's the question. It's almost like coal.

There's a point in time where it's cheap enough and it's not as bad to run it forever. But at some point, you decide, well, it's more expensive to extract and it's got environmental problems. From an economical and ecological perspective, coal is not the way to go today. Shell gas is cheaper. Shell gas is cheaper, yeah.

Alright. Listen, gents, we've kind of run out of time here. Is there anything you'd like to say to our audience, Matt? No, I truly enjoyed the time. I think our conversation was certainly diverse. Well-ranged, wide-ranging, I guess. So I did certainly enjoy it. But now I get to get outside and enjoy the very cold weather in Colorado today.

I think we're all of zero. We got you beat. It's 25 degrees here in santa fe that 400 miles south matters it's cold here in denver cold cold and snowy so how was there any last questions for matt no i think we got it covered i you know we'll be talking about royalties on deep storage soon let me know i we my my, it's just next door to me. Oh, good. Nothing I like better than talking to attorneys. Yeah, there you go. Well, this has been great.

It's been a pleasure to have Matt with us on our podcast. Next month, we'll talk to another startup storage technology person. Any questions you want us to ask, please let us know. That's it for now. Bye, Howard. Bye, Ray. Until next time, thanks again, Matt. Thank you.