Grey Beards on Systems - 41: Greybeards talk time shifting storage with Jacob Cherian, VP Product Management and Strategy, Reduxio

Episode Date: February 17, 2017

In this episode, we talk with Jacob Cherian (@JacCherian),  VP of Product Management and Product Strategy at Reduxio. They have a produced a unique product that merges some characteristics of CDP storage and the best of hybrid and deduplicating storage today into a new primary storage system. We first saw Reduxio at VMworld a couple … Continue reading "41: Greybeards talk time shifting storage with Jacob Cherian, VP Product Management and Strategy, Reduxio"

Hey everybody, Ray Lucchese here with Howard Marks here. Welcome to the next episode of Greybeards on Storage, a monthly podcast show where we get Greybeards storage and system bloggers to talk with storage and system vendors to discuss upcoming products, technologies, and trends affecting the data center today. This is our 41st episode of Graybridge and Storage, which was recorded on February 13, 2017. We have with us here today Jacob Cherian, VP of Product Management and Product Strategy at Reduxio. So Jacob, why don't you tell us a little bit about yourself and your company? I'm Jacob Cherian. I'm the Vice President of Product Strategy for Reduxio Systems.

I started my career in storage at Dell. I'm now based in Tel Aviv, Israel. I moved to Israel after Dell acquired the company Exinet. I was part of the team that selected that technology for Dell to partner with and eventually led to the acquisition. Today, that is Dell's FluidFS. About two years into my expat assignment in Israel, I decided to go back home, and that's when I met the founders of Reduxio, who happened to be the founders of Exonet also. I really love what they had to say about technology and what we could do with the

technology, and that's how I ended up staying on in Israel to become the vice president of product strategy at Reduxio. You know, Jacob, you're the first person I've ever spoken to who was at Dell Storage that didn't come via an acquisition. And so I'm very impressed. All right, so Reduxio is you and the excellent technical team, but what's the secret sauce? Why do I want to buy Reduxio? What makes you so special? If you think of stores, architectures, and I'll start from the architecture pictures, because I think it's always important to understand why certain architectures exist.

And, you know, the genesis of architecture tells us why products are the way they are. And if you look at the architecture of many of the storage systems that are in the market today, those architectures are a throwback to the time when storage was captive to a server. So storage was really a device. Storage was a volume, which was actually a device inside a server. Now, we have moved from that model to shared storage, shared storage between multiple hosts, different models of deployment and storage. But the fundamental principle that the volume as a place where data is stored or a container of storage, that model has persisted through all the architectures that have come in products since the day when a volume was a drive inside a cell phone. When Reduxio looked at the problem, we found out that that architectural principle was too limited. And I think that's the innovation of Reduxio.

That's what allows Reduxio to do what it can do. Doesn't that argument lead us to object storage as well? It's hard to look at a CleverSafe or an AmpliData and go, oh, look, that's a volume. Yeah, I guess buckets and stuff like that, which might be considered something akin to a volume. So I will set aside the concept of fixed content storage, which is basically targeted at a different use case. We're focused on the primary storage market and high-performance transactional storage. The architectural model has stayed pretty constant since the days of the disk drive. And that architectural model is basically the definition of volume as a container

where you have some structure that maps logical locations that are accessed by a host through pointers to physical locations of drive. And the number of indirections that might happen when you go from a logical location to a physical location might be different in different architectures, but the basic concept remains that you're basically mapping a logical location to a physical location. And that's what we set out to change,

because that basically imposes constraints on what you can do with the storage system and what the storage system needs to provide in today's data center, which is really, in our opinion, manage and protect data. What does the, I'll call it the redirection architecture of Reduxio kind of look like at a high level? So Reduxio does not stand for redirection architecture. Reduxio stands for Redux IO. You can think of Reduxio's core metadata architecture as an IO recorder.

When a command comes into our system, we basically look at the associated metadata for that command, which is really where the data is going to, which is a volume and the offset. And we take that data and offset, and we put that in a special tag store, which I'll come back to later. We take the data, we chunk the data into smaller pieces that are manageable for us on the backside. We compute a unique name for the data and then we associate the tags with the data. There is one additional thing that we keep for every piece of data that is received on the system. We also store the time associated with that data, which is when did we see that data coming into the system. What this essentially does is we basically have a tag store, which is organized for quick lookup, and we have another layer that manages the data blocks. And the interface between the tag store

and the data blocks is a name for the piece of the data. And it's very simple to compute name for data today, which is create a secure hash on the data. So the data on the back end, like in a solid fire or an extreme IO, is essentially content addressable, that you're storing database. Yeah, so using the hash as a name for the data, and you've got another tag store which has the other metadata associated with that data, and the tying thing that ties the two together is actually a hash on the data, which is effectively the content, is a mechanism used to hash the name.

Correct. Hash the data itself. Right. So similar to how SolidFire or ExtremeIO stores their actual data. But sounds like your metadata structures are very different because you've got that time factor. Correct. So if you think of a dedupe system, right, most dedupe systems have to do a hash on the data and then basically associate data to hash. What makes Redux unique is not so much the fact that we hash the data, it's the fact that we have a tag store that basically manages the logical view of the data, which also contains the timestamp for when we receive the data. Now, given that information, given the fact that we have the tags that are separately managed from the data,

we have the ability to do a few things which basically makes Redux unique today. The first thing is, given that we're storing time of data, our system does not have the concept of snapshots. Instead, what we're able to do in our system is we can go back in time to any point in the past and recover data at the granularity of one second. Okay, we've heard this before. When I first started covering storage full-time, we had several companies pitching CDB systems systems but you seem to have found the secret sauce the problem with all those systems is they were keeping the data and journaling it and it ended up being huge amounts of data and recovery was slow because the data was organized in that journal, but you're doing it all in metadata. Correct.

Clever. So I think, as you pointed out, Howard, the issue with CDP systems were twofold. The first one is CDP focused on trying to be an automated backup, which is the data was stored on a separate device. So you have the issue that to get very fine granularity, you have to have the second device that could actually keep up with the change rate of data. By moving that capability into the primary storage system, we've eliminated that problem. We're giving you that very fine-grained recovery without having to deploy a second storage system. And plus, since this is inherent to the metadata

architecture of a system, you're not paying any additional cost in terms of performance or recovery granularity, which you suffered with CDP-based systems. From my perspective, what's unusual with this architecture is your tag store. It's almost as if it's a structured database type of solution. Most of the solutions we're familiar with, log structure, volumes, and that sort of stuff, have a fairly, I'll say it, you know, a fairly structured two or three or four tier pointer structure that gets from an offset volume to a physical address. You seem to have something that's a little bit more sophisticated.

Can you talk a little bit more about that? So, you know, when Howard talked about CDP, in most CDP systems, the method that was used to recover data at very fine granularity was to keep a journal of data. And the problem with journaling is the recovery time depends really upon the depth of the journal. How far into the journal do you want to do the recovery? Yeah, and since the journal is sequential and we're trying to do random access to the data, the rollback sometimes took a long time. Exactly, exactly. So if you think of backdating, which is this one second recovery capability we have, we're able to recover data instantaneously to any point in the past. So, you know, most people when we do a demo of our system, they're amazed that recovery of volume to any point in the past takes less than a second.

And that is because of the way the metadata is structured. Our metadata is a hash key store. So the tags are basically stored in a proprietary hash key store where what we do when we're backdating is essentially looking up data for the timestamp that you've requested. So if you think of most systems where a volume is a construct which manages data in our system, a volume is just a query into that tag key store with the timestamp attached to it. And a volume now is just the timestamp on the query is the timestamp for now, which is the latest time always. And if you want to create, if you want to get access to data to any point in the past or state of the volume at some point in the past, we just create a query with that timestamp attached. The offset comes from the command that requests the data.

So we're basically recreating the volume of the fly. So for us, creating a clone has no metadata operation other than creating this predefined query in our system. So when I access a block in a retrospective view to last Tuesday, essentially I'm sending a query that says, give me this block where the timestamp is less than last Tuesday. Exactly, exactly. So there is no difference in our system between a clone or the volume now.

Basically, the internal representation is exactly the same. And the data, there's an index on the timestamp, so there's essentially no performance penalty either. There is no performance penalty on the lookup. The way we ensure that there's no performance penalty is inherent to the hash key store. That's, again, as I said, that's something proprietary, but that's actually inherent to the design of the hash key store. That's, again, as I said, that's something proprietary, but that's actually inherent to the design of the hash key store so that we basically pay very little performance penalty in any lookup to the core metadata structures.

But most IOs come in with, you know, volume and offset. They typically don't say, I want it as of timestamp, you know, last Tuesday. I mean... Yeah, so the default is no filter, give me latest. Yes. Yeah, but how would you add a timestamp to a SCSIO command or something like that? I mean, yeah, so that the default is no filter. Give me latest. Yes. Yeah. But how would you add a timestamp to a SCSIO command or something like that?

I mean, are you actually coming in with a SCSIO command that says, give me something from last Tuesday at this offset in this volume? Or is it are you creating like a I'll call a view of a volume from from last Tuesday and then going after that volume? OK, so so when we when we talked about recovery, right, we basically said we can recover data to any point in the past. Essentially, we can roll back a volume. Now, exactly, going back to what you said, SCSI does not allow you to actually put a timestamp on a command. So the method that is used for recovery that most customers are familiar with for any volume is to either do a clone or an in-place revert.

So the recovery operations that we support are exactly what you would find in a snapshot-based system. Now, how we do it is completely different. In snapshots, to create that view to the data, you basically essentially have to create a copy of the metadata. For us, when we create a clone, all we're doing is creating a query with the timestamp and the read command from the host gives us the volume and the offset information. Because on every write, you're creating the necessary metadata. Exactly.

It's every time a new data block is stored. Very cool. Now, a secondary property of what we do is, because we're taking a hash every time so that we can manage the data block separate from the tag data, we know when we have duplicate data. So we automatically get dedupe. We didn't have to go create dedupe that is layered on top of a metadata. Dedupe is inherent to our system. Okay. So now the problem with keeping every block at every point in time

is we end up with an awful lot of data. It's every second. It's not every point in time is we end up with an awful lot of data. It's every second. It's not every point in time. So it's... Gosh, I don't know. We'll allow Jacob to clarify, but Howard is right. They're keeping the metadata as they

write. The point in time, the per second is just the views you can create from a query perspective, because the filters don't go down to the millisecond. Yeah, Howard is right. So for us to give you one second recovery forever would mean that we would have to have infinite amount of metadata and infinite amount of data. That's just not practical. So what we do is when you create a volume by default, you get a default policy on a volume that basically manages how we keep this historical data, which is this data that we have kept around for providing you backdating. The default policy

on a volume gives you about eight hours of one second recovery. And after that, we thin the data down to every hour, every day, every week, and every month up to a maximum retention period. These policies are just basically something that we came up with. Customers can go and change the policy. If you change the policy, reduce any of these time windows, then the policy takes effect immediately and the system either starts keeping more historical information or less historical information. And there's a garbage collection process that comes around and figures out what's old data and recovers the space. Exactly. Now, this is also why, you know, when we get a question, we get this question quite a bit, which is what is a performance penalty of backdating? And our answer is there is no performance penalty of backdating? And our answer is there is no performance penalty of backdating

because keeping one second recovery is the normal operation of the system. Now, we do have some performance penalty in the garbage collection process, which is when the system has to free up resources. We have to use resources to actually clean up data. Now, the way we prevent this from affecting IO is that we basically are proactive on the garbage collection process. So as data expires, we basically go in and clear data. We also know, as we approach the policy, what data needs to be freed up

and what data might live together so we can get efficiencies in being able to free up blocks in large chunks instead of actually trying to clear up blocks one by one. Okay, and I also see that you're storing a bit more data than a typical system would, because one block being rapidly updated would just hit their NVRAM and be replaced 27 times,

and you have to store all those values. Yeah, the question becomes, is it at D-stage that this takes place, or is it in cache, every time an update occurs to a cache value? To describe the flow of IO in our system, so when data comes into the system, we basically extract the data from the command. We have the command context, which is basically the volume and the offset, and we have the data. Now, we chunk that data, and we compute hashes on the data. We then apply our dedupe process, and then we compress the data, and then we put that data in memory cache, which is our NVRAM cache.

In parallel, the metadata is also updated with the context of the data. Okay, so now what we end up with is dedup and compressed data sitting in memory cache. Now that data is the data across all volumes because in the cache, when we place the data other than the tag store, that data has no context of what volume it belongs to. So our cache is not volume organized. It's basically a cache of data.

Now, we do provide volume context in the cache. So to do effective caching, we need to have volume context. But because we're doing this as data gets ingested, we get the benefit of global dedupe across all volumes in the system. So our cache is extremely efficient. I think the problem comes, and Howard hit it on the head, is that if a particular block is being rapidly updated, it's not unusual in these environments to see something like that.

You're going to have potentially a different data load for every one of those block updates, which is going to have to be somehow placed and compressed, de-duped and cached, and ultimately de-staged one after another. Yeah, but it's an outcome of the architecture. Yes. But it's not a penalty for the CDP-ishness in that, you know, you see it all the time. And so that just comes down to when you run your POC, is it fast enough or not fast enough?

Because it's happening all the time, I agree. Because it's happening all the time, yeah. And so it's the good kind of stuff where it's all predictable and you just go, yeah, is that fast enough or not fast enough? As opposed to, and what happens at the 400 edge cases? I just want to go back to just to add to what howard said we don't think of ddup as a penalty in our system ddup and compression always on is a normal state of the system you cannot turn off ddup and compression yeah we weren't

really talking about the data reduction but the the fact that you you store all the intermediate states of a block which you have to for you for it to be able to recover the way you do where a snapshot based system taking snapshots every 15 minutes is going to store less data because a lot of those intermediate states just aren't stored there but that's an ingest, and so it just comes down to, is your system fast enough? Right. And just to clarify that again, we have to track every block as it comes in, because that's the way the metadata works, because every block is an update which is tracked by a system in metadata, right?

So the question is, do we have enough metadata performance to handle fast-changing data? And the answer is yes, because, you know, if you look at what our system is benchmarked at, on a one terabyte working set, on 7030 read-write workload with 8K IO size, we do about 129,000 IOS at about a half a millisecond average latency. On a one terabyte data set. Everybody listen carefully. These are honest people. Because normally we have to ask how big that data set is. And that data set is tiny compared to the performance tier of the storage system being measured. So it's a working set, all right? It's not the back end. It's not one terabyte working

set, which means that, in my mind, that that one terabyte, after it's compressed and deduped, fits into whatever your cache size is. Is that how I should read that? Well, no, because when you have random data, right, we don't know what the compression is or what the D-DUPE is. What we expect is that that one terabyte is basically being served out of memory or from a primary flash tier. Okay, and so that brings us to the next question. This is a hybrid, right? It's Flash and spinning disk? Correct. Because, you know, in an architecture like yours, some percentage of that, some higher than usual percentage of that data is going to be cold.

Data comes in, it goes through the metadata process and gets deduped and gets written Flash first? Correct. How do you demote? You know, this is another unique piece of technology we have, which is our tiering engine. Now, if you think of tiering as a piece of technology, it really has a bad reputation because of all the tiering implementations

that exist in primary stores today. There were some really real stinkers. Well, when Flash first appeared on the market, the vendors who were fitting it into their existing products were dealing with controllers that had very little CPU. And so they did things like, we'll tier overnight in one gigabyte chunks. Exactly. And so tiering got a bad name because tiering overnight in one gigabyte chunks. Exactly. And so tiering got a bad name

because tiering overnight in one gigabyte chunks isn't nearly good enough. Exactly. So when we look at it, we think of that as you've designed your tiering engine to optimize for the slowest tier in your system, right? Because that's why you've decided to do tiering at one gigabyte. Now, the problem with tiering at one gigabyte and doing tiering every 12 hours is this assumption that traffic or data patterns of the last 12 hours is somehow going to predict what's going to happen in the next second. And it's a very poor approximation. Tomorrow we close the month.

We access completely different data than we did yesterday. Exactly. However, tiering as a technology is attractive because if you want to create a system with multiple different types of media, which is going to be the reality of the future because we want to provide different classes of storage because customers require different classes of storage, but customers require different classes of storage.

Well, hybrids aren't going away. Five years from now, we're going to change our perspective on hybrid from a three-tier system with NVRAM flash and spinning disk to a three-tier system with NVRAM 3D crosspoint and flash. And maybe the cloud is a tier, right? So you could be a four-tier system where the cloud object store is another tier in the system. So it is definitely feasible that you could build a system like that. So that's exactly why when we looked at an approach to build a system with multiple different types of media, we figured that tiering was the right answer. However, tiering has

this problem that it is not very reactive to change in workloads. What we decided to do was design a system which used tiering, but tiering was optimized for the fastest tier in your system. And what does that mean? So when you say when you're optimized for the fastest tier in your system,

you basically want a system, a tiering engine that is very reactive. And to make a tiering engine reactive, you now have to tier at very small block sizes. So in our system, all the heat indices are kept at 8K. Now, if you're moving data from flash to disk drives, and you have to move things at 8K, obviously, you're going to kill the performance of that disk drive. When we started without any optimization, when we were just tiering between flash and disk drives, we were getting 4 megabytes per second from 16 spindles, which is obviously not even enough to basically keep a backup running.

So what is the innovation that we came up with? The innovation that we have is recognizing the fact that we can keep our heat indices at 8K, but when we move the data, we have to move the data in a way that is optimized for the target tier. To do that, we have to recognize what data belongs together. And that's a heuristic, that's a proprietary heuristic that we have come up with. It's a core patent of the company, of this idea of how we recognize what data belongs together.

So once we know what data belongs together, when any given block of data gets cold, if our heuristic is correct, then all the predicted blocks that belong together should also have become cold. Otherwise, our heuristic

is not working. And that allows us to move data in larger chunks between the faster tier and the slower tier. So now we end up with a tiering engine that is working continuously, it's not working on a schedule, and is able to move data rapidly between the tiers. So that's the first advantage we get. The second advantage is when we have to promote data, because we've placed the data that belongs together together, when we promote data, if any one of those blocks were to get caught again, we can move all the data that belongs together into the faster tier. Because if a prediction algorithm, again, is correct, then all those

blocks have a very high probability of becoming hard again in the near future. As opposed to just grabbing the hot data and all of the data in the logically surrounding blocks. Exactly. So not only is your tag store, your metadata storage, maintaining the timestamp and the offset and the content hash, you're also maintaining some indication of how data is accessed together? No, that is actually done outside the core metadata structures. Because that's dynamic data. That's changing continuous, right?

And that's, again, the other thing we do is we don't make that prediction once. We basically are continuously making that prediction. So if we find that our prediction is wrong, we go back and fix our prediction. Yeah, and since you've time-stamped every block, you have a lot more heat data. Exactly. One of the problems with hybrid systems is the data that's neither hot nor cold and cycles back and forth between the tiers because it's hot for 20 seconds and then it's not quite as hot as something else and keeps moving back and forth and you know

in a tiering system you use something like lru and you don't keep a heat map for anything but hot yes and you guys can keep a heat map all the way down to absolute zero. As in cold nuts. Yeah, okay. Oh, God. You know, so I had one of the questions I had. You mentioned NVRAM for the write data.

Do you cache any of the read data, or is it all coming off of Flash, or can it come off of Flash and disk? No, both. Oh, so you do cache both. We cache both. Reads and writes are cached, correct. So is the NVRAM used for both write data and read data? Correct, yeah.

But again, that is a property of the fact that we are a globally dedup system. So if we have the data already in cache, then we don't need to read it again because it's already there in cache, then we don't need to read it again because it's already there in cache. If a block of data has been written and it needs to be read, we don't need to bring it back from the flash tier because the cache is globally deduped. We know that the data is already present in cache. So, I mean, can the data be resident in both the Flash tier and the disk tier at the same time? So, I mean, if you needed to, for instance, demote the data, you could just erase it from Flash.

No, it can only be in one place at one time. It is always in one place at one time. And, again, that's the reason why we chose tiering, because when you create a multi-tier system, caching as a technology doesn't work. Because, you know, let's say if you think of the future of hybrid where the primary tier might be resistive RAM or 3D crosspoint, and the second tier is 3D NAND, and the third tier might be an object store. In that case, caching as a technology quite doesn't work because you you end up with two different cache layers right because the 3d crosspoint acts as a cache for the 3d nand and then the 3d nand then acts as a cache for the object layer now so that's the first issue the second issue it's it's wasteful because you're not getting the benefit of the capacity of all the different tiers. Yeah, but what you're wasting is the lowest tier.

Not always, right? If you basically have 3D Crosspoint, which is getting pretty cheap, and that's actually a tier in front of an object store, then that 3D Crosspoint, sorry, the 3D NAND capacity is wasted. Jacob, you can buy Crosspoint today? I'm going beyond thinking of a hybrid system. I got you. Just having two tiers. Yes, but you said 3D Crosspoint was pretty cheap.

I'm sorry. I said I meant 3D NAND. I didn't mean 3D Crosspoint. Of course 3D Crosspoint. Since Ray and I are lusting after some 3D Crosspoint, we were kind of hoping you could hook us up. But that's all right. If you find 3D Crosspoint for cheap, I'm in you could hook us up. But that's alright. If you find 3D Crosspoint

for cheap, I'm in the market for it too. As we all are. As a thousand storage vendors are, I would say. So that's interesting. So yeah, essentially Reduxio is sort of a CDP solution only for primary storage with

tiering and dedupe and compression and all that stuff kind of rolled in. Is that how I kind of read this? It's a primary storage solution, but in the back end of it, it really is a CDP solution kind of thing. Maybe it's the wrong words to use, but that's what I'm seeing here. Except that the data is still in the same file system. Yeah, yeah, I agree. It's not a backup device. You're absolutely correct, Howard. Yeah, I agree. I think terminology in the industry matters, and I think

this is why we don't use the word CDP, because our goal is not to become a CDP system. Our goal is to basically say snapshots are obsolete because snapshots were invented close to 30 years ago, and they were invented at a time when data change rates and the amount of data was small. The amount of volumes you had on a system was small. So, you know, the granularity that was provided by snapshots was sufficient. The number of volumes being managed was not very high. So the overhead of creating snapshot schedules and consistency groups and deciding what applications belong in what consistency group and all that was okay. And today with the data change rates the way they are, is 15 minutes of lost data okay anymore? And I think if you talk to a lot of customers,

once they start using backdating, they realize that 15 minutes was not okay, and the reason they were still using it was because there were no other alternatives on the market. Okay, let's get to some of the basics of the system. You present this as block storage via iSCSI? Correct. This is an iSCSI block storage system. Okay. And I assume the primary target is assumed to be VM hosting.

So the use cases where most of our customers have deployed the product are server virtualization, desktop virtualization, smaller production databases, and test dev and DevOps. All these use cases, the value of fine-grained recovery of data and applications is very high. And this is where customers basically find very attractive about backdating. And given that we have very good performance, you combine that with backdating, it becomes a very compelling story. There are still advantages to application consistent snapshots. Yeah, yeah, yeah. That's where I would go. Is there a hook that will annotate your system so that I can say, yeah, I can recover it at any second, but at 12.02.02, I flushed the database and that was a consistent point and it would be a good idea to restore it

in there. Absolutely. So what we have is something called bookmarks. You can come in and put a text label on any second in our system and you can put a retention on it. You can basically say, either retain this point as part of the overall retention policy for a volume, or keep it forever. So you can basically go create an application consistent point, and you can tag it with a text label, we provide a VSS provider, so that if you want to take up consistent snapshot under Windows, you can create that under Windows. Of course, if you're using Linux or some of the applications which don't run on Windows, you can actually script it.

And again, the mechanism you use is called bookmarks. Now, going back to what you said about app consistent and crash consistent recovery points, what we're finding is given that we provide this one-second recovery, many customers now prefer to go and try to see if they can find a point that is crash-consistent and recover from it because it's an issue of how much data they're going to lose. And applications have gotten better at that. And applications definitely have gotten better at that. For example, with Oracle, with stored snapshot optimization, without anything done on the Oracle side, you can recover from any stored snapshots. That's what Oracle supports, which means on Oracle 12, I forget the actual update, which supports stored snapshot optimization, you get one second recovery with Redux. where a customer had to make a choice between an app-consistent point and a crash-consistent point was a customer of us

who was running Atlassian Bitbucket on us. And this is a company that had 60 developers who were writing software for a network device, and they had a virus corruption which completely corrupted their configuration database. They were not able to get to their source code. They went back to their application-consistent point, which was then saved off using traditional backup. And they found that that data was 22 hours old. So they called us up. They said,

hey, can you help us? And we said, we've never tested this application, but we'll try. We tried five points in the last minute. And the point at one minute ago was actually a good point so we were able to recover this customer to one minute ago compared to 22 hours yeah compared to 22 hours exactly and they will be eternally grateful for a whole month maybe even a month and a half hour you know it depends on the team yeah and stuff that. I am from New York. We do things fast. So the other question, of course, is having all this stuff on a DarkCL storage system is great, but ultimately you still want to do backups and stuff like that.

Do people do backups using your recovery capabilities, or do they take backup from the current level of the storage, or how does that work? Okay, so today we don't provide native capability for DR in the system. We only provide backdating, and this is something that we're working on. This is on our roadmap. Now, if you ask me about traditional backup, I really don't understand why people still use traditional backup. Because ISIS might break into my data center and put bombs in your storage system. Well, I mean, there's a challenge here that if the reduxio system goes bad for whatever reason

and your HA capabilities are not fully functioning properly because they took out two of your power supplies, what happens to the data? Where is the data? The data needs to be someplace else, right? Absolutely. I completely agree with you. Now, I basically made it very clear. I said traditional backup. The question is, given that we have the primary data, can we create a capability that uses some of the virtualization capability that is inherent to our storage system and create a copy of the data somewhere else and provide instantaneous recovery of the data? That's

what we're working on. We still don't announce this capability. It's going to be announced later this year. Okay, so replication coming. Or backup, so replication coming. Or backup, something like that. Yeah. Something like that, exactly. Yeah, well, I mean, the logical thing would be to replicate. Well, replication has some characteristics that you may or may not want,

depending on, you know, the virus level. I mean, if you replicate the whole, let's say, backdating store, then yeah, it makes sense. But if you're concerned about somebody going in and, you know, encrypting all your data, then you really want some point in time prior to, you know, yeah, I don't know. So replication is good, but it's not sufficient. Yeah, I mean, if I had infinite bandwidth, then I want to replicate all the metadata updates in real time. Yeah, yeah, yeah. You don't have infinite bandwidth? Not having infinite bandwidth, then you drop back to point-in-time replication like you were using snapshots.

Yeah. Yeah, I'll give you an idea of, you know, what our vision is, and I think we kind of think of an industry where there's too many silos and a lot of the silos exist partly because this is the way the industry grew up. And, you know, new technologies evolved in those silos. And I think it's time that we took a fresh look at how we deploy storage infrastructure and see if there's a way to collapse the silos. And that's really our point of view. As Reduxio's point of view is that I think technology has evolved to the point, virtualization has evolved to the point, which is virtualization capabilities, processing capabilities have evolved to the point,

and network speeds have evolved to the point where you can now think of how you can create a storage platform that basically now brings in a lot of those capabilities as just attributes of the platform instead of actually having multiple cibers of capability on multiple platforms. And that's the general direction of where we're going as a company. Okay. So we kind of run out of time. Gents, Howard, do you have any last questions for Jacob? Yeah. I mean, the last question is, you know, what's the range of scale I can buy today? The system today is 40 terabytes raw, and we think we'll easily get 4 to 1 savings. In fact, all the systems we've deployed with our customers, basically, we have a cloud-based monitoring system called StoreSense, which allows us to provide proactive support to our customers, but it also sends telemetry data back to our cloud-based monitoring system.

The average savings we are getting across our customer base, which is two petabytes plus of production data, is more than four to one. So, a 40-terabyte system should be about 120 terabytes of user data. Very good. That's good. That's good. And, Jacob, is there anything else you'd like to say to our listening audience? Thanks, guys. I think this was great.

Okay. Well, this has been great. Jacob, thank you very much for being on our show. Next month, we'll talk to another startup storage technology person. Any questions you want us to ask, please let us know. That's it for now. Bye, Howard. Bye, Ray.

Bye, Jacob. Hey, bye. And until next time.