Grey Beards on Systems - Greybeards talk server DRAM IO caching with Peter Smith, Director, Product Management at Infinio

Episode Date: March 4, 2014

Welcome to our sixth episode. We once again dive into the technical end of the pool with  an in-depth discussion of DRAM based server side caching with Peter Smith, Director of Product Management at Infinio. Unlike PernixData (checkout Episode 2, with Satyam Vaghani, CTO PernixData) and others in the server side caching business, Infinio supplies VMware server side … Continue reading "Greybeards talk server DRAM IO caching with Peter Smith, Director, Product Management at Infinio"

Welcome to the sixth episode of Greybeards on Storage. It was recorded on February 20th, 2014. We have with us here today Peter Smith, Director of Product Management for Infineo. Why don't you tell us a little bit about yourself and your company, Peter? Hi, my name is Peter Smith. It's really a pleasure to be here. I started with Infineo very early on. I was actually the first employee. And from the beginning, we saw Infineo as a way to improve storage performance without the excessive cost of adding extraneous hardware that wasn't really necessary. And, you know, I think we've really delivered on that vision, and I hope to be able to tell you a little bit more about that. Gosh, you know, being in the storage industry and helping to sell all those extraneous, expensive equipment that you're talking about is obviously a lot of interest in what you're doing and what you're not doing and how you're talking about. There's obviously a lot of interest in what

you're doing and what you're not doing and how you're getting that done, but I guess we'll get into that as we go farther on. So you are a software-only solution? Let's start there. Yeah, so Infineo is 100% software. There's no hardware involved. You don't need to purchase SSDs or anything like that. What we do is we take a little bit of memory from each one of the ESX hosts. It's eight gigabytes of memory. And our software pools those resources across your entire ESX cluster into one large global cache. So it's not an individual cache per host. It's actually just one big cache that's seen across the entire cluster. And within that cache, it's actually fully deduplicated.

It's perfectly deduplicated down to the 4K block so that any block that any one of the hosts sees down to 4K will only be stored once within your entire cluster. So we make great and efficient use of those very small amounts of memory resources we borrow. Okay, so you're effectively providing another caching tier using DRAM memory across all the ESX hosts in a pool, I guess, in an environment. You've got it, exactly. So I mean, the first question we always ask of these sorts of things, do you do write caching as well as read or just read caching? We do write through, and of course we populate the cache with writes. So subsequent reads would be offloaded, but we do not do write back caching. Okay, okay. So all of the eight gigabyte chunks across the various servers in the cluster create a single pool.

How big can that pool be? The maximum size today would be 2 terabytes. And that's based on the 8 gigabytes per host times 32 hosts, which is currently the maximum cluster size supported by VMware. So that's really just a long way of saying that we'll scale to the maximum supported cluster size of VMware. But of course, you can have multiple Infineo clusters. You could have two independent ESXi clusters, each with 32 hosts, and that would allow you to have two independently scaled Infineo caches. I saw someplace it mentioned NAS. Do you support block storage as well, or just NFS? Today, it's an NFS-only solution. We have put this product in front of all of the major brands of NFS-based storage? Okay, here's where I have to get honest and say that Infineo is a client of deep storage.

I'm in the middle of doing a performance evaluation and have to point out, I have to ask Peter to describe how it is Infinio installs in the data path. Oh, sure. Yeah. So the data path for NFS comes out of a VM kernel interface, goes through a vSwitch, and then out a physical NIC, a PNIC, onto your storage network to access storage. And what we do is we effectively tag packets coming out of the VM kernel interface with a unique VLAN ID. And that unique VLAN ID allows those packets to be viewed and intercepted effectively by the virtual appliances that are the Infineo Accelerator. And in doing so, what we can do is actually intercept the NFS connection, look at all of the operations that are happening,

and collect information and insert responses. And when I say insert responses, what I really mean is offload requests from the back-end storage system. And we do that using this VLAN tagging mechanism. And it's really just a mechanism that allows us to see the raw NFS traffic. So when you're installing Infineo, do you have to take a reboot of the SX? Is this pretty much non-disruptive, or how does that play out? There is absolutely zero disruption. So there's no hardware to install, so there's no reboots associated with installing PCIe cards or anything like that. And the software slips into an active

IOPath with no interruption. And when I say no interruption, I should qualify that by saying that it's the same level of disruption that you would see in a vMotion. So a vMotion is considered non-disruptive. But the reality is that it probably drops a ping or something like that. It's the exact same case with Infineo. We slip into an active IO stream. We've slipped into, you know, 10,000 IOPS without really any disruption to the workload. The workload doesn't know anything happened except, you know, it got faster. And, you know, we've done this at a number of clients for all sorts of very interesting workloads. So the disruption is basically short enough that one retry works to pick things back up once the Infineo cache is in the path? Yep, precisely. Okay, so a lot of these systems have caching that's outboard at the storage. Some of the more advanced customers

have PCIe server caching with associated software. Do you operate, presumably you're operating in front of all that, I mean, do you operate well with somebody that has like a PCIe SSD cache, something like Proximal Data or Burnix Data, those kinds of guys? Yeah, yeah. When you look at system performance, cache hierarchies are the strategy by which we're able to deliver the exceptional performance we do today. And to a large degree, that model hasn't followed suit in the storage industry. You've typically had, you know, substantial caching on the controller itself. You have some modicum of caching on the actual disks themselves. And you've probably got a block buffer in the operating system. And in this case, you know, Windows and Linux both do some level of caching at the operating system level.

But there's this intermediate sort of step where there's just sort of vacancy. There's a great opportunity to offload from all of the downstream components in your storage architecture by using server-side caching. So I wouldn't say that if you're using some sort of DRAM or SSD PCIe flash card in your storage system that it negates the need for server-side caching. I guess the other question is, NFS historically has been pretty metadata intensive. Now, in the VMware implementation, these sorts of things are not nearly as active. I mean, do you maintain caching for just data blocks, read and write data, let's say, or do you maintain caching for directory lookups and things of that nature, or do you care? So we care to the extent that it pertains to the accuracy of the data that we have. And we use a mechanism called WCC to maintain the cache

consistency in our cluster. So what we care about is making sure that the metadata that we see passing through us is reflective of reality. So we're constantly looking at it and constantly updating and checkpointing where we stand as far as the state of data in our cache. Although in the VMware world, there's a lot less metadata updates than in, you know, a Sun workstation NFS kind of implementation. It is kind of striking, actually, because if you do a wire analysis of iSCSI versus NFS, what you find is that to a large extent, there's just read and write operations happening on both. And, sure, you see some get adders and whatnot on the NFS side,

but it's pretty well confined to the read and the write operation from the NFS RPC stack. All right. So you mentioned cache consistency across the pool. So if I'm a little VM in a small ESX server and I do a read to my NFS file, that's the first time it's done. Presumably, you'll go ahead and allow that activity to go to the storage, and then you will bring that data into the cache for the local ESX host. Does that data also get propagated to the other cache elements across the pool? And inside Infineo, there's two distinct pieces of content that we track. There's the metadata, and then there's the actual content itself. And the metadata in Infineo is a cryptographic fingerprint of the content along with access pattern data.

And so there are two places where that exists. The metadata is kept locally for a very good reason because that metadata is the item that correlates in address to content. So the addresses in a VMware system, they're all single writer, meaning that a given address is only accessed by a given host at any given time. And the only reason a different host would ever access that address space of a given VM is if vMotion were to hand off the lock on those address ranges. Now, the content, on the other hand, in Infineo, given that it's deduplicated, the content could be associated with one or more VMs. So what we do is we keep the metadata local to the host where the lock on the VM exists, and then

we take the content and we spread it across the cluster. And the reason for spreading it is to amortize the lookup cost across your entire cluster so that one node in the cluster doesn't become a bottleneck. So imagine if we sort of just lazily placed content and we sort of had some hot content that was on host one and it was really, really picking up pace. Now host one would become a bottleneck for that content. So instead, by evenly distributing it across the cluster, we get very consistent access times. So are you doing data location based on access frequency? So it's not based on access frequency. The eviction is based on access frequency.

Okay. So, I mean, the obvious mechanism to me would be I take a block and I generate a cryptographic hash, and then each of the nodes in the cluster is responsible for some fraction of the hash space. Yep, precisely. Okay, so it is content addressable. Exactly, yep. Yeah, so that's a consistent hashing scheme. Got it. And that's a consistent hashing scheme. Got it. And that's exactly what we do. We do consistent hashing across the cluster, and the metadata that points to that consistent hashing circle, that all lives on a per-host basis, whichever host has the VMX lock on the VM.

Right. So in this case, it's likely, you know if I have a 32 host environment, then I'm going to have to go to another host to gain access to that data. And there's certainly a network return time and all that stuff to do that versus going to the storage to get the data. And the storage, of course, is also a network hop as well. It's the same administrative distance, to pull out some network nomenclature. It's that same distance from the storage system to a peer,

unless we're talking about something like a metro distributed cluster or something like that. That's certainly it. In which case, it's further. Yeah, there's no question that metro clusters are a totally different beast. Quite frankly, most of the customers that are looking at this kind of stuff, they aren't necessarily doing active, active, you know, sort of split down the middle metro clusters. Maybe campus-wide.

But that begs the question, is an Infineo cluster by definition a VMware cluster? It is, yeah. Okay. Yep, by definition it is. And our clusters do not span multiple ESX clusters. You can have, technically, you can have multiple Infineo clusters in an ESX cluster, but you cannot have an Infineo cluster that's across multiple ESX clusters. That's for good reason, though. It's because we want to make the process as configuration minimum as possible. And it's a pretty good assumption on our part that if you deliberately made multiple clusters and segmented them, that you probably want to segment the caching strategy as well. Yeah, so Peter Ware, I was going with the question about administrative distance and all that stuff. So in the case of a data read, let's say,

coming out of the cache in a storage controller versus coming out of an ESX DRAM cache doesn't appear from my perspective to be that much different. So the big difference is that the storage controller has a finite amount of CPU resources. And those resources are being used to do everything from data persistence, protection schemes, backups, replication, snapshotting, restores. The list goes on and on, a list you're very familiar with. But those CPUs only serve those functions. Now, imagine, though, if you could take two CPUs from every single ESX host, pool those resources together. All of a sudden, you've got far more resources to deal with the performance aspect of storage delivery than that centralized controller has. And far less responsibilities, by the way.

We aren't dealing with snapshots or replication or anything like that. It's a simple key value. I need this content. Give it to me. Yeah, but on the other hand, and I don't want to belabor this point, but, I mean, obviously the ESX host is doing other stuff for other VMs and, you know, processing their activity as well. Yeah, and there are CPU reservations set on the Infineo appliances

to guarantee 50% of the CPU resources we need for serving this content. And it's worth noting that as a result of this strategy, the empirical evidence supports that distributed lookups from the DRAM cache within Finio have an average fetch time of sub-millisecond. It's just under a millisecond. And that's very consistent and reliable. deviation of latency on a storage controller once it's under even a modicum of IO pressure the the standard deviation of these systems goes all over the board and it's a very wide standard deviation now what Infineon does as a result of this very consistent latency is we narrow the standard deviation, effectively giving you more consistent and more reliable IO performance.

Not to mention that that 32-node cluster has 256 gigabytes of deduplicated Infineo cache, where the FAS8060, the top of the line they just announced, has 128 gig of non-deduplicated cache. Yeah, no, Howard. I mean, to some extent, deduplicating and doing a cryptographic hash on a 4K data block is a non-trivial CPU-intensive activity. It is, but that's only on the right function. So now let's imagine VDI, where I have 600 identical virtual PCs. Or even if I used linked clones, I've got five different routes to which all of those clones link.

Now the deduplication becomes really effective. I'm looking for the specific numbers right now, but I did a study on using my laptop, it's a MacBook Air, right? To determine how many SHA-1 cryptographic caches a single core on a MacBook Air could produce. And the numbers were staggering. It could easily saturate a one gig ethernet connection with the amount of content that it was processing. And on a 4k block it was somewhere north of like 10 000 sha-1 hashes so that's that's one core being on a consumer grade laptop being able to produce what is effectively hashes for 10 000 iops yeah so that that's a macbook air less than two years old right yeah yeah it's it's uh it's a it's two years old yeah Okay, because that's about the point where

Intel put SHA-1 into the microcode. So there is an assembly function in current Intel processors to calculate the hash. So, I mean, the other question, of course, is that SHA-1 is not a perfect hash. I mean, there's one chance in a bazillion. There are no known collisions on SHA-1. No known? Okay. There are zero known collisions on SHA-1. And there are supercomputers literally, as we speak, trying to find collisions. They've been successful in finding them in md5 and a couple

other algorithms but cryptographic caches are they will eventually find collisions and the moment they do i guarantee you that us and every other deduplicated storage system in the world will stop using sha1 yeah yeah so you don't go through the process of doing a byte-to-byte compare to make sure it's the correct data as everything else, I guess, right? That's just something people do to make old farts like you feel happy. I am an old fart. Thank you, Howard. Otherwise, I wouldn't be a gray beard and be talking about this shit.

When I was teaching backup school, Curtis Preston actually hired a Ph.D. mathematician to calculate the probability of a Shaw 1 collision. And the other thing you have to remember is the probability of a random collision and the possibility that I can create a collision on purpose are two different things. Yeah, I agree. But we calculated that the probability was greater that the audience at my seminar would be killed by an asteroid strike while I was speaking. You do tend to speak on. I might add that. Go ahead, Howard. That is true.

It was a full-day seminar. Yeah. But still, the odds were 10 orders of magnitude greater. All right. I concede the point. I'll go back to sleep here. But okay. So back to the question.

So a write occurs. I'm a small VM. I do a write, and it goes to your cache. You do a write through, so it goes to the storage right away. In the meantime, you do a cryptographic cache. You say, is this a duplicate or not? So in order to determine if it's a duplicate or not, you'd have to have that cryptographic caching data located either in all,

replicated in all of them or distributed. So let's say you've got it distributed, which is what I understand you said. And now I have to go to host B over there and say, okay, is this in your current hash space? And if he says yes, then you just drop it and you say, okay, it's a duplicate and I know where to go to get it. If it says no, then you have to send the data over to that host to apparently record it in its little hash space. Is that correct? Is that how this works? So we know on the host that the request came from or went to, your ESX host, we know whether or not that content is going to be located on the remote node. So if we separate these two things, there's address to digest and then digest to content, and we couple those two tables, that means that wherever there's content that says

where an address to digest lives, we now know that there's a digest to content entry that's viable in the cluster. So the digest itself using consistent hashing tells us where the location is of that content. So it's not a question. It's just a request. It's a request to say, give me the data. That's it. Yeah, no, I'm talking about the write process, not necessarily the read process. Oh, I see what you're saying. So writes are asynchronous for the distribution of content. And the reason that's possible is because to maintain the consistency of the tables, you only need to make sure the metadata is updated. It doesn't actually matter if the content is placed in the cluster for a write. All that matters is that the address

to digest entry is updated. So what we do... At the local host. Yep, exactly, precisely. So what we do is we'll take your write, we'll insert the address saying there is something happening with respect to this address. If another request comes in while this update is happening, you're going to go to the storage system because we don't have all of the data that you need. But keep in mind, this is happening at the speed of light, right? So we get your write. We put the address into the ADD table.

And then in the background, we calculate the SHA-1 hash. We populate the ADD entry with the address and the digest, and then there's a separate asynchronous process that pushes that digest to content map into the cluster. So there's never an opportunity where the wrong data could be believed to be in cache. And that's really the process. By making these two latter portions, the digest calculation and the content and digest push asynchronous, it means that the system can continue servicing requests as fast as it can without having to wait for hash calculations, which I think was part of your earlier concern. Right, right, right. And since we know the map of the digest to host, when new data comes in, you can just

send the data and the digest to the host that would store it, and it can look in its table and see whether it needs to store it or whether it already has it yep you don't have to go through the asking the question stage you just send you're getting like reference counting right so now now it's like it's not even just about um do you have it it's how many people how many nodes have it so that now once once you see a reference count go to zero, you can say, well, you know, why do we need to keep this around anymore? Right, right, right. And evict it.

Yep. Yeah, so you could evict, you know, you could evict if reference counts got to zero. You could evict if there was a long period of time where a block wasn't necessary, but reference counts were greater than zero. And the questions about when and why to do it are really part of the… Proprietary logic.

Yeah, exactly. Now we get into secret sauce. Okay, the other question now becomes what happens when I add a node to the cluster? Yep. Obviously, you need to redistribute all the caching and the content and all that stuff. You could effectively, let's say, I'll call it flush a portion of all the old hosts cache space and let the current, the IO that's currently going on start to repopulate. Is that how it would work?

Or you could also migrate the data and the hashes and all that stuff to the new host. Yeah. So, um, with, uh, with consistent hashing, the, the technique is, is pretty, pretty well defined in, uh, in that if you, if you think about this as a, as a clock face, I'm going to make up an analogy on the fly so this might completely be awful, but... Go for it. So if you think about consistent hashing as the face of a clock,

and you've got, you know, noon, you've got six, and let's say from 12 to 1 o'clock, you assign any addresses between 12 and 1 o'clock to host A, right? And then you assign any addresses between 1 o'clock and 2 o'clock to host B. Now, what happens, let's add a third one. And then from, so that was 12 to 1 is host A, 1 to 2 is host B, and 2 to 3 is host C. So what happens when you add another node? What happens is you subdivide those ranges.

And you say a fraction of the ranges now go to host D. And what happens is that all of the other hosts that were previously responsible for a set of address ranges, they simply age out. They don't need to be there anymore because they simply aren't being accessed. So they age out as a result. So you've effectively redistributed your content, and it's really as simple as that. Now, I think a natural fallout from that question is, so how about migrating VMs within the cluster, right? So that's like warm cache vMotion, right? Our approach there

is pretty simple and pretty powerful. What we do is we populate into this distributed cache all of the address to digest entries associated with a VM. And then when the lock changes hands between a host, we simply look into the distributed cache for all of those address to digest entries that the new host now needs. And what they find is a little bundle. It's around, excuse me, it's around 45 to 100 megs, depending on the size of the cache content for the VM. It sees it in the distributed grid, and it goes and grabs the bundle, populating the ADD table in the new host. Now, that's the first half.

The second half is now that you've got this new VM on a new host, post-vmotion, with an accurate, up-to-date ADD table. What about getting content? Well, the beauty of this solution is that we were always amortizing the cost of a content fetch across your cluster. And just because a VM moves to a new host doesn't change that story. The only thing it needed was the mapping to know where to find all of the content in the cluster. So now we've got that problem solved, right? We pushed that map through the cache to a new host. And now that new host can find all of the cache content across the cluster. Yeah, sounds good. You know, we've got like seven or eight minutes left. I've been kind

of asking a lot of questions. Howard, you might want to step in here and ask away. Sure. You know, the idea of RAM caching makes a lot of sense. Unfortunately, RAM is a critical resource in a lot of VMware hosts. You know,, in clusters I've worked on, we run out of memory before we run out of compute resources, although newer servers with more slots help. How do you battle the, why don't I add an SSD and not lose, so I can run more VMs in the memory I have argument.

Yeah, you know, I think there's a few ways of looking at it. One is from an operational perspective, I'm not really too interested in, like, RMAing additional hardware when they invariably die and given given the frequency of of ssd death um i'd say that the burden of such rmas could be relatively high um furthermore um as far as just the the operational burden to even deploy these things uh there there's a lot of just subtlety about the process of managing infrastructure that you and your viewers are certainly aware of. But just to sort of belabor the point, let's take, for instance, if you were to deploy a PCIe-based SSD into one of your rack mount servers. Well, the majority of people aren't using cable management arms.

So right off the bat, even if you had hot pluggable PCI cards, you probably have all of your power cables cinched down to the rack rails. So just to get that server out, you actually have to power it. Leaving the physical part aside, I have yet to meet anyone who trusts the concept of hot plug PCIe. Yeah, no, no. I totally get that. I think that's a pretty bonkers notion as well.

But I'm sure there are people that would love to try. Well, I mean, if we're talking about something like the PCIe front mount two and a half inch slot on a Dell R720, sure, I trust that as hot plug, but if I got to pull the server out at the end of its rails and take the cover off, I powered it down. Especially because I've got

a bunch of one-use servers crammed full. And as soon as you take that lid off, the temperature in there starts to climb dramatically because the fans aren't blowing air over the processors properly anymore. So, you know, yes, electrically it might work. Practically, I have yet to meet anybody who trusts that.

Right. Yeah. So you can imagine when you have to RMA that device, it's not just the cost of installation and the hassle of installation. It's also the ongoing burden of when that device invariably fails and you now have to go take care of this thing, probably at an off-site colo. And, you know, it sounds, to an outsider, it probably sounds like it's just a nuisance. But I think one of the goals of an operations organization is to not be interrupt driven. And insofar as you can eliminate these interruptions to your daily workflow, like failing hardware that needs to be replaced, I think you're putting your organization into a better position. Okay, there's a simplicity argument there. I can go for that. Yeah, just operational cost alone, right? Right. Although we've been seeing some very interesting devices lately cropping up from Vikings NVDIMs to Diablo and SanDisk's UltraDIMs that put flash on the memory bus. You know, those do look very cool, by the way. I really genuinely hope that they can convince some of the big vendors to support that in their BIOS.

Yeah, they got IBM so far. Oh, do they really? Oh, yeah. But only on a pair of, you know, the two servers IBM just announced, which are beasts. You know, it's the four-socket and the eight-socket server. And we all know that in the VMware world, the two-socket server is where the bulk of the market is. I mean, you can imagine the benefit of that kind of technology. I mean, if you've got a bunch of pages that are just sitting in memory and taking up space, but not really being accessed all that frequently, and you're

not going to swap these things out just because, you know, swap is terrible. If the BIOS could actually reshuffle using, you know, virtual memory addressing, reshuffle it into Flash, that's just, that is a remarkable achievement. I really hope that that really catches on. Well, the UltraDIMM specifically can be addressed as lower performance, can map into main memory. Right. So then it starts being, oh, look, and we can be persistent too.

Also another characteristic, unlike DRAM today. Yes. Although I am having flashbacks when I think about persistent RAM. Can you imagine? Like, I remember back in, like, the Apple IIe days where you would hit the power button and you would count to five to make sure that everything was flushed out before you restarted. And now you've got persistent RAM. Is it, are we going to regress? Peter, Peter, you are a near child. Ray and I remember, Ray and I remember core, which was, but it was persistent main memory. I remember we used to turn the PDP-8 off just by hitting the stop and then turning the power off and then turn it back on and hit run, and it would pick up right where it left off.

Yeah, yeah, yeah. We are old-style people here. I should have known I shouldn't have referenced historical artifacts when I'm on the Greybeard podcast. Thank you, Peter. Thank you for acknowledging that. All right, so we're hit about the time frame here. Are there any last questions? Yes, I have.

You can ask your own question, Peter, if you'd like. Where does Howie Claus come from? That would be a Howard question. Go for it, Howard. We're going to expose this, huh? I guess. So Howie Claus comes from my daughter being at university in the UK. And Skype being the most reasonable way for us to communicate

and her sending an email about how horrible it was to be away from home for her first Christmas. Oh, that's good. And so I immediately set up a Howie Claus Skype account and have just never felt sufficiently embarrassed by it until now. There you go. To change it. You know, the other, I mean, the last question I have is why are we limited to 8 gig? And some of the servers coming out now will take

512 gigabytes of memory and even if we take that as the usual okay you could do that but you can't afford it because the biggest dims cost way too much it's still 256 gigabytes and that's a lot yeah yeah so um while i can't speak publicly right now about the ultimate feature, what I can say is there's something very special coming with respect to resizing Infineo caches. Okay. And the 8 gigabytes was merely an artifact of my sort of back of envelope calculations on what was a reasonable ask from current server platforms. Right. If people have 96 or 192 gigabytes of memory, how much can I ask them to give up? Yeah, yeah, precisely. And we know that, you know, obviously we can increase the size of the cache, and we need to make that something that, you know, that the user knows why they might want to do that. Right.

Well, this has been great. Thank you, Peter, for being on our call. It was great to be here. Thanks for the invite. Great. Next month we will talk to another startup or storage technology person. That's it for now. Bye, Howard.

Bye, Ray. It's been a pleasure. Thanks again, Peter. Until next time, thanks for listening.