Grey Beards on Systems - 90: GreyBeards talk K8s containers storage with Michael Ferranti, VP Product Marketing, Portworx

Episode Date: October 2, 2019

At VMworld2019 USA there was a lot of talk about integrating Kubernetes (K8s) into vSphere’s execution stack and operational model. We had heard that Portworx was a leader in K8s storage services or persistent volume support and thought it might be instructive to hear from Michael Ferranti (@ferrantiM), VP of Product Marketing at Portworx about … Continue reading "90: GreyBeards talk K8s containers storage with Michael Ferranti, VP Product Marketing, Portworx"

Hey everybody, Ray Lucchese here with Matt Lieb. Welcome to the next episode of the Graybeards on Storage podcast, a show where we get Graybeards and storage bloggers to talk with system vendors to discuss upcoming products, technologies, and trends affecting the data center today. This Greybeard on Storage episode is recorded on September 24th, 2019. We have with us here today Michael Ferrante, VP of Product Marketing at Portworx. So, Michael, why don't you tell us a little bit about yourself, what's happening at Portworx, and why Kubernetes seems to have taken off. Well, thanks, Ray, And very glad to be here. Hello, Matt. Looking forward

to a great conversation today. You know, I'll start with maybe a little anecdote about myself, which I think leads into, you know, our discussion, which is I was just checking my LinkedIn because I wanted to get the dates right. So I've been doing storage for Kubernetes since June of 2014. And if we take the Wayback Machine, in June of 2014, Kubernetes didn't exist. And so I should clarify and say I've been doing storage for containers since June 2014. Right around that time, Docker was really taking off, but didn't yet have a concept, a native concept of storage or data management within Docker.

So some of the early steps within the Docker ecosystem to introduce storage concepts, I mean, the ability to run stateful services like databases was around volume drivers that automated the provisioning of storage. And so in June of 2014, none of that existed. And so I had been working at a company, a great company, Rackspace, for about five years

and was enjoying the work there. I was working with one of our SaaS business units and I really enjoyed it, but I was thinking, I want to do the work there. I was working with one of our SaaS business units. And I really enjoyed it. But I was thinking, I want to do the startup thing. And the team at the time that was running that large scale SaaS service, it was an email API for developers. And we're sending about a billion emails a month. Billion? Yeah. Very, very significant. Not so much traffic there, huh? Yeah. Yeah, exactly.

So I apologize to all of those people who probably received some of those. But that team, in running a large scale SaaS service, they had to think about automation and management of that SaaS service. And they also had to think about how... We were getting requests from enterprise customers saying, Hey, I really want basically the API that you provide for this service, but I don't want to run it in your cloud. I want to run it in my own cloud. So I basically, I want you to package up your service and let me deploy to my own data center. And so there were a bunch of operational management challenges for that team. And I've never seen anybody so excited in my life as these engineers were about

Docker. And I said, you know, okay, there's something there. This is an area, I don't know exactly what I want to do around containers and Docker, but that's the area that I want to go into because I saw kind of the boots on the ground perspective from real system engineers with a real stake in making containers work. And so as I looked and said, you know, what could I do? Could I do security? Could I do networking? Could I do storage? Just kind of, you know, the big boxes of functionality. And, you know, I met an entrepreneur named Luke Marsden who had this idea about doing data management for containers. And I knew things. One, that historically,

storage has been a very valuable and important part of enterprise IT. I mean, you know, let's talk Oracle. I completely concur. Yeah, completely. And so, you know, if containers were really going to be the basic building block

of enterprise IT going forward, which is, you know, what I was betting on by quitting my, you know, comfortable job and joining a startup, then someone is going to have to solve the storage issue. But the other thing that I knew is that there was this, you know, idea that 12-factor applications are basically not running stateful services inside containers was the trend. And I had to reconcile these things. On the one hand, I said, you know, storage is going to be huge for containers. At the same time, a lot of people that I interviewed said, hey, I don't want to run databases and containers. And where I came down, and I think, you know, I'm glad that I did, was not that people didn't want to run databases and containers and where i came down and i think you know i'm glad that i did

was not that people didn't want to run databases and containers it's that they they didn't feel like they could with the operational reliability and performance and security that those applications require and so it wasn't technology the truth is at the, that was a reasonable response. I mean, it really wasn't in place. Yeah, I mean, if you were running your databases in Docker containers in June 2014, man, I would question some of your judgment. Your sanity? Exactly. And so, you know, to make a long story short, I decided to take the plunge. That company, you know, we didn't find the right product strategy and business model to make it work. But the segment of the market, data management and storage for containers, is absolutely something that's critical. We're seeing it in our customers

today and we're seeing it, you know, through the growth of the ecosystem overall, not, not just Portworx. And, you know, I think there's a really bright future and I kind of liken it to, you know, what, what would VMware be today if, if you had never been able to make the transition from running test dev workloads in your VMs to being able to run production applications. And I see storage and data management as one of the key components that's going to let us take Kubernetes and containers from, you know, test dev or even, you know, production like workloads of a CICD environment. Something that is itself very important to a business. I mean, that's your dev pipeline, but it's not itself the end production application. Going from there to being able to run really kind of, it's an overused term, but those mission

critical services on Kubernetes. Yeah. So that's my story. God, you've been there forever. So, I mean, the last couple of, I don't know, last couple of releases of Kubernetes, they've started to create more of a, you know, a nice plugin environment for storage. I think it's called CSI and that sort of stuff. Is that where, so where does Portworx fit in all this? I guess I should ask. Yeah, that's a great question. So CSI is, I think, one of the projects that within Kubernetes that we're most excited about, in fact, we have engineers on our team who are contributing directly to that. Luis Pobon is one of them. He's kind of really an old hat storage engineer, you know, very good working with the community and kind of understanding how you can make, you know, generic systems that provide capabilities for a lot of diverse applications. So we're really glad that Luis is working on that project. We see it as absolutely critical to the success of

Portworx into the broader ecosystem. But it's important to understand what CSI is and what it is not. CSI, it's well named in that it's very descriptive of what it does it's the container storage interface so you think about what csi is well it is an interface for connecting containers orchestrated by kubernetes um although csi started by having a single interface not just for kubernetes but also for Docker, also for DCOS. I think that ecosystem is changing so much now that really the important interface is for Kubernetes itself. But it started out as a general purpose interface

for multiple container orchestration systems. But what is it interface? It interface that container orchestration system with underlying storage. That's really it. And so I think the important thing to understand is, I was interviewed for an article that TechTarget wrote a few weeks back, and it was called Yunking Five Container Myths About Data Storage Containers. And the myth, it's a little bit of a clickbaity headline. But the myth in air

quotes that I chose to address was this idea that because I have CSI, my storage is container native. In other words, because I can connect my storage to a container, I now have container native storage. And that really fundamentally is not the case. And kind of the pithy quote that I used, which I wish I could come up with something better because it's not my favorite analogy. I don't think it's perfect, but I think it's helpful, which is to say, you know, USB is a standard charging interface for mobile devices now. So, almost any phone manufacturer can build a phone to use the USB 3.0 spec and charge on a standard USB plug.

Just because you have a USB plug on your phone does not provide any particular functionality to the phone itself. In other words, it doesn't make it a smartphone because I can charge via USB 3. The same way, just because I can connect my Kubernetes via CSI to say my SAN within my data center, it doesn't provide any container native functionality necessarily. All of that functionality still needs to be provided at the storage substrate layer. It's just now I have a language to access that functionality within Kubernetes. And what this means to get to the point and kind of it, this really isn't kind of

just an esoteric, you know, objection, but rather to say, okay okay let's say i want to do something like dr for for kubernetes i have a bunch of applications that i want to be able to um uh to back up that actually use storage and stuff like that yeah yeah exactly and you know i want to let's say i have an rpo zero requirement. Synchronous replication with containers, okay. Yeah, exactly. And let's say I've got an RPO that's sub one minute. I'm being very aggressive on these for a point, but 451 Research recently did a survey called Voice of the Enterprise Storage Workloads and Key Projects in 2019. That's kind of a mouthful.

Maybe we can link to it. And I don't know if the show notes is a possibility, but they talk about, you know, what are average RTOs and RPOs for the enterprise? And for mission critical applications, they're seeing that 50% of the enterprise has an RTO of an hour or less. But then you have some mission critical applications that have an RTO of less hour or less. But then you have some mission critical applications that have an RTO

of less than a minute. And when it looks at RPOs, 60% of the enterprise is saying, I need an RPO of under an hour. And a segment of those mission critical applications needs an RPO of under five minutes or even zero RPO. So these are not theoretical concerns. So if you have a bunch of applications running on Kubernetes and you require very low RPO and RTO, how do you do that? Now, the problem fundamentally with this idea that just because I have CSI, I have container native storage, is that most storage arrays do not speak in a container granular manner. They speak at a machine level. So most storage systems speak at a block level or at a file level. And containers, I'm trying to understand, when you say a container, you're talking about multiple microservices that are running across one or more pods in this environment.

And they're accessing this storage through a block interface or a file interface, I guess. Is that correct? Yeah, so I'll give you a concrete example. So I have three servers. Each server is running two pods. So I have six total pods. And three of those pods are a Cassandra database, which is distributed, meaning I have, you know, it's the same quote unquote database, but it's distributed across three servers.

And then I have three individual instances of MySQL. So a question I could ask is, how do I back up that three node Cassandra database? Now, because I'm operating at, so within VMs, typically there's a one-to-one correspondence, and maybe this is the piece that was missing in my description. Typically within a VM environment, there's a one-to-one correspondence between a block device and the VM itself. So if I want to back up. So a VMDK kind of thing. Okay, I got you. Or a VHD.

Right, exactly. So if I want to back up my SQL, then I would take a snapshot of the machine, which by extension, that's a one-to-one correspondence to that VMDK. That model breaks down in containers because on that single VM, I am now going to have multiple block devices. So I need to be able to snapshot at the individual

block device level. But then my application that I'm backing up in the case of Cassandra is a distributed application. So I can't take a machine-based snapshot to back up my Cassandra because then I'm going to have a bunch of My SQL data and I have a, I have a pretty chart that, that illustrates this. So I'm trying to walk through it descriptively and hopefully it's not going to be lost in the details. But if I have multiple pods running on a single VM, then I can't take a snapshot of the VM DK itself and only have the data that I'm looking for. I'm going to have other data that's included in that snapshot, which is not ideal. At the same time, if I have a multi-node Cassandra database,

where I have multiple Cassandra pods running across multiple hosts, now what I need to do is take an application consistent snapshot of a subset of the data on my three nodes. It's very, very complicated very quickly. In addition to that, I'm giving, I mean, because I don't have a diagram, even this simple example seems a little bit complex, but now let's multiply that complexity by hundreds of applications and say, okay, now what I want to be able to do is I want to be able to set up DR for these distributed containerized applications running across a fleet of 100 servers, and I want to do it in bulk for what's called a namespace within Kubernetes. A namespace is basically just an organization.

It would be like an application or something like that, right? Exactly. Typically, a namespace would have multiple applications. You might have a namespace, say, for, I don't know, within a bank, you might have like the, you know, customer. Teller namespace or something like that. Exactly. So, you know, I have hundreds of pods, thousands of volumes that I need to back up and recover as a unit, existing storage systems do not provide a mechanism to make it easy to back up and recover individual pods

because those pods can be running, it's a subset of the apps that are running on a single VM, and they don't provide namespace granular controls for it. And so just to give you a concrete example. Just to be clear, Michael, I mean, replication and backup are not necessarily the same things. And storage replication might be able to provide you an RPO zero kind of recovery point objective kinds of capabilities, but it's not necessarily an application consistent view of the world, right? Right. And where's the cache coherency, right? Yeah. So Portworx does, but you're exactly right, which is where the complexity comes in. So later in the day, I'm doing a webinar with David Linthicum, who's a kind of a very well-known kind of cloud architect and consultant and strategist. And one of the things that he writes a lot about is kind of, you know, he kind of debunks a lot of, where I guess rains on a lot

of people's parades. Because, you know, his basic MO when he writes an article is to take something that people are really excited about, and show you what the reality looks like instead. And now it's not going to be, you know, it's not going to solve all your problems. And here are three reasons why that's basically how he writes his articles. I mean, it's a really refreshing perspective because like you've just pointed out, even what I've just said, okay, here are these problems, FortWorks solves, you know, one or two of them. But what you just pointed out, Ray, is there are still these other problems. And that's why, that's the core reason why FortWorks exists as a company, which is to say enterprises have requirements around performance and security and reliability for applications running anywhere. What we're

going to do as a company is we are going to address the requirements, those requirements, performance, resiliency, security for applications running on Kubernetes, which means as a, so, you know, fundamentally we provide storage and data management software, but, you know, we can't just provide replication for containers because as you've just pointed out, you know, just having a copy of a data volume somewhere else in the cluster does not equal disaster recovery. There are other parts, you know, just having a replica somewhere doesn't solve application consistency. So what we do, and I'm trying not to make this into just kind of an overview of Portworx, but what we do is we say, okay, for DR, what would DR look like in a containerized environment? And we solve one by one

all of the different problems that are preventing, say, we're working with a large bank who had a requirement for a particular application to run on Kubernetes, had to have RPO zero and sub one minute RTO. We said, okay, what do we need to do in order to solve that business requirement? And we basically built a product that we call PXDR that does just that. I'm happy to go into it. Suffice it to say, you're exactly right to point out, as I'm saying, you know, let's talk about data replication. Just solving that is not sufficient. So as an example, we looked at another storage system that could provide, you know, site-to-site replication that would allow us to get to the zero RPO. But then we said, okay, well, let's look at RTO.

How low can we get that? do backup and recovery at a namespace level, basically with whole entire groups of volumes. You had to do things at an individual volume level, and then you had to map it to individual application configuration. So when you started up the new application in the new environment, bringing everything online, suffice it to say there were nine steps per volume with that existing system to get to the similar functionality that Portworx was able to provide with two commands. Simply because we worked at a higher level of abstraction at the Kubernetes level of abstraction, which none of this to say that no one else could ever do it. But it requires a different view of the world in order to provide that functionality. So Portworx builds on top of storage synchronous replication, or you provide synchronous replication for the storage yourself? So we are, you could think of it as, at first

glance, as a software-defined storage layer. So we're in the data path. So we are actually responsible for writing data to disk. So we do all of the traditional storage capabilities that you would get from a software-defined or an array-based storage system. In fact, our CTO, who himself kind of worked at Dell EMC and other companies like that, building storage systems in the past, had recently said to me, you know what we are? We're a sand for containers. And as the marketing person, I don't like that.

Yeah, I wouldn't either. It's a different paradigm. and you know as the marketing person i don't like i don't like that uh yeah i wouldn't either uh i don't like that framing of it but i think it's clear like we we you would when you buy portworks you are buying a storage system so you don't need to run up on top of an existing storage system you need all we need is some vms or some bare metal servers, some EBS volumes from Amazon if you're going to run in the cloud. And then we provide, we actually are in the data path. So we handle all of the writing to disk, all of the replication, all of the capacity management that you would expect from an enterprise storage system.

But we do it in a way that is native to Kubernetes. And when I say native to, what I mean by that is everything that we do is at a container granular level, not even just a block device granular level, because you can have, you know, we can have a single block device, say a single EBS volume in Amazon running to attach to a single EC2 instance. And then that EC2 instance itself can have say a hundred pods running on it. With thousands of containers or something. Exactly. And we provide virtual volumes to that, which, you know, one, one problem just as an example that that addresses within Amazon is that if you

wanted to say, let's say you wanted to run 50 containers per EC2 instance, and then you needed an EBS volume. You actually can't do that in Amazon because there's a hard limit in how many individual block devices you can attach to a VM. I believe the number is either 24 or 36. So once you pass that limit, even if you have enough compute capacity to run the next container, there are kind of no more mount points for block devices. I'm a little confused, though, about the model. Let's just take a step back. you're saying that as a software-defined model, you are basically containerizing storage on whatever physical device is being presented. It can be an EBS volume.

It can be your local SAN. I imagine GCP and Azure as well. But there is no physical hardware sold. So how's the licensing work? For each host that you run FortWorks on, you would count that as one and then count up how many hosts you're running FortWorks on

and that's the number of licenses that you would purchase. So you're correct that it's not a storage capacity issue. We don't, you know, it's not, you know, cents per terabyte. It is how. Basically, the way it works, just to simplify things further, is typically people will install it on each of their Kubernetes worker nodes. So if they have a 100-node Kubernetes cluster, then they would purchase 100 Portworx licenses. And how would you address things like IO for,

let's say you've got a high transaction Cassandra database and the IO isn't matching what you require and it's sitting somewhere. Do you add more nodes to it? How does that work? Yeah, so there are two ways to scale. And actually, I'd love to come back to you at KubeCon because we actually have a product announcement that's around automation of this capacity planning and kind of you know based on

policy so you give us something like here's the IO here's the my IO requirement for this application instead of giving us here's how many servers I need. You just give us the IO requirements. And then Portworx actually provisions the infrastructure to meet those requirements. We have some automation coming around that. But to talk about it more simply atomically without the automation, there are two ways to basically grow your cluster. You can add additional nodes where each node itself has additional storage. So that would be kind of, you know, scale out.

We have some customers that will run us on Amazon using what's called Instant Store, where they're actually using the local SSD on the VM itself, which is not persistent. So if that VM reboots, then you kind of lose your data, but that's where Power where poor replication comes in. So you would add additional EC2 instances in, or if you were running hyperconverged within a bare metal environment,

you would just add a new bare metal server into the cluster, or you can add additional block devices to existing services, existing clusters. Yeah, I mean, it would be much easier to do that on-prem, just configure the storage to support a higher IO, whereas, you know, I imagine in the cloud, you really have a different paradigm to follow. Yeah, it does typically differ in if... Well, does it differ?

Well, you just add EBS volumes to the EC2 instance, I would guess, right? Sure, like adding disks to a volume group, however many spinning disks that is, versus however many IO they provide. I imagine that that also supports a level of redundancy and disk synchronization across that as well. So you didn't mention anything about RAID protection and that sort of stuff, but you did talk about replication as being a Portworx function.

So, I mean, is it possible to replicate between something like an AWS container application running Portworx and an Azure application running Portworx if such a thing exists? Yeah, absolutely. In fact, so I'll talk about it in terms of this bank we're working with for the PXDR. So the zero RPO sub amendment RTO.

So asynchronous replication. Okay. They have two, in their case, it's two OpenShift clusters and they happen to be running on prem, but it's two separate shift clusters and they happen to be running on-prem but it's two separate data centers two completely separate networks um and what they have is two individual kubernetes clusters completely independent isolated kubernetes clusters running in each of those data centers and then they have a single portworks cluster that spans both of those data centers. Now in there-

Almost an active-active cluster kind of scenario. Exactly. And because, you know, obviously you wouldn't want to do this if, say, one of those was on Azure East and the other was AWS West. That's not a model that we would recommend. In their case, they have, I believe it's sub two millisecond latency between those data centers, which can happen if you're, say, in an Equinix facility direct connected into an Amazon region, or if you Amazon and Azure in Northern Virginia or Frankfurt or Hong Kong where the clouds are kind of clustering. Special clouds where they are coexisting or close to one another. Exactly.

So we have this model where it's basically synchronous replication, which is going to give you that. That's how we can guarantee zero RPO. But then for customers who have data centers that don't meet those latency requirements, we can do snapshot based. It's not replication. It's not snapshot based data protection, in which case your RPO would be a function of your snapshot interval. We support both asynchronous and synchronous data protection.

And, you know, the key there is, again, you know, since Portworx is in the data path, we can optimize those types of capabilities in ways that say that if you're going to use Amazon EBS and the native Kubernetes driver there, you're going to struggle in order to get a volume from AWS over Azure because you're operating at the level of the AWS API. And there's more restrictions about kind of how easy it is to, you know, to it's easy to move things on, but it's harder to move things off. And that's where your RPO or excuse me, your RTO can end up suffering. And I would say 80% of our customers run us in at least two environments. And, you know, it's not, we're not in this world yet where it's, you know, kind of I'm triaging between cloud providers, but people are, you know, we're not in this world yet where it's, you know, kind of I'm triaging between cloud providers. But people are, you know, they have a backup site, they have a primary site, they have some apps that run in one location and other apps that run in another. So they love Google's DNS service. So they have some apps running on GCP. And they love, you know, Amazon's, you know, image recognition, library recognition for AI. And so they run some apps there. You know, so they have multiple clusters and they have to think about, you know, what if I'm going to, you know, recover or of it abstracts away the underlying storage resources in ways that, you into, God, standard backup applications such as, I don't know,

Veeam or Dell DPD, those sorts of things. Does Portworx play in those roles as well, or how does this work? In what roles? The backup? Yeah. So, I mean, as a storage device, I mean, backup would sit there and it would look at, you know, what's been changed or would scan your files and or volumes and, you know, relatively quickly, you know, extract the data and put them someplace else. And, you know, that's a simple backup solution, obviously, but it's in a nutshell, that's what backup does. Because Portworx is

containerized storage on, you know, something like a block volume. I'm just trying to figure out how does backup work with Portwor i guess yeah it's it's a great question and you know there's the way that we do it today and then again you know uh we should talk around kubcon because we have some new capabilities that are coming there as well um you know drop the hint um build um but you know you can think about backup as, well, let me back up, no pun intended, or maybe I intended the pun. We have a capability that we call kubemotion. And kubemotion is basically what we believe container native backup should look like, with a couple of caveats. And those are the things that we're further building on.

But basically backing up a containerized application is two parts. So one is the data itself. And you need to be able to backup the data based on changes. And the way that we do that today is we allow our customers to provide a backup policy where they determine the time interval that they want to back up. And, you know, it's like if you have a very infrequent application where really what you want is, you know, to back up once there's been a change versus once every hour, then we don't have a solution there today.

But we could extend it if customers are asking for that. What customers are asking us for is, you know, I want to back up this nightly. I want to back it up every hour or maybe even every five minutes. You know, if I really want to push a low RPO, in that case, we probably steer them more towards the PXDR functionality versus pure backup. But, you know, so the first thing is, how often do I want to back up my data? But the second part, and this is what's so critical within a containerized environment, is in order to be able to recover that application, not just back it up, but recover it, containers give you a packaging for the app code itself, which abstracts a lot of the dependencies

in the environment. So theoretically, it's easier to spin up my application with that data volume in the new environment. But in order to do that, you also need the application configuration, which itself is state. So when we talk about backup, we're talking about backing up state. One type of state is the data that lives in, you know, on the volume or the file system. Another type of state is the data that lives in, you know, on the volume or the file system. Another type of state is the application configuration, which in Kubernetes takes the form of, you know, dozens of different YAML files. YAML files, right? Yeah. Proliferation of these things, right? Yeah. So what Portworx does with kubemotion is we back up an application, two parts, we, the data itself at the container granular level.

So even if you have hundreds of pods running on a single VM, we can go in and back up just the data of the pod in question because it has its own volume that we can kind of manage independently of all the other volumes running on that host. And the application configuration itself. So we will basically snapshot those things, move them as different objects, but related to each other to the new environment. This is where the kube motion comes in. So, you know, we're going to take the backup, then we're going to move that backup to some other location. Then when it comes

time to recover, you have your data, you have your application configuration, and you have your container image version. So you can spin up that application basically as fast as aeeam, etc. You know, if I were at Veeam, I would be asking myself, how do we do this too? But, and maybe one day they will. But what they have to do is think about not just the data, but also the application running on Kubernetes and provide a Kubernetes native way to do this for an entire namespace or group of pods. And the container image registry would be accessible to both locations. Is that how this would work? Or you take a backup of the registry? So that's kind of the third pillar. And what we found is that if a customer is running containers in an environment, they have a trusted registry that they're using. It could be the public Docker Hub, or it could be our private registry. We don't provide the registry solution ourself.

The customer already has that. Okay, so that's exact. So once you have the YAML file with the configuration of a container app, and you've got the data for the container app, then you can effectively run this thing anywhere in the world that the data happens to reside in. Is that how I understand this? Yeah, exactly. So I'm running in my own data center, and I want to have a backup location in the cloud. Very, very common. And in order to recover my application in the cloud, I'm going to need the

data, I'm going to need the app config, and I'm going to need, you know, a registry where I can pull down my container images. Okay, so, so, yeah, so in this environment with that, now I'll call it, you know, the package of the container backup, is that something, is that written to an object store or is that written to EBS or where does that, so if I wanted to do this with AWS, let's say, and I'm running on-prem Kubernetes cluster and I take the backup, where does it reside in AWS until I actually reconstitute it? So it depends. We have two models. Basically we have something we call CloudSnap, which is where we can snapshot and send the data and the app config to any S3-compatible object store. So obviously, Amazon is one option, but Azure, Google, or an on-prem object store that's running something like Minio or any S3-compatible object store. Or we can go point to point. So if you want to, so, so we call it kube motion because,

you know, backup is one use case for needing to move data and app config around. But another one might be, you know, I just, I need to drain off the apps that are running on this rack of servers and I need to migrate them somewhere else. In that case, you probably don't want to go to an intermediate object store and then pull down. You just want to go point to point, in which case the volumes would be written on a block device running in the new location. So basically, it depends. What about, yeah, so I mean, backups typically, I'll say, compress the data. They might even encrypt it. They could deduplicate it and all that stuff. I mean, to try to, A, secure the data that's being backed up, and B, minimize the actual space consumption of the data at the final destination.

So if you're doing something like CloudSnap, I guess encryption would be based on what the object store provides? The way we do encryption is by, so we leverage the encryption in the Linux kernel itself, which you can plug into any key management store that the customer is using. HashiCorp Vault is a very popular one. And so you can encrypt it in your,

say your primary site, and then you can snapshot it and move it to your backup site and then you would decrypt it in the backup site using the same key that you encrypted it with on the primary site side so we we we basically leverage the customer's key management system and the the keys persist with the volume throughout its entire lifecycle. Yeah, you don't want that responsibility for them. Yes, exactly. Let me try to understand.

So as I'm running the application on-prem with Portworx, the data could be encrypted at rest? Yes. Okay, depending on whether I want to do it or not. It's something I could just enable at Portworx and point it to a key management store in order to generate the key that then would kind of public privacy encryption system. And so, I mean, I'll take this opportunity just to say, again, as a company, we exist to provide the performance, reliability, and security of a traditional enterprise class storage system. Our friends at NetApp, EMC, et cetera, built amazing storage systems. We want to do that for Kubernetes. And so we just go down the list and say,

what's our security story? What's our performance story? What's our backup story? What's our DR story? And we build functionality that provides enterprise class capabilities, but manageable within the way kind of modern DevOps teams are working and the way that there's control via Kubernetes. And as a company, that's our strategy. And so you can think about any important enterprise storage capability as something that is in our purview. Those are capabilities that if our customers ask for, we'll go and build. We have that core storage engineering capability, but for Kubernetes, that's the real thing. And I think our customers are trying to leverage what's in their data center already. I mean, I would do the same thing.

Absolutely. My storage system has a CSI driver. I'm going to try it. Where they end up stumbling is when they try to use those storage systems to do things like advanced capabilities like you know dr for distributed applications within kubernetes or they you know their array might have been built for a relatively small number of manual operations driven by an

operator a human operator um and when they test when POC, it works fine because I'm provisioning a volume, I'm deleting a volume, doing atomic operations to do functional testing. But once Kubernetes gets in the mix and is controlling and is making placement decisions and rescheduling decisions based on its own algorithm, not kind of a human operator, the volume of those operations goes up by an order of magnitude and you end up getting bottlenecks that end up leading to slow provisioning, downtime, things like that. And so they start there and then they say, no, we need a container native solution. They try Portworx and just because we built with those use cases in mind, our assumption was that

thousands of volumes are going to be provisioned and deprovisioned every minute that was just kind of a requirement going into the system um so we were able to architect it in ways that could could meet those that level of scale um it's just you know it's it's a it's a question of what what were the requirements that went into building your system and we're lucky to have always had automation driven by a container scheduler in mind. And so we built from those from day one. That's an interesting story. I would say most storage vendors, you're going beyond a normal storage vendor in this environment.

Nobody would provide, you know, they provide replication. They might provide asynchronous replication, but they don't provide, you know, full RPO kinds of capabilities, RTO kinds of capabilities where they're bringing up apps and stuff like that. That typically is provided by the operating system, in my mind. I look at VMware, it's Site Recovery Manager. If I look at ZOS, it's GDPS. I mean, those sorts of things that bring the applications up are distinct from the data. Well, I think, Ray, in that case, there's an element of that that relies on the physical storage as well. Since we are working with this product, we arerating against a an open storage backbone that the the focus on

on functionality like replication and and dr it would be part and parcel to this but you know that's presuming a lot right i don't know a whole lot of companies that are doing anything along these lines yeah i i you're exactly right and i I think the, so why is that? Well, we think that A, Kubernetes is like, we're lucky enough to have Kubernetes. So just to be clear, we don't spin up the applications. That's still Kubernetes. But what Kubernetes needs in order to spin up the applications is data and application configuration. Like, so, and those, that ultimately is a storage problem. So the reason we do it is because we can, and secondly, because customers want it.

You know, they don't want to have to have, you know, my storage system and then my backup vendor, and then my DR vendor, like they want to be able to use Kubernetes API to manage their application. And that's what we provide. The second thing is we've heard the feedback that you've just given me multiple times, including from some of our advisors, which is to say, you know, you guys are doing things that typical storage companies don't do. And they're exactly right.

But, you know, we're building this company in 2019. You know, most of those companies, if they were starting today, wouldn't do it the way that they do it either because, you know, there's new technologies and there's new expectations on the part of the user. And so I think, you know, we're very customer driven as a company. And so we look to our customers to say how do you want to operate your apps and how can we help and that's what's led us to the path that I've described earlier. Okay, great.

This has been great, Michael. I really appreciate it. Matt, do you have any last questions for Michael? I didn't actually get to ask a whole lot of questions but Ray, you were very complete, I guess. So, no, I don't, but Michael, wow, what a really interesting product and something that I can see

benefiting a whole lot of the customers that I face on a regular basis. Really, anybody that's doing a DevOps environment that's playing in the Kubernetes space should be looking at this. I agree. Michael, anything that you would like to say to our listening audience before we close? Yeah, I'll just say watch this space and never stop pushing. I think that, you know, the people that inspire me at the moment are the, you know, the application developers and the architects who are taking these new technologies and really pushing it into new spaces.

I mean, I'm seeing things now about, you know, Kubernetes orchestrating VMs in addition to containers. And I think that's super exciting and what that means for, you know, the on-prem data center. So I think it's, you know, it's very exciting for me to work in this space. I'm inspired by our customers and, you know, always excited to talk about the technology and the challenges that Pullworks helps address. So thanks for having me on. I really appreciate it. Appreciate that. Well, this has been great. Thank you very much, Michael, for being on our show today. My pleasure. Thank you for having me. Next time, we'll talk to another system storage technology

person. Any questions you want us to ask, please let us know. And if you enjoy our podcast, tell your friends about it and please review us on iTunes and Google Play as this will help get the word out. That's it for now. Bye, Matt. Bye, Ray. Bye, Michael. And bye, Michael. Bye, guys. Until next time. Thank you.