Storage Developer Conference - #89: OpenSDS Flash Manageability using Swordfish for Cloud-native Frameworks

Episode Date: March 20, 2019

Hello, everybody. Mark Carlson here, SNEA Technical Council Co-Chair. Welcome to the SDC Podcast. Every week, the SDC Podcast presents important technical topics to the storage developer community. Each episode is hand-selected by the SNEA Technical Council from the presentations at our annual Storage Developer Conference. The link to the slides is available in the show notes at snea.org slash podcasts. You are listening to STC Podcast, Episode 89. So, I'm Stephen, and this is Reddy, and we are part of the OpenSS project.

So there's some disclaimer here from Intel that I'm going to skip. And this is the agenda. I'm going to be doing the easy part, giving an overview of what the OpenSS project is about, and Reddy is going gonna cover the technical stuff. So, how many people have heard of OpenSDS? Oh, okay, not too bad. So OpenSDS is a Linux foundation project.

It's an open source project, an open source community that's working together to solve data storage challenges. Okay. So the problems that we have, especially with all these cloud-native environments that's coming up, the trends in that, and then all the flash kind of management and stuff like that. So this is what the community is about, I mean, and then all the flash kind of management and stuff like that.

So this is what the community is about, I mean, to solve all these challenges. So the companies that are involved are those up there. So there are vendors. There are also end users that are involved. But I think most important, I think some of the things that I want you to focus, some of the companies that I want you to focus on are, I mean, you can see some of the biggest companies, like Vodafone, NTT Communications, KPN, Yahoo Japan, Toyota. These are some of the largest companies in the world that actually support this sort of project.

Okay. So the goals of OpenSD, we try to narrow it down to three very focused goals. One is to be open, meaning that this is really gonna be a vendor-neutral project. Because end users now, no matter which end user you're talking about, they're using multi-vendor products, they're using different kind of cloud storage services and stuff like that.

So it has to be open in order to satisfy the needs of the end users. It's going to be real. Again, you hear me talking a lot about end users because we want this to be something that end users can really use. So all the stuff that we are doing, the roadmaps, the features, the technologies that we are building are all going to be based on what real end users requires. And what we eventually hope

to do is to have a ready kind of ecosystem of products and solutions whereby if it's anything that you mentioned is OpenSS compatible, that means you can just plug and play and you put things together and they'll just work. So that's our eventual goal. You can look at the OpenSS platform in this way. If you split it into control plane, control and management plane, data plane, and storage

plane, the openness of this project right now is focused on the control and management plane. So what it does is that, I mean, there's the standard RESTful APIs, single pane of management. You have policy-based storage and data services, and also orchestration automation for the platform. So what it does is it tries to enable all the data, like the stuff that's available in the data plane and the storage or service layers, okay, like the high availability, performance,

data mobility, reliability, and making use of all kinds of storage, whether it be on-premise kind of storage or cloud storage. And in terms of technologies, there are three things that we're working on. One is the orchestration and automation part. So how do you enable data protection, data lifecycle management, and replication across different kinds of storage. The second thing that we are focused on

is the multi-cloud data management across the different kinds of public cloud services that's out there, as well as if you have a local cloud storage, I mean, on-premise, we're going to be supporting that too. So, and the last technology focus is actually how do you enable data and storage intelligence.

So for this part, actually it needs a lot of participation because how do you collect the data? How do you analyze the data? I mean, in a standard way. So, and then eventually be able to monitor reports, I mean, create reports, do predictive analytics, and machine learning, and AI kind of stuff. So we want to do that kind of stuff.

So these are the core technologies that the project is focused on. And in terms of projects, there are two projects in the OpenSS community right now. So one is the Sushi project, which is essentially an off-ground plugins project to the different kind of frameworks. So to the left side, the cloud, and for the cloud kind of focus frameworks, we have plugins for Kubernetes that we've already developed, and then we're developing, I mean, we have also

developed a plugin for OpenStack. And then the other frameworks are things that we can do, I mean, to create plugins to connect to all these different other frameworks. And then on the core of the project is actually what we call the OpenSS Hotpot project.

And what this Hotpot project does is it's actually the controller that provides all these storage services, like block, file, object storage, and as well as connecting to the softbound site to the different kinds of underlying storage, as well as cloud storage here. So this is another view of the project. So the OpenSS controller is right here, and then connecting to the different kinds of frameworks here and the cloud integration that we want to do, as well as supporting these different kinds of frameworks here and the cloud integration that we want to do, and as well as the supporting the,

these different kinds of storage. And what we, we, what we're trying to enable is also the extension and the tools for any kind of, for any kind of data center, I mean, to do the deployment, CACD, even things like security, BCDR, AI, and machine learning. So the architecture looks kind of like that.

The project is written in gold, and we use the ETCD KV store for distributed persistence and OpenStack Keystone for the authentication. The project, I mean the architecture itself comprises this Sushi that we talked about. So Sushi basically allows you to, I mean, it actually enables the plugin with the August Trader, so Kubernetes or OpenStack in this case.

And then the Hotpart controller provides the centralized unified kind of management and scheduling using standard RESTful APIs. And there's a Hotpart doc over here, which actually allows the connection to the backend. So it allows driver integration. So what we have done is that we have taken the OpenStack Cinder project and integrated it within OpenSDS.

So wherever there's supported by the Cinder, whatever drivers that's supported in Cinder can actually be used within OpenSDS itself. And we do have native support for SAF. And then there's other custom drivers that we could actually develop, like maybe I mean Swapfish in future. So we also have developed a dashboard.

So this dashboard here, I mean this page, for example, shows how you can create a profile. So the definition of profile, Red is going to cover that later. Essentially, what you can do is when you create a profile, you could specify the QoS kind of stuff, the replication policy, protection policy, and stuff like that. Within a profile itself, it does whatever that's needed

for you to provision a storage and specify the protection and the performance required for it during the provisioning time. So these are some of the end user scenarios, the requests from end users for OpenSDS. So we have not done everything yet, but essentially, I mean, if you look at it,

like number one, the consolidation of storage control and management across OpenStack and Kubernetes. This is very common. I mean, it's like a lot of people, I mean, a lot of end users, they have Open stack environment, and then they're starting to play with containers, microservices, and stuff like that.

So they consolidate the control of the storage layers. And snapshot backups to the cloud, okay? To any cloud, actually. And then how do you restore to any kind of storage backend? The host-based replication, so that you can actually do replication across different kinds of storage backends. Data lifecycle management.

This to move across the different storage tiers as well as to the cloud for archiving and stuff like that. Number five, the multi-cloud data control. Again, to be able to use any cloud. And also the orchestration and automation. Like, for instance, when you want to provision a volume and then you want to specify the connect to a backup software to actually do the scheduling for the backup.

How do you do that? So they are hoping to do something like that within OpenSDS. So all these are real user requirements. So some of the use cases that OpenSDS is able to support right now. These use cases are actually being tested right now by some of our end users.

So the basic operation of provisioning using different backends. And then array-based storage replication. This is also supported right now. But there's a reference architecture using Huawei Dorado storage for this. And for host-based replication using DRBD, DRBD is available in any kind of Linux software.

So you could actually, through DRBD, do host-based replication. This is available right now. And OpenStack integration, this is also available. So we have an OpenSS Cinder compatible kind of API. Compatible because we haven't implemented every kind of Cinder API. We only implemented the core APIs over here.

Okay, so this is also supported right now. And lastly, the Kubernetes integration using the CSI through CSI plugin. So for this part, Reddy is going to cover the integration and talk about Kubernetes and how we can actually enable the management of especially Flash Flourish for cloud native kind of environments. So I'm going to hand over to Reddy for the rest of this presentation.

Thanks, Stephen. Okay. Can you guys hear me okay? All right. So how many of you know Kubernetes, heard about Kubernetes? Okay. I don't have to cover it then. So the rest of the talk is all about northbound, Sushi, expanding on the Sushi layer in OpenSDS, and then how do we take the Sushi component, integrate with Kubernetes.

So there is a second part of the talk which is all about how are we taking the existing SportFish, you know, the standards and using that for the not-bound API. And the third part is really the flash manageability. Very simple, one or two slides on that one and then we'll cover the community aspects too. So if you look at the Kubernetes, there are two major components. The top is essentially the central cluster-wide components, which are called really master servers. You have API server. API server is really your gateway into doing anything from a command line, GUI,

anything and everything you want to control. Kubernetes, you need to use API server. So everything goes through API server. Replication controller essentially makes sure that your parts and replica count is always meeting the goals. And then the scheduler is really the brains behind the deployment of your parts to ensure that you are essentially getting a smooth balance among the pool of nodes.

And the bottom, where you see is the Kubernetes node has, Kubernetes operates at a level where it's a part. So it's not actually going to deploy a container, but it is actually going to deploy a pool of containers. And the concept that Kubernetes uses is part. Part is a collection of containers. Your deployment unit is really a part. So that's what you're going to see in the Kubernetes across the board. So when you look at the Kubernetes

node architecture, you essentially have a bunch of parts, and the kubelet is really a runtime environment sitting on a node to really manage the deployment, monitoring the deployment of containers running on the node. So? So that's really at the node-level architecture. So the theme behind Kubernetes is we should be able to deploy, you know, the microservices using pod abstraction on a pool of servers and manage it that way. So there is a central component called HCD.

Stephen talked about HCD, which is really a distributed key-value store. Anything and everything that's persistent in Kubernetes cluster-wide management is really stored in HCD. So HCD really plays a very critical component in Kubernetes. And then if you look at OpenSDS, OpenSDS architecture kind of takes most of the Kubernetes functionality that's really proven at a scale, and we wanted to start off at a level where it's, we are not going to revisit the scale problems, but rather start from where it's proven.

So if you look at etcd, which follows the, which is really the basic foundation in OpenSDS for the state management, for example, all the, you know, the deployment of the microservices for OpenSDS actually follows the container and pod model through Kubernetes. So we kind of wanted to actually take the advantage of the Kubernetes scale-out architecture

and lay the foundation on top. And that right-hand side is really the volumes. I'm going to click down on the storage management in general for Kubernetes. So when it comes to Kubernetes, the way Kubernetes manages the volumes is called a persistent volume. So persistent volume is really nothing but a volume that is actually persistent in a car route from a pool of storage nodes, storage services, anything. It could be iSCSI, it could be SEF, or it could be local storage, or it could be backed by third-party driver through Cinder, for example.

Any storage abstraction, the basic foundation of storage abstraction is called persistent volume in Kubernetes. Now, a persistent volume can be statically provisioned or dynamically provisioned. Static means admin essentially goes and creates a bunch of persistent volumes. You go and consume them based on the demand. It's not a scalable model, but if you look at Kubernetes, that's how it evolved. Over the course of time, it started with create a bunch of volumes, pool of volumes. You consume them through the concept called statically provisioned volumes by the admin. But over the course of time, it transitioned from static provisioning to dynamic provisioning. The way the dynamic provisioning works is you essentially have to say the persistent volume

claim is really a glue between the users or tenants, whatever you want to call, who will be really consuming the storage services with the underlying storage infrastructure. But you're not going to expose all the storage infrastructure into one massive pool, but rather carve them based on quality of service levels, which are called storage classes. So carve out your storage resources using storage classes, and then you essentially tie the storage class to the persistent volume claim. So that's really the glue between these two. And persistent volume claim typically happens to be consuming a specific class of service that is exposed through storage class and you consume them

through the you know basically the persistent volume claim abstraction on top by the user so this abstraction is very important and we have to follow the Kubernetes abstraction for us to go integrate even the OpenSDS components now what happened was And we have to follow the Kubernetes abstraction for us to go integrate even the open SDS components.

Now, what happened was over the course of the last two years or so, Kubernetes storage integration kind of matured over the course of time. So initially it started off as if you really want to integrate storage backend, you essentially create a driver, deploy the driver pretty much on every Kubernetes node. Very painful to manage because any time you change certain pieces in your driver, deploy the driver pretty much on every Kubernetes node. Very painful to manage because anytime you change certain pieces in the Kubernetes, your driver, you have to go blast off everywhere. So that abstraction, very monolithic abstraction, actually changed from monolithic implementation to somewhat streamlined architecture such a way that you can actually consume the storage resources

and service the storage resources to any container frameworks. It could be Mesos or it could be Kubernetes, it could be Docker Swarm, it could be the cloud foundry based infrastructure, it doesn't matter. So the goal was, through the CNCF work, the goal was to essentially figure out what is the best way to abstract the storage that can serve up container workloads, which is called container storage interface abstraction. Or in short form, it's called CSI.

If you really want to expose your storage resource, you have to write a CSI plugin. That's the way you consume it. Now, if you look at a CSI plugin, there are two abstractions. Anything that you need on the node itself, from a container or a pod that needs to consume a storage resource, that's one element of abstraction that you have to worry about. And then there is a backend abstraction, which is really your storage backend in the driver that actually costs out the persistent storage, right? So if you look at those two abstractions, it's extremely difficult to understand through these terms, but look at this as the node-level abstraction. This is really the centralized component.

So what it means is you write a driver, you deploy the driver in one central place, and you can actually manage the scale of the driver by saying I want one instance to be running all the time. So you can actually use the Kubernetes plumbing to make sure that your driver is up and running all the time to service your storage backend. And then you actually deploy the runtime

on every Kubernetes node. So you can actually mount, let's say, a Cepобщ RBD volume, you know, through that abstraction, and you need to have that runtime on every Kubernetes node. This kind of enables you to deploy that abstraction, and then you can actually manage this seamlessly. So take your monolithic implementation,

split them into two. One is a client-side runtime. The other one is storage backend. This kind of streamlines majority of the storage management in Kubernetes, and that's essentially called CSI, right? Okay, so if you look at the CSI and how do we integrate the OpenSDS, obviously we have to write the CSI plugin for OpenSDS so it can actually consume the storage resources that are actually managed

and orchestrated by OpenSDS via the CSI plugin that OpenSDS provides. So there are two components that OpenSDS has. One is the OpenSDS CSI plugin. This plugin is essentially deployed on every Kubernetes node. So you have essentially a plugin. But this plugin can orchestrate any backend that it is managing. So for example, it is managing commercial storage appliance like Huawei storage appliance, for example. So it will have runtime to really understand that specific detail.

And the driver component actually deployed as part of this container. And it can actually deploy that specific runtime based on the backend that it was really exposing. So it has visibility to the backend, what kind of client runtime it needs, and it abstracts everything through the one OpenSDS CSI plugin. So that's a critical component. So if you don't have this one, what happens is you will have to have a CSI plugin for every storage backend that Kubernetes is going to consume. So you may have Scale.io, you may have Extreme.io,

you may have Dorado, you may have Ceph, you may have any of your fabrics. You will need all those five drivers. And they are actually sidecar containers, so you'll have to deploy those many containers to be able to manage them, right, because they don't talk to each other. So that's one critical component

that actually comes together when it comes to managing the storage backends. And then the dock component is really not deployed on every Kubernetes node. It is only deployed on one Kubernetes node because you don't need, you know, everywhere, which essentially manages all the storage backends. So that's kind of the combination of, you have a OpenHDS CSI plugin on, deployed on every Kubernetes node, and then you have the

doc instance managing the storage backends, deployed on one node. And for high availability, you can deploy it on multiple Kubernetes nodes, but that's essentially one or two. And if you click down on the architecture of OpenSDS within the context of Kubernetes, you will have to expose your APIs through Kubernetes API server so you can actually create the volumes, you can attach the volumes. You can delete the volumes through the CSI layering in Kubernetes that actually flows through the CSI layering and goes to the OpenSDS controller and costs of the storage resources. So if you expand the CSI plugin,

there are actually two different main entry points in the CSI plugin on the Node side. One is called publish the volume. The other one is staging the volume. Staging the volume is really bootstrapping the volume, and then publishing is really mounting a volume. So it's kind of a CSI driver terms that Kubernetes uses, but look at staging as really the bootstrapping, and publishing is really mounting the volume.

And on the back end, you'll have to create a volume, so there will be entry points to create volume. There will be entry points to actually publish the volume. Controller publish volume is essentially saying that, hey, I'm going to mount this volume on this Kubernetes node, and I have this volume that is going to be attached to this Kubernetes node. You can actually keep track of the state of who is attached to whom

by actually using this interface. So you know exactly who is connected to what volume and what backend as well. And then you can actually create the snapshots, delete the snapshots, and all that stuff. So the components that are actually in the blue, these are actually canned sidecar containers. That's what they call in Kubernetes. They essentially give you a mechanism to bootstrap your drivers very quickly. And they use Unix domain sockets as a way to communicate to your containers through the socket, local socket, Unix domain socket, and then there is an RPC server instantiation as well.

So it kind of gives you a very abstracted, easy-to-use way to write the plugins and consume the storage resources in Kubernetes. Okay, so here is the click down on the OpenSDS architecture on the plugin side. Now, the fundamental abstraction in OpenSDS, we kind of vetted this out quite a bit. We decided we are going to use a concept called profile. Profile is essentially, it stitches together pretty much all the key characteristics of what the tenant storage consumers will be looking for. Things like do you need replication, do you need snapshot, do you need specific quality of service like megabytes, number of IOPS per second, min, max, all that stuff. Encryption, compression, anything and everything you can think of that essentially calls out as a line of service within the storage backends, you essentially encapsulate them

using a profile abstraction in OpenSDS. Profile abstraction kind of maps to more of a class of service. It's Fortfish. Somewhat similar, but not exactly the same thing. So when you look at a profile abstraction, we'll have to take a profile abstraction and somehow map it to Kubernetes abstraction

so that actually this whole thing can flow through the layers from Kubernetes OpenSDS northbound plugin to the OpenSDS southbound driver interface to the driver very seamlessly. So the way it works is you create a profile. Stephen showed the GUI where you can actually create a profile. And you can actually, within a profile, you can say this profile belongs to this specific storage class. Storage class is really an abstraction that Kubernetes uses to cardboard storage resources

based on specific quality of service levels. So we'll have to tie in that. So you have a storage class, and then you essentially say, here, this profile uses this storage class. And then you create a persistent volume claim that attaches the storage class. And then it also attaches the plug-in that actually goes and costs all the storage resources for you, which is really OpenSDS CSI plug-in. And then everything else gets provisioned automatically through top

to bottom layers. OK. So, the, switching the gears a little bit on how the profile abstraction really maps into Swordfish specification. As you guys know very well, we have Redfish. This is really, you know, standards-based for describing a platform. Then we have SvortFish, which is really, you know, standards specification for describing the logical constructs for storage resources within a data center to really orchestrate the storage resources. And then OpenSDS is really an instantiation of a bunch of those profiles, and I will go through the details of what they are. Okay, so within the context of OpenSDS profile,

we have currently implemented data storage, class of service, I-O connectivity, data protection are the three profiles that got implemented. There is also replication and snapshot profile as well. So those are the five. There are a few caveats here and there where we felt that certain things are not comprehensive. We also felt that certain things are not inclusive. So there are certain minor changes in these Swordfish resource mapping that we ended up changing. But I would say in a grand scale, things that are pretty minor. In general, we ended up using most of those constructs. And we have taken some subset of the concepts because some of them we did not know what to do with them. So, for example, data security, virus scanning.

We looked at it and said, okay, what can we do? What use case will fit the bill very well? Things of that sort. So there are a few attributes that are actually part of the Swordfish resource profiles that we did not know what's the best way to make use of them. But apart from that, you can essentially say there is a data storage line of surveys which has the recovery time objective. What is the provisioning policy?

Is it thick or thin? You know, is it space efficient? IO connectivity, for example, access protocol is a very important part of plumbing for us to actually carve out the right driver on the Kubernetes node. If you look at the snapshot profile, you know, exactly the frequency of... How do you describe the frequency at which you really need to take a snapshot? It could be a volume that is actually sitting in a profile, and you need to be able to take a snapshot at a certain frequency.

There are some changes that are needed in the Swordfish profile description for us to actually describe the schedule aspects of these snapshots much more elegantly. But in general, it's fairly comprehensive, I would say. And then data protection, recovery, replicas, all that good stuff. And then a lot of work has been done in the replication side. Specifically describing where are you, where are you actually doing the replication? Is it really on the host side or is it based on an array-based replication? How are you actually doing the replication? Is it really on the host side, or is it based on an array-based replication?

How are you actually doing the replication? What is your source? What is your target? What exactly is the protection domain? Is it a data center, or is it a cloud, spanning a different cloud provider, and all that stuff? So this gets extremely complicated

as you keep implementing more and more advanced use cases. So it's fairly a mature stage for customers to do the POCs. And then the migration profile as well. So you can actually move the data from place A to place B using a migration profile. Okay, so how do you map a profile that is using set of Swordfish resource specifications as building block elements,

specifically the line of service and class of service construct. So we waited out the selection part quite a bit because obviously you have storage backend. They really need to be able to bubble up what class of service and line of service capabilities they provide. How do you do the

matchmaking and how do you actually bubble it up to profile that the admin can actually set it up? And then how do you consume them? So I talked about the consuming part from Kubernetes, but the challenging element is how do you describe, how do you describe, sorry, how do you discover those attributes from the storage backends? And then how do you mash them together to be able to really say, here is a profile that gives you compression and encryption to really get you highly efficient encrypted storage. So, we

talked about a rule engine to be able to do this, but in the initial instantiation it is a simple rule-based if equal to, less than, or equal to. So, very trivial implementation to really describe the profiles, but it will get very advanced over the course of time. Okay, so a little bit of,

if you have a Kubernetes type of instance, how do you go about creating the, you know, describing a plugin, and how do you consume the plugin? So if you look at the profile, this profile says it's a high performance profile. Who is provisioning that profile? Who is supporting that class of service? It's really the CSI OpenSDS plugin. And then if you look at the storage class name, it's really a tie-in to the Kubernetes, right? So storage class name

is the only way Kubernetes will consume the storage resources. And then what is the persistent volume claim that actually will consume this class? So you can actually see the tie-in. So this is the driver. It's actually using a high-performance profile. These two terms, high-performance profile is meaningful only for OpenSDS. It has no meaning to Kubernetes.

And then within the context of Kubernetes, it understands that there is a storage class, and you tie the storage class to the plugin in the profile, which is on the left-hand side, and then you essentially create a persistent volume plane in the Kubernetes to be able to go create the persistent volumes. So it's kind of a little bit,

you have to navigate through the profile, plugin, persistent storage class, persistent volume play. So you essentially have a four-step process to really describe your storage classes in Kubernetes. And then when you essentially, you basically say Kubernetes, kube control, create,

here is the Kubernetes, you know, plugin. And then you can actually see there is the attacher, there is a node plugin, there is a provisional. Provision is the central component. Node plugin is the Kubernetes component that gets deployed on every Kubernetes node.

And then there is an attacher that essentially says, here is the volume that got created on this backend, go update your status information. Go ahead. Is the node plugin employed as a daemon set? Node plugin is the daemon set, yep. And the controller and attacher is the stateful set, controller one. So daemon set is essentially what it does is it, everywhere there is a Kubernetes node, it will ensure that your node plugin is deployed. Of course, there are lots of caveats. You can constrain which nodes you want to deploy.

But in general, daemon set is an abstraction in Kubernetes that you can use to deploy your client-side runtime components very easily. Otherwise, you'll have to manage your own runtime, and then if something goes down, you have to bring it up. It's extremely painful. So that's really a Kubernetes abstraction. Okay, once you have the OpenSDS plugins actually bootstrapped, both on the back-end and the front-end,

and described in such a way that the Kubernetes can actually deploy the runtime for the driver instantiation, you can actually consume those resources by essentially saying, here is my claim, persistent volume claim. And notice there is nothing else that is actually deployed. So there is nothing else that is mentioned here that says I need to talk to OpenSDS, I need to talk to this profile, I need to talk to this storage class. None of that.

As a tenant subscriber or a user, Kubernetes user deploying part, all you are saying is here is my claim that I want to use to carve out storage resources. The relationship to the claim, to the storage class, to the profile, to the plug-in is buried behind the scenes. It's one-time configuration. Okay, so you got the theme of the Kubernetes architecture, click-down architecture, fairly high-level, I would say, and then how storage resources are actually consumed

within the context of Kubernetes using CSI abstraction and how we tied in the CSI abstraction to consume the storage resources that are actually called out by OpenSDS through northbound and southbound interfaces. Here is a click down on how we are actually consuming the flash-based storage resources that are exposed through NVMe or Fabrics.

I know all of you pretty much have heard quite a bit about NVMe over Fabrics. I'm not going to bore you to death on this one. But in general, NVMe over Fabrics, look at this as if you have an NVMe-based drive, instead of actually connecting locally and reading and writing from the drive, NVMe over Fabrics will essentially give you a mechanism to reach out over the network, read and write from the drive, without you actually going and heavy lifting on the client runtime through the driver abstraction and RDMA protocol plumbing and all that stuff. Now, the question is, how do we take this instantiation that is actually sitting over the network,

consume those storage resources in OpenSDS? Nothing different. It's exactly the same CSI plugin abstraction in OpenSDS? Nothing different. It's exactly the same CSI plug-in abstraction in OpenSDS. But the way we are looking at it is there are three different abstractions that we'll end up having once it is completely implemented. First and foremost, when you look at the NVMe or Fabrics-based targets, there are kernel targets available. There are probably, you know, you're

going to see targets that are actually exposed by storage, you know, app plans vendors. And there are also backends that are actually exposed, like, with, like, SPDK, for example. User mode SPDK targets, right? So you're going to see different types of backends. User mode, kernel mode, app plans based, whatever, right? So the goal is to be able to essentially discover all those different types of targets as a starting point and consume and ingest them as the managed entity in OpenSDS. So that's really the step number one. The second step is we should be able to provision

those storage resources dynamically. But the challenge is if you look at kernel driver, it doesn't give you a virtual volume abstraction. It is really a physical abstraction. You take a specific slice of a physical device and you consume that physical device. So the backend needs to have somewhat smart understanding that it's a virtual provisioning capable or is it a physical provisioning capable and it has also the resiliency characteristics beyond one storage node or one, you know, fault domain as well. So OpenSDS provisioning will essentially need to comprehend all those scenarios

to be able to really carve out storage resources that are exposed through NVMe or Fabrics all the way to the Kubernetes runtime. So the goal is to essentially understand and provide an abstraction both for logical, physical, with HA, as well as data service capabilities in the target. We currently have a CSI driver that only works with the kernel driver, but the goal

is to essentially, and it's very easy to extend the kernel driver, but the goal is to essentially, and it's very easy to extend from kernel driver to, you know, SPDK or any other, you know, backend, but in general, we wanted to get off with something that customers are asking for and then expand that beyond the kernel driver mode. And then over the course of time next year, we want to really expand onto pooling concept, scheduling, you layer two versus layer three, what is the best way to schedule the storage resources, all the good stuff around it. Okay.

Roadmap. So OpenSTS is fairly new. We started this effort last year. In 2016, I gave a talk on just a pure conceptual, what exactly is the software-defined storage? How do we want to abstract the control plane and data plane? This is really the, 2017 was really the first year where we started the community project

under the Linux Foundation umbrella that Stephen talked about. The goal was we need to be able to have drivers that can actually be part of OpenSDS framework so that we can actually carve out the storage resources. And then we need to be able to integrate with OpenStack and Kubernetes. Without that integration and able to consume at least a few stock drivers, we

are not even in the game. So you see kind of that has been the focus area. And we said Cinder already invested in quite a bit of that effort. Why do we need to reinvest this? So the goal was really to take existing Cinder abstraction, decouple from OpenStack, and consume that in the OpenSDS so that we get the advantage of consuming, taking advantage of all the driver investment that happened over the past five to six years. So a lot of focus on taking the Cinder, decoupling the Cinder, integrating with OpenSDS,

getting the volume provisioning working with Kubernetes, as well as some stock drivers like Ceph and LVM. So native drivers, Cinder-based drivers, get that to work in an orchestrated environment has been the focus. And then first half of this year, most of the focus has been replication side because lots of customers are asking for,

I need to be able to replicate my volume, both host-based and array-based. So a lot of investment on the replication side. And then, of course, you need to have UI, and you need to have a very nice abstraction around profiles to group your resources into certain class of service, you know, storage backends, and then you consume the storage resources.

And then you should be able to enumerate them, and then you need to be able to consume the block services. So that has been really the most of the focus area. And then going into this year, and to some extent early part of next year, we'll be extending the block-based storage resource provisioning to object-based storage resource provisioning. The reason is, so you could have an on-prem object store, but you could have a replica on the cloud. So you could essentially have bucket policies that can actually replicate these two things behind the scenes without admin actually going through the heavy lifting. So this essentially

gives you onboarding type of, you know, architecture for you to actually do this very elegantly. So object-based, you know, access method. Multicloud has been one of the key focus areas lots of customers are asking for. We are going to start off with object-based multicloud, but it will expand into block-based as well as file-based. But the key focus area will be object-based. And then we need monitoring. We need to be able to snap, take replication from a pool of resources. And then obviously there will be a preview for the NVMe or Fabrics-based implementation. And then going into the first half of next year,

extend that to multi-OpenStack. Multi-OpenStack has been one of the key painful use cases most of the customers are looking for. You have a bunch of islands of OpenStack clusters, but you want to manage them together. But you don't want to have each one of those clusters managing the storage resources, but you want to centralize the storage resources. So OpenSDS kind of

gives you that platform. So the goal is to centralize the storage management and then instantiate multiple OpenStack clusters. A lot of other stuff. Okay, so quick click down on the governance. So we have two groups in OpenSDS. One is the Technical Steering Committee.

The other one is the End User Advisory Committee. We consciously started End User Advisory Committee because we really don't want to just look at the problem statement purely as a technical problem. Go and address the technical problem using a bunch of architects and senior software developers and development community, we will miss the boat in terms of what is the reality. So we consciously looked at spinning up end-user communities

so that actually they can figure out what are the top high-priority use cases that we as a community need to go solve those problems together. So you kind of see those two focus areas. And then there is a lot of governance structure behind the scenes on community and outbound marketing and all that stuff, right? Release process, development reviews, and all that stuff. Okay, so we covered OpenSDS architecture, talked about how we integrate the OpenSDS into Kubernetes, touched upon the Swordfish standards-based integration in the northbound,

and some of the pieces that got implemented through the Swordfish resource profiles, and then the flash manageability and the community governance. We would love you guys to join the community and contribute. This is something that we have always been looking for. Come and participate and pitch in whatever you think is the passionate area that you would like to drive and lead the industry.

Lots of interesting opportunities. Like I said, multi-cloud orchestration is something that is extremely green, not just in OpenSDS, but across the board. So lots of innovation opportunities, and this could be your gateway into influencing the world through the innovation. Thank you. Questions? Questions? Multi-tenancy control is something that, yeah, we do support.

So it becomes a natural thing if it is OpenStack. Because if you have OpenStack, multi-tenancy is already baked in and designed through Keystone. We have a Keystone as an add-on module. But the challenge in Kubernetes is Kubernetes kind of delegates, has two different types of layers, right? So the bottom layer is extremely, you know, architected, service orchestrated. Controlling the security, authentication control for the services is very, very tight, but when it comes to the user land, it's extremely weak. So... I'm worried about the...

Namespace abstraction in Kubernetes, yeah. So. Namespace abstraction in Kubernetes, yeah. Right, exactly, exactly. So there is a boundary, yeah, there is a clear boundary that we have to navigate when you bring in the OpenStack components into Kubernetes and manage through OpenSDS, obviously. That element, we haven't completely sorted that out, but in general, the theme is we want to, if it is a pure OpenStack type of environment, we will essentially use the OpenStack way of doing things. If it is pure Kubernetes, we'll do pure Kubernetes way of doing things. But at some point, if we really want to go to the next level and say Kubernetes doesn't solve the tenant, you know, the authentication and the control

aspects, we'll have to bring in the keystone, for example, as one of the options, and how do we consume them is still being vetted out. Great question, though. Any other questions? So you were saying that one of the focuses for next year in 2019 was the virtual volume providers and I'm just trying to

like how does that work, so for example like with SPDK, if we built like an SPDK sender plugin trying to understand the difference between the two, so like if you have a sender driver that allows you to do like volume creation, like in a virtual Yeah, you can. You can. So, obviously, if you have a SPDK driver underneath the Cinder, you could actually use the Cinder-level abstraction and Cinder driver in OpenSDS as a way to consume.

But the certain things, I'm not sure exactly how to do this very elegantly. So certain things like in Kubernetes, you need to know what type of backend it is so you can actually kick in the right plugin on the Kubernetes node, right? So those things will have to flow through the Cinder layer and get the Cinder attributes.

And if the driver is not capturing those attributes, it's very difficult. So you may have NVO or TCP IP instantiation on the SPDK target. We wouldn't know unless you bubble it up all the way through the cinder layer and OpenSDS layer, so we can actually do the right thing on the client. But the goal is

on the client side, we want to really streamline the abstraction, which is really a painful part. And those attributes will really help us. Okay. Did you have to modify the profiles in Swordfish? Yeah, there are certain profiles. We haven't actually figured out what the best, the line of service, you know, the resource mapping, we haven't figured out what's the best way.

What are the use cases that we really want to use? For example, the data security. Data security, to some extent, we look at tenant security as a data security, service security as a data security. But here you are talking about extremely low-level construct, which is virus scanning. So we could not figure out where to modulate that type of resource mapping. So the line of service resource mapping, sometimes they are very nicely articulated at a data center level. Some of them are actually extremely at a low level. So we have to really figure out how to modulate and shuffle this up to do it correctly.

Some of them, we did not. A bunch of those attributes, we could not figure out what to do with them too. So we have that and we have certain resource mappings which we don't even know how to map them. Yeah, so we are actually working with the Swordfish guys, like Eric, I don't know if you know Eric, to actually try to resolve all these discrepancies and stuff like that.

Yeah, so we have an active dialogue going on with the Swordfish working group community members. So is the idea that the sushi will support Swordfish as the one? Sushi will support... Swordfish fit in that. Yes, exactly, the northbound. So Swordfish can actually fit in the northbound as well as the southbound, right?

So northbound, essentially, to figure out what is the best way to carve out your storage resources through the abstraction called line of service, right? So that, and then the click down Swordfish resource mapping will play a role. And then discovery of those line of service capabilities in the storage backends will also play a role, but you need to have Swordfish compatible driver on the backend to be able to discover, otherwise we have to shim it. Right? And then completely a southbound Swordfish compatible driver. So the way I see it is, with NVMe or Fabrics target, we may be

able to get away with custom drivers for target based implementations, but, you know, Ceph and others will need its own. All right. Yeah, one more. Is the prognet support compress or dedup? Dedup? Compress or dedup? Compression dedup essentially will be a set of the line of service data storage capabilities.

So if you can discover them, you can consume them. If you can't discover them, you can consume them. If you can't discover them, you cannot consume them, you cannot provision them. But the answer is yes, as long as the backend provides. All right, thank you. Thanks for listening.

If you have questions about the material presented in this podcast, be sure and join our developers mailing list by sending an email to developers-subscribe at sneha.org. Here you can ask questions and discuss this topic further with your peers in the storage developer community. For additional information about the storage developer conference, visit www.storagedeveloper.org.