Storage Developer Conference - #169: Completing the Picture for NVMe and NVMe-oF Management: Guidelines for Implementations

Episode Date: June 7, 2022

Hello, everybody. Mark Carlson here, SNEA Technical Council Co-Chair. Welcome to the SDC Podcast. Every week, the SDC Podcast presents important technical topics to the storage developer community. Each episode is hand-selected by the SNEA Technical Council from the presentations at our annual Storage Developer Conference. The link to the slides is available in the show notes at snea.org slash podcasts. You are listening to SDC Podcast, Episode 169. Hello, I'm Curtis Ballard with Hewlett-Packard Enterprise, also a member of the SNEA Technical Council. Coming to you today, representing the SNEA Scalable Storage Management Technical Working Group, specifically the Swordfish for NVMe Task Force from that technical working group.

We've been working on creating Swordfish for NVMe requirements and implementation guidelines, and we'd like to share with you some highlights and tips for Swordfish for NVMe implementations. This presentation is a snapshot of work in progress within SNEA, but I would like to point out that SNEA Swordfish is a public specification that you could implement to today. We'll be continually developing Swordfish for some time to come yet as we create new profiles and guidance for different device types and define new properties as people come up with new features that they'd like to have implemented.

If you're not familiar with SNEA Swordfish, it's an extension of the DMTF Redfish specification, using all the same models, constructs, as the Redfish specification does, but extending that and developed by the Storage Networking Industry Association to provide a unified approach for the management of storage equipment and services. In this presentation we're going to give you an introduction to some of the most recent work where we've been adding implementation requirements for specific configurations. We're working to make sure that

NVMe and NVMe OF environments can be represented accurately entirely with Swordfish. So what's been going on in Swordfish in 2021? 2021 has been a lot of filling out the NVMe story. We've been working on NVMe and more NVMe. In Stories Developers Conference in 2020, we introduced the NVMe mappings for Swordfish. Basic objects were mapped at that point in time, and many of the properties were mapped. A lot of the guidance was available then. We didn't have the requirements defined, the profiles defined, but now we have the most common NVMe and NVMe-OF models fully

mapped and documented. There are new profiles since last year for NVMe drives with standard features. These are your typical NVMe express drives you might find in a laptop, workstation, something in a data center. All the basic features that NVMe drives implement are describable with the Swordfish NVMeDrive.json profile. There are some NVMe drives that support advanced features such as nvm sets and endurance groups if you are familiar with those NVMe express capabilities. Those are described in NVMe drives with advanced features which use the Swordfish NVMe Drive advanced

features.json profile. Another profile has been published for an emerging technology space of native Ethernet-attached NVMe drives in the SwordFish NVMe Ethernet-attached.json profile. This is a technology space that SNEA has been actively involved in for some time, with the SNEA Object Drive Technical Working Group working on a native Ethernet-attached drive specification, describing many of the enabling technologies to enable this emerging technology space. One of the most recent things that's been published is a profile for arrays that expose NVMe-OF host interfaces.

This is called the Swordfish NVMe Frontend.json profile. We have an array example in the mapping guide called the opaque array that uses this NVMeOF host interface in this profile, but the backend is an opaque, hidden backend. Only the frontend is described and the resources on the backend that are used to make the logical devices are not described. Kind of like a typical array often hides the exact back-end implementation. There are future profiles under development. We're actively working on a profile for NVMe over Fabric JBobs, just a bunch of Flash, the follow-on to the SAS JBODs that you may be

familiar with. Also working on a profile for a native Ethernet NVMe OF JBov. This would be a NVMe OF JBov that is composed of the Ethernet drives that we described in the previous page. Yet another profile that's under development is a profile for logical subsystems that are composed from underlying resources. That profile is not final named yet. We'll have a name for that one soon. That's intended to align with a technical proposal called TP-6011 that's under development in NVM Express, also looking at logical subsystems composed from underlying resources. Please come join NVM Express if you'd like to get more information about that work. Now all of the things that we're continuing to try and describe in these future profiles could be implemented today using the Swordfish specification and some of the mapping guidance in the SNEA Swordfish NVMe model overview and mapping guide,

and using some of the other resources that are available to you. We just don't have the final profiles and all of the requirements specified, and we're working on creating the rest of the material to make it easier for you in the future. If you're interested in what else is new, there's more details on all the new Swordfish features in an SDC presentation from Rochelle Albers called What's New in SNEA Swordfish in 2021. And if you're just watching the video of this, I've copied the entire link there that you could copy out of a video. If you download this slide, all of the links in here are clickable. We said we talked about Swordfish requirements. Where do we get these Swordfish requirements? That ties right into the question of what are Swordfish profiles? Swordfish profiles are where you go to look for your requirements.

If you're like me, I came to Swordfish from the NVMe side. I've been working on host interfaces for over 20 years, worked on storage arrays for a great number of years, been a member of the NVMe technical working group, helping to find NVMe expression specifications for several years. I have not been a Redfish developer or a Swordfish developer so when I started working with this Swordfish for NVMe group and got a copy of the Swordfish bundle as I downloaded it and zipped it there's all these profile files. What am I supposed to do with those? Well, I'll try

and give you some tips. These profiles are a template for a category of device. Now I can describe every device out there. We've tried to segment devices into common categories. Ideally your devices will fit pretty closely into one of these categories. If you've got a completely new category of device we haven't thought of, give us a shout. We could probably describe it as well. These profiles specify the Redfish, Swordfish implementation requirements. And of course, as usual, your underlying architecture is going to be open as long as requirements are met. There are some tips in the NVM Express mapping guide

that we're going to be talking about a little bit later those are places that you can go to get the information to fill in exact properties that that's guidance but your implementation is open as long as you fill in the correct information these profiles provide a common baseline for clients clients know what to expect minimum set of features and targets have a, minimum set of features, and targets have a reasonable minimal set of features to implement. There's lots of optional additional features that could be implemented, and you can take a look at the different properties available in all of the different objects and see things that could be interesting for your implementation. These profiles are published in a human readable JSON format.

As with anything like this, it can be a little bit cryptic, but it's all human readable with a few tips. Now for the purposes of this presentation today, I just used your simple Vim text editor with built-in Vim highlighting to create the screenshots that you're going to see. There's a lot of more capable tools for implementing JSON based code, implementing Redfish, Swordfish. I went simple for this presentation. The full profile specification is defined in a document from DMTF called DSP-0272 and here's a link to all the published versions of that DMTF document. A quick start on how to read profiles. The header describes the file and where it came from. I've

grabbed the header out here. We've got a schema definition that says what schema this profile is compliant with. We don't have to worry about those for the level of detail we need today. The profile name that I've grabbed out here is the Swordfish NVMe Front End. This is the profile that we talked about a few minutes ago that is for NVMe over Fabric front end host interfaces. And you'll see one of the field entries in the header is the purpose that describes exactly that. We use this for an NVMe front end interface such as a storage array. There's also your typical profile version and then a owning entity and a contact info. In this case both of Mercenia.org. At the top of profiles there may be requirements for other required profiles. This particular profile has a conditional requirement

that you have to support either the Swordfish discovery or the Swordfish co-Discovery profiles and so they have a requirement any of these additional profiles must also be supported. Some of the key concepts beyond the header, empty properties by default are mandatory. These are some of the most interesting ones. All of the properties that just have simple parentheses or brackets after them are mandatory. I've copied a couple out here from a category called identifiers. It has property requirements of durable name and durable forming. Those properties are fully specified in other specifications, but there's no special requirements on exactly how you implement as long as you comply with the specifications. The requirements may have some property requirements embedded inside of them. For example, I pulled out one here called the NVMe version.

The NVMe version is required when the underlying NVMe implementation is revision 1.2 or higher. So there's a conditional that you only implement it in these scenarios. So it has a read requirement that if implemented, it must be readable. Properties may also be included simply to restrict or advise against implementation. There's a number of properties that don't make sense for NVMe devices. And those are called out in profile as well. I've called out an example here for asset tag. The asset tag in this case has no requirement, and the purpose tells you that this requirement is recommended

as a do not implement for this device and protocol type. I went beyond that and gave you a little bit more information that this information is available in the drive resource if needed. So an implementer seeing this can see, well, I'm not supposed to implement this because I'm implementing it over here. And it's more useful in another place. If I'm remembering correctly, I grabbed this asset tag out of the volume. And the volume tag in NVMe terms maps to a namespace. And there are multiple namespaces in a drive. The asset tag is much more interesting on a drive that can have multiple namespaces

than the namespaces themselves. Some other key concepts. Your requirements can be an if-then conditional. If there is an if-then conditional, there will be a comparison that specifies the test. In this case, I've grabbed out a requirement for attached volumes. It has a conditional requirement of attached volumes must be implemented when the controller type is IO. Do not implement when controller type is discovery or admin. And this purpose statement is a descriptive text purpose to make it easy for the reader to understand the purpose. In this case, an I.O. controller for NVM Express is the controller

that can have namespaces where data is stored, and those namespaces are just presented in Swordfish as volumes. NVM Express controller types, discovery or admin, don't have namespaces. They're not going to have Swordfish volumes. It doesn't make any sense to implement attached volumes if you're a controller that doesn't have any volumes. So the exact comparison that you should run is described in detail following the purpose.

In this case, the property that you need to look at is the controller type property. The comparison type that you need to run is an equals and the compare value is IO. If the controller type equals IO, then you have a mandatory read and mandatory write requirement. So what about optional properties? I don't have any screen grabs here for optional properties. We haven't talked about optional properties. Well that's because all of the properties not listed are optional and have no special requirements. So the profile guide kind of goes to both ends of the spectrum. It lists the things that are required and lists the things that says do not

implement. Everything in the middle is an optional property that you can see in the specs or in the case of Swordfish for NVMe the mapping guide lists out all of the optional profiles and gives you a good good document you can go look for for additional things you might want to support for more information on profiles, there's a helpful DMTF presentation on reading profiles linked here and the full link described there as well. This is a few years old, however I did confirm with the author of that presentation that that is still an accurate presentation for the current version of profiles. There's the full specification for DMTF interoperability profiles.

We've mentioned this one previously, and there's that full link called out for you. And there's also a DMTF presentation posted on YouTube giving a talk on understanding reading profiles that was given at an OCP summit using a variant of the slides from the first bullet up above and there's the clickable link in that full YouTube link there if you want to look at that. So another helpful resource is the Swordfish NVMe mapping guide for short. We always call it the mapping guide. Full name, Swordfish NVMe Model Overview and Mapping Guide. What are we supposed to do with that?

Well, the NVMe mapping guide is a guide intended to help explain the mapping between redfish, swordfish objects, and NVMe concepts. It's kind of intended to be something to help bridge the gap from NVMe to swordfish or swordfish to NVMe. Help somebody come in from either direction to understand the material. Starts out by introducing the basic NVMe mapping model. We'll take a look at how that's introduced here in a few slides. Then it gives some examples of mapping for some common NVMe subsystem types, so you can look through those for some interesting examples.

And then there's lots of pages and pages and pages and even a few more pages of Redfish and Swordfish to NVMe concept mappings. Lots and lots of hours of work representing those mapping details. All of those properties were reviewed and documented and intended to make it easy for the NVMe developer, the Redfish Swordfish developer, to go from the thing you understand to the thing from the other domain. So let's take a look at how some of these things in the mapping guide are described.

So the basic model, there's a number of documents in that mapping guide that use this basic model. It can be a little bit overwhelming at first, but it's really helpful once you're familiar with it I've had a number of people ask me how they're supposed to interpret this this model so we'll take a couple look at some slides and dig into a little bit further zooming in on just part of that mapping model I try to make it bigger here so you can read it and zoom the legend even larger yet. All of these figures are set up with the redfish swordfish objects on the left and the Nvimi objects on the right and lines between them to show how they all tie together. If you look at the legend here you'll see that the green lines between objects says these are equivalent objects. The

blue or purple lines show a relationship and the dashed line indicates that there's a link between the objects. So if you're an NVMe developer you want to find your NVMe object on the right use those relationships lines and you can map that to the Redfish Swordfish objects on the left. Conversely, if you're a Redfish Swordfish developer, you want to go to the other direction and start with the figures on the left. The major model objects are shown in the figures, and here's an example that I pulled out from the simple SSD model. The figure here looks a little bit different than in the document because I've scrunched it a little bit to fit on the page.

And, of course, zoomed the legend for readability again. So if we take a look at a couple of examples here, the NVMe namespace, roughly bottom center, it has the green line that is connecting over to volume, indicating that the NVMe namespace is equivalent to Redfish Swordfish volume. It has a relationship line showing that it is part of the NVMe subsystem, and it is linked to an I.O. controller. In NVMe Express terms, we would say that it has been attached to an I-O controller. Going and looking at the I-O controller, we see that the NVMe I-O controller has a green line over indicating it is equivalent to a Redfish or a fish controller.

It is part of an NVMe subsystem and it is linked or attached to a namespace and it has the additional linkage of being a physical element. The relationships that you see here for the NVMe side are of course copied over and very similar on the Redfish Swordfish side where you see the controller it has a relationship being part of storage and it is linked to the attached volumes. Digging deeper, the lower level properties can't be shown in a graphic or at least one you can't read without a really big screen. We can't fit them into the mapping guide. So the mapping guide has a lot of descriptive text to make it easy for you. It's been written from the Redfish Swordfish perspective.

So it's important to remember you're coming into written from the Redfish Swordfish perspective, so it's important to remember you're coming into it from the Redfish Swordfish perspective. If you're an NVMe developer, that's a little getting used to, but it's usable. So you want to find the Redfish Swordfish object or property that you want to describe. Most of these objects are existing objects that are just properties that we just mapped to. Most of these objects are existing objects that are just properties that we just mapped to. There are a few things where we had to create some new NVMe specific properties for new NVMe concepts. Once you've found the object or property you want to describe, you look it up in the mapping

guide and most of the objects or properties have a mapping to NVMe with pointers to the specification location for where to get the information you need from the NVMe Express specifications. These current mappings refer to NVMe Express 1.4, specifically I believe they refer to 1.4a, which has some edits from an engineering chain notice. If you're an NVMe Express developer or been reading the latest news, NVM Express has released the NVM Express 2.0 specification bundles. This document has not yet been updated to refer to the NVM Express 2.0 specifications. We do plan to get that in a future

revision. I find the list of tables on page five a good way to get an overview. All of the properties are available, have a table that describes the property, so they show up really nicely in the list of tables on page 5. We've been thinking about some ways to make the document a little bit more easily navigated and there may be some improvements in the future that provides some navigation aids in the document. So from examples in this document, going back to the JSON profiles and looking at the Sorteficie and Bimifreni in JSON profile, I grabbed the volume and the volume has two property requirements for allocated bytes and consumed bytes. I've

highlighted the consumed bytes here and we're going to look at how to fill out the consumed bytes. If we look at the way that these properties are stacked, we're in the property, the property requirements underneath the property are capacity, then data, then consumed bytes. So we call that property, or capacity.data.consumedbytes. So how do I number them? How do I map the number of bytes used? We found our property, capacity.data.consumedbytes. Looking up in table 5, we see that that shows up on page 211 table 127. And looking at other index information inside the document, we can see that that is in section 6522, which is part of the namespace. We grab volume,

volumes, map to namespaces, and so we've got the right information. I did put an asterisk on this because there are multiple NVMe objects that have the concept of capacity, so this consume bytes thing shows up in several places. We're specifically interested in the one that's part of the Swordfish volume or the NVMe Express namespace. Pulling out just a couple of the key values from that section, the description, the number of bytes consumed in this data store for this data to type, and mandatory, yes, which matches the profile which had a list that is mandatory by nature of being included with the brackets. So how do we fill that out? The top there, I've grabbed a screenshot

from the table showing the property capacity..data.consumeBytes, and over on the right-hand side, it says that the matching NVMe spec property and field is called the namespace utilization, or NUSE, often called the NU's field in the NVMe specification. The NVMe Express specification section and figure is NVMe 1.4a, section 5.15.2.1, identify namespace figure 247. Walk you right into the NVMe Express specification to the exact location you want to be. If you've implemented for 2.0 that's easily transposed over so that's we've tried to do that everywhere possible points you to the exact NVMe construct where you could get this information you may not fill in the

information directly from an NVMe construct you may have some sort of an internal database and use the same values from that database that you would use to fill in the values for reporting over an NVMe host interface. But this helps you see that they're expected to be equivalent values. I do strongly recommend you read the notes too. Sometimes they have very important information. Over in the bottom right hand corner I've copied a screen grab of the notes from this one. It's not really very readable. This particular one doesn't have a lot

of additional information, but there are some that have very useful information. For example, some properties do not map directly to values that are reported in NVM Express, and there's information on how you may combine a couple of NVM Express concepts. Sometimes run some math, sometimes just add a couple combine a couple of parameters sometimes the parameters come from other places so read the notes as well so now you can combine the profiles in the mapping guide start with your baseline profiles from the baseline requirements from the profile then fill in the properties using guidance from the mapping guide, review the mock-ups from the mapping guide and http://sarfishmockups.com

for additional recommended properties for some common configurations, and review the properties and reach-in view me object in the mapping guide to identify optional properties that you might want to expose. For an example of how it all comes together, there's a Storage Developers Conference presentation from Phil Caton on using SORTfish for creating and exporting logical NVMe subsystems called Managing Exported NVMeOF Resources and Fabrics in SORTfish and Redfish. Now this has some useful examples showing how everything all comes together.

If you'd like to get involved or get started, here's all of our standard links. Everything we talked about today comes from the Swordfish Standards, available at the link in the top left. The DMTF Redfish Standards, mentioned in the bottom left, are also critical to support of swordfish. And of course, don't forget to visit nvm express and your nvm express specifications link there on the bottom right and there's a number of other useful resources and groups that we work with for snea swordfish please do take a moment to rate the session your feedback is important to the to us i can tell you that the snea technical council spends a lot of time reviewing feedback and trying to make next year's event better than this year's event every year, and we do appreciate

your feedback. Thanks for listening. If you have questions about the material presented in this podcast, be sure and join our developers mailing list by sending an email to developers-subscribe at snea.org. Here you can ask questions and discuss this topic further with your peers in the storage developer community. For additional information about the Storage Developer Conference, visit www.storagedeveloper.org.