Storage Developer Conference - #75: Deep Sea Fishing: A Swordfish Deep-Dive

Episode Date: September 18, 2018

Hello, everybody. Mark Carlson here, SNEA Technical Council Co-Chair. Welcome to the SDC Podcast. Every week, the SDC Podcast presents important technical topics to the storage developer community. Each episode is hand-selected by the SNEA Technical Council from the presentations at our annual Storage Developer Conference. The link to the slides is available in the show notes at snea.org slash podcasts. You are listening to SDC Podcast Episode 75. All right, so my assumption for this presentation is that you've just sat through the last two presentations. So I don't really do a lot of explanation about what swordfish is at this point or redfish or anything like that. So we're going to mostly just jump right into things here.

Let's see. So, again, yes, building on the concepts presented in both the intro to redfish and swordfish sessions. What we're actually going to talk about is more, I'm going to jump into a couple of different areas here. One of which is the kind of the overall storage system and storage service concepts, but I really wanted to dig into a little bit on the class of service and give an example of how that works. This is an area that is kind of critical to the system, but you guys get brand new material today.

We have not covered this before, so George is going to be sitting in the back critiquing me. The other thing we're actually going to do, and John talked a little bit about this first thing this morning, is I'm actually going to talk, and this is kind of a precursor to the implementer stuff a little bit. I'm actually going to talk quite a bit about the CSDL and JSON schema formats and just kind of show schema in quite a bit more detail.

What are the schema? How are they constructed? Because one of the things, once you start looking at the spec, you actually start digging into the schema quite a bit. So I'm going to talk a little bit about how those are constructed and what they look like and the two different formats and comparison of them side by side a little bit. So that's pretty much what we're going to cover now. And we'll see if George can actually comment, whether he'll actually comment or not on the class of service stuff.

Standard disclaimer, and also the website. Again, cna.org slash swordfish. Oh, I'm sorry. I didn't put my three notices. Sorry, Linda. The bowl is once again empty. So there is once again a giveaway. Another Echo Dot.

So enter again for the Echo Dot business card or your name and contact information. And make sure you put that in there at the end. This is a smaller group. You have a better chance this time. So the glass bowl up here, Linda will take that around and make sure you get your information in there for that. Second, again, reminder, Twitter contest, no one is tweeting apparently apparently, except me,

and I don't count, so I can't win. So hashtag STC17, hashtag storage management. Apparently, all you've got to do is have one tweet out there. So, and what was the third one? Oh, take the coasters and the uh you can take the coasters with you and the two little the swordfish cards the little swordfish cards have the same information on the little bit of a blurb on there um you're welcome to take those with you all right standard disclaimer and reminder of the pointer to snea.org slash swordfish.

All right. So, you know, I said I wasn't going to do a lot of recap, but I'm going to give us a starting point, the resource map. We talked about, you know, here's the basic Redfish resource structure. When we add swordfish, build, come on. There we go. When we build, we focus on and we add the storage systems and storage services. And so let's talk a little bit more about those storage concepts.

We're on slide nine already. We're just zipping right through here. All right, so what are storage systems anyway? We said this a little bit already. But storage systems, again, it contains those computer system instances that are used for storage applications. And I mentioned this in the last presentation, but if you actually go look at the scheme and look at the spec, those are, the storage systems are actually exactly the same thing as computer systems.

So what you'll actually see when you do a real implementation, if you're a real boy, you will actually see all of those storage systems actually show up in systems as well. So there's a couple of different kinds of implementations you can do.

You can do, you know, if you're doing like a, think of this as, you know, host-based management if you're doing, you know, that kind of management. Host-based management that's got a lot of different systems as opposed to embedded management. You might see a system up here that's got servers and other things in it as well, this storage system collection down here would only have your storage systems in it. If the only thing you're managing is storage, these systems would be identical. This becomes relevant when there's also another class of redfish management ecosystem,

a redfish service that we call an aggregator, because what you can actually do is build a redfish service hierarchy. So an aggregator that's building up a management, so swordfish that manages swordfish or redfish that manages, you know, based on different kinds of systems. So you could have a storage system that's built out, you know, by pulling in servers, you know, it's composed. I hate to use the word composed because it's not really using a good composability system that John talked about earlier. But if you take my word composed and realize it's not that same one, it'll work.

But it's comprised of, you know, servers. And it's using that Redfish information or even that swordfish information from different systems to build a bigger swordfish system, you might see things in those different structures. I'm probably going into too much detail there. I'll stop. But kind of the key point is you'll see it in both spots. You will see people's implementations most likely telling you,

look in storage systems to find them. Okay, storage services, again, we'll spend most of our time here. So the storage service instances manages all the different storage-related functionality. Different implementations can choose how many different storage services they want to instrument. They can support one storage service per storage system, or they could support many. It's going to be kind of implementation-specific. There are clearly some storage services or some storage systems you can look at today and think, yes, those will clearly support lots of storage services,

other kinds that it just clearly makes sense to only support one storage service. It's going to be completely implementation-specific as to how many storage services they choose to support. This has a redundancy element built into it, so we support implementations already working either way. Okay, so I'm kind of diving down into things that I didn't intend to,

but, you know, hey, that's okay. So the storage service, again, we talked about this already, you know, class of service volumes, file systems, storage pools, all of those things we saw as we were navigating around in the mock-ups earlier. So let's take a more specific look at class of service. What is this thing? How is it structured?

How does it work? So what I did in here was I put an example together that shows the very simple case of this for you because this thing can be structured in a very complex model as well. So classes of service are fundamentally composed of lines of service. There are five different kinds of lines of service. So there's data protection, data security, data storage,

I.O. connectivity and I.O. performance, and all of these have multiple attributes that you can associate with them. And so the way that you specify what properties are available for each one of these are in capabilities. So what is your system capable of? And so, again, it's kind of up to the implementation to say how flexible it wants to be. So if an implementation just wants to say, I'm going to just predefine a set of classes of service,

you won't need to see all this complexity. But if it's a large-scale system, it's going to be deployed with a lot of different flexibility to it, you might see a lot of detail here, and it might populate a whole bunch of capabilities so that a storage administrator could come in and configure a whole bunch of detail here. So what I'm going to show you is actually something that could be very tiny. So the hierarchy here is kind of like you have this, again, you have this class of service that's comprised of lines of service, and then this detail down here where the definition of what you can specify for the lines of service

may be specified in those capabilities. So, again, very, very simple example here. If you have a system that says I want to use a single property to define class to define class of service so I want to go in and here here is one property I go into the IO connectivity capabilities there's a property called supported access properties or support called supported access properties, or sorry, supported access protocols. What that actually translates to is it's actually connection type, but it really translates to drive type, right? So this is things like SAS, SATA, NVME. It's got a bunch of other things in there too, but when you actually look at it,

for these purposes, we can translate this to say, that's actually going to tell me the drive type. So what I can do is actually I can go in and I can create a line of search, two lines of service that says all it is is this property. And I can create a line of service that says, I want a line of service that says it's NVMe. I want a line of service that says it's SAS or SATA. Right?

And that actually says NVMe, I know that these are fast drives. These other ones aren't effectively, right? So high-capacity drives, high-performance drives. Right? So high capacity drives, high performance drives, right? So there I've got that piece done. So here's our little eye chart. So this is kind of what I did here. I put them both on the same one, hopefully, so you can kind of see this. So here's my one class of service named high capacity.

Let me see if I can point it this way a little bit. So over here we've got high capacity, and I created this line of service and specified this line of service down here where that same property, the access protocol, is a SATA. And over here, we've got another class of service. This is high performance, and its access protocol is NVMe. So, again, super simplified example. You know, you could, but you get the idea, right?

I can use, I could get this as simple as one property that I could use to bubble this up to say, I want my system to have two classes of service keying off of one property. Okay, well, great. Now, how do I use it? You know, what do I do now? How about these two classes of service? So I've got classes of service defined. And so I, turn that off.

So once the classes of service exist, whether I pre-built my system that way or whether I went and created them, then I use those classes of service when I create my storage pools. So there's multiple ways you can actually do this, right? So I had a question earlier when we were looking at reviewing these slides about what are we going to actually ask the implementations to do here? Are the implementations going to always create the lines of service and the class of service.

And the spec doesn't actually dictate this. So one of the things we'll likely do as we start running through some of these initial implementations is start working on some best practices for this and try and come up with some guidelines to say, you know, what should, you know, what do we recommend best, you know, implementations to do? Should implementations generally have

a couple of classes of service, you know, at least one class of service, a default class of service that comes out of the box? We don't require it now, but what's the right best practice? Should they have at least one? What would we recommend? I'll just head that question off because I know it's likely to come.

Okay, so when you go to create a class of service, there needs to be, you need to specify, or sorry, when you create a storage pool, you need to specify then a class of service, whether it's, you know, the default one for the system or you pick one. And then, so you can go create multiple storage pools and have them all ready, or you can just do one, doesn't matter. And then when you go to allocate a volume, you use that attribute. You can use that class of service attribute to pick from either the one pool you know the pool, or from the set of pools,

to say, you know, pick which one to use, right? So we created that high performance, high capacity one. I can just say, hey, pick a pool that's got that high performance class of service and I need this much capacity. Or if I happen to know that pool Boston Performance, you know, I created as a high-performance pool, use that one. So there's lots of different ways, and it's kind of up to the, you know,

the specifics of the configuration for the naming and how you want to do that. So here's kind of a quick example of this. I actually pulled the template for this from our user's guide. I haven't yet mentioned today our user's guide. One of the things we released in conjunction with our specification was a user's guide which has a whole set of example use cases from a client perspective for, what do we have, about 20-ish use cases in there

for things like allocate volume with a default class of service, allocate volume with a default class of service, allocate volume with a specified class of service. So just to give folks an idea of how to interact with the Swordfish system. So this is one that would basically be, okay, how do I create a volume? So here's the post, and here's kind of what a body of the post request would look like. So I specified its name, its capacity, and the class of service that I'm looking for. That's about it.

So none of that bothering with RAID levels, which drives to use, all of that kind of stuff. We didn't see any of that in any of this process. And the user's guide provides more context for how this whole use case works, but it's really pretty straightforward. Yep? What would you get in the response? Only HTTP code or anything else?

I think you pretty much get a success and potentially a pointer back to the volume, depending on if it's created right away. And that's implementation-specific. You might get a pointer back to the volume, depending on if it's created right away, and that's implementation-specific. You might get a pointer to a task if an implementation takes a while to create the volume. So it's one of those two things.

It's either a task pointer or a pointer to the volume itself. Yep? Yeah, it's in side parentheses. I'm just doing the list. Those are it's inside parentheses. I'm just doing the list. Those are essentially like indexes. You have storage services one, for example. Yeah, so...

Is the one in index that the client site knows about, or where did that come from? So these... George, you want to explain the OData format there? There's two ways to specify keys, but they're keys. They're keys, okay. Okay. One's probably unfortunate because it's a small number.

If you think about it as an index, but in a real system, it's likely to be a UID that's real big. The high performance there, that was a term defined by the vendor implementation. That was that same class of service that I had in... Yeah. Yeah. So if that's defined through this process, that can vary from system to system, is that correct?

Yep. Yep. So the single manager of this whole thing basically has to drill down and decipher kind of what these different things mean if we're trying to provide a unified view of multiple systems in a different number. Is that correct?

Yeah, but those names can be, well, some of this may... For the names you aren't getting. Yes, yeah. So the names could be user-defined names depending on the specifics of the implementation, right? Some implementations may come with predefined classes of service. Some may be entirely user-configured. Okay.

Let's see. All right, so that's, you know, kind of a quick over, quicker than I, went through that faster than I expected for some reason. So, all right, so let's jump into schema a little bit. So when you're actually starting to interact with and work through the spec and deal with, you know, looking at implementations, there's two different ways you'll be looking at schema, right? If you're building a server-side implementation or if you're dealing with from a client perspective. But let me just kind of start walking you through how the schema

is constructed and what it looks like. And there's, we'll talk about this a little more in a minute. There's actually two different variants of the schema. But the schema is we fundamentally develop the schema in, as John pointed out earlier, in CSDL, the Common Schema Descriptor Language, which is developed by OASIS. And it's that standard schema language. It is not – Redfish, actually, and Swordfish, therefore, use a subset of the full OData specification, but it does conform – the portion we do use conforms to the OASIS OData spec.

So there's several items here, and I'll try and remember to highlight them all when I go to the schema. I tried to put little highlights in to remember every point. I may miss one, so bear with me. So there's a lot of similarities to other languages here and other mechanisms for defining these things. There's one file per schema type, so volume, pool, class of service. Each one of these things has one file for each one of these.

So we have 29 or 30, something like that, schema files for Swordfish itself. So each one can pull in external definitions. So, you know, clearly storage service will, you know, have includes for all of the various things that are in its hierarchy, so volumes and storage pools and storage groups and everything else. So there's an include format, just as you'd expect. Each response contains that OData type, and that provides that top-level decoding.

So there's a way to kind of define namespace. Everything inherits from this base type of resource, and then when you're looking at the namespace, it'll be, you know, something that's like here, you know, volume.volume or volume.version. Volume.version.volume shows us a volume entity of a particular version in the volume namespace. And so there's kind of two versions here.

Like I said, clearly when you're doing implementations, you're going to be, you know, developers are really going to be dealing a lot with these schemas. Purpose-built clients may not be reading the schema directly. Generic clients probably actually want to in order to be doing formatting and dealing with everything properly. In the last session, I talked about our basic web client. It is actually the way I've been working with the folks building that is I am actually having them read the schema

in order to be able to know how to format things. So, for example, I'm having them look at the annotations to say, is this property read-write or not? They're not looking at the specifics of everything. They're just looking at the annotations in order to be able to determine, yep, this one's read-write. Okay, I know how to mark that one. So that kind of thing is right.

So this is the same kind of guidance here. So, and John mentioned this as well, all of the standard schema files are published or republished on the DMTF website. So I know one of the things folks are used to seeing from SNEA is that we put all of the stuff on a SNEA website. Well, that's actually different with Swordfish. One of the things we're doing with Swordfish is we're putting our schema files as the primary place we point stuff on the DMTF website.

And the reason we're doing that is to make it more convenient for users, clients, developers, anyone who wants to see the schema. So the primary place we publish them from a legal perspective is on this Neo website, but we put them in a zip file. We don't ask people to say, okay, go to DMTF to get all of these schema files and then come over here, and then if you're a vendor, go to your own website. We highly recommend everyone put them on the DMTF website. And there are guidelines for companies, SNEA and everyone else,

to say you have to have a primary spot on your own site where you publish your files, but we highly, highly, highly recommend that everyone, when you do your implementations, republish through the DMTF portal and put them there because it's so much easier for clients to say, go to one website, find all your schema files there. So that's what we're doing.

We're recommending to all of our Swordfish vendors that, as folks implement, that they also publish through that portal. We don't have any actually live and real implementations yet that have shipped, so there's none out there yet, but we expect to see that when we do have some implementations, hopefully next year. So not to say also clients and services

may also have local copies. So we do expect to see a lot of implementations where people are in dark sites, right? Or have embedded instances or whatever. So we do not say only put things out on the DMTF website.

It is also good best practice to have a local copy. Every instance of every schema, including OEM schemas, OEM being the name, the branding, or the type, I should say, for every vendor-specific schema will be versioned. So they will be identical. You can have a local instance of it as well.

So there's no issue with having those be different because they will all be versioned. Oop, did I miss a page? Okay. So schema file types. There's two. They're identical. This is completely up to the developer which one they want to use. But we do all of our design development work in the CSDL and the XML versions. And then we use tools

developed by Redfish, not by us, to convert those over to JSON. Those files are identical. I will show you an example of that in a little bit as we walk through what the schema files look like. Again, I already said, you know, the common schema definition language. I've already said these first few bullet points already. But the JSON, one primary difference here is the files are generated into multiple versions for JSON, one per version. And in the CSDL and the XML versions,

all the version information is in one file. So that's pretty much the only difference in delivery that you will see from them. The information is identical, just in different formats. So the JSON follows the json-schema.org format. But again, it's really just up to your personal preference which version to use. I end up looking at the CSDL format because that's what we use 99% of the time in the technical work group. So I have a hard time looking at the JSON schema without trying to make the mental translation back to the other format.

Okay, so here's a little bit, and I might have made this a tiny bit bigger, but again, you know, you guys are mostly sitting towards the front of the room. So, okay, so we talked a little bit about, but again, you know, you guys are mostly sitting towards the front of the room. So, okay. So we talked a little bit about, you know, some of the elements in here. Again, there's basically one version for each schema type. And this actually is kind of, we picked, this is doing, what is this doing? This is doing session. It's actually one of the simplest objects.

It's one of the redfish objects because it was pretty short and highlighted everything. So we talked earlier about, you know, they'll include the related objects. So you can see up here, this is what the includes look like. One of the things you'll see, let me get the highlighter since I can't reach that high. So you've got the include formats. Look here. Look up here. Some of the basic includes that everything does is pull in these basic definitions, the OData capabilities,

files from OASIS and the core OData definitions. You see these aliases like this and capabilities. But then you'll also see these are referencing our own ones. So like this redfish.dmtf.schema.v1.redfish.extensions is another Redfish file. Here's the base resource one that everything inherits from. When it's a Swordfish file, it'll be redfish.dmtf.schemas.swordfish.v1 because that's that republication portal where all of our files go. If that was a republication for, say, Broadcoms,

I would expect that to be like slash schemas slash Broadcom slash v1 because we haven't pushed anything out there yet, but when we did, I would expect that to be a similar structure for republication of various companies' files. So that's basically the includes. We talked about namespaces, and so here's the basic format like that. So you can see that entity.version.entity to define the namespace down here. You know, it's a resource.version.resource. It's kind of a strange format, but that's how to specify it.

Okay, and there's some other elements in here. Oops, where did I go? I should look on my own screen here. And then there's a bunch of other fun little things that I didn't really talk through. So there's annotations. So everything is, some of the things that we require are pretty much everything has to have a description and a long description. This is how we, these are, you know, basically required properties in the system.

How we define properties. I've just reused property in a different context there. I apologize. So an actual property of the session is defined this way. So you have a property and then a name and then a bunch of annotations here. So again, see this annotation? This is where you'll see things like a permission that says this property is read-only.

And then in this case, this one's actually also required on create, right? So the username, when you create a session, it must be, you must have a username. And then again, you've got a description and long description. And similarly, password has all those same properties as well. A couple of other kind of redfishy-centric things.

We talked, you had a question earlier about where, you know, was there a lot of active management, basically. And I mentioned this notion of actions in here. Actions is a way to represent functionality that doesn't map well to CRUD. And so CRUD, for anyone who's not familiar with that term, is a restful notion of how to create, read, update, delete, basically. But anything that doesn't map well into that notion would be things like reset, right? When you go to, or mapping and masking, or some functionality like that, where it's like I can't just apply it to an object and have a clearly stateful change to it.

There's not a lot of things in the storage space that actually this applies to, but this is a really interesting way that Redfish came up with to say this thing doesn't fit into the model very well. Let's provide an obvious way for that thing to be interjected into the system. So we have this notion of actions. And so they're explicitly called out this way. What they actually map to is a post that, you know, tells the system to go off and do this thing.

And like I said, we don't have very many of them defined at all in Swordfish, but it's a notion that you can add anything you have in your system that you want to add in explicitly. Or even if it's an operation that you could map to something else, but you want to make obvious, you can also use this mechanism. We wouldn't recommend that, but from an OEM perspective, you could do that. And so again, you see property name up here property name up here and then there's this OEM actions down here which I didn't highlight so there's a space down here

in this complex so these complex types are a way to basically define the behaviors of that property. And then there's an OEM actions here, which is where an OEM extending one of these schema can come in and add the action, add their own actions. So there's kind of space.

I didn't see regular OEM in here. Oh, there's not an example in there. In every schema, there's this place. There's an OEM section and an OEM actions so that vendors can come in and basically add additional functionality into every schema. So you don't basically take and create a whole other copy of it. You basically just add your own functionality in to that existing schema.

Okay, so the same, as I said a couple minutes ago, the schema is identical when it's in JSON. It's converted to this, but everything, so we can kind of look through the Redfish schema here and see on this session that all of the same properties and attributes and annotations and everything are converted over. They're just represented in the JSON schema format here.

So you can see all of that, you know, original OData context, ID and type get generated here, and the references are in place. So ID, oh, so here's the OEM section that I just mentioned in Africa to highlight. OEM and ID are actually inherited from resource. That's why we didn't see them explicitly. One of the things you'll notice about the JSON schema is that everything is included explicitly. So anything that's inherited from another schema

will show up explicitly in the JSON schema. You may not see it in the CSTL schema. So, like, you'll see OEM here. You'll see ID. Those came from resource. Description comes from resource. Name comes from resource. So here's a username that we saw earlier.

And so you see the same thing, right? Read only true. We saw that. Read only true. Some of those other attributes and annotations don't necessarily come over because JSON doesn't have a notion of those. Okay. And then here again is the actions, and then it's got that OEM section underneath it.

And, okay, so that was kind of the end of that. So again, we've seen this slide before. Just a reminder, sneha.org slash swordfish. Is anyone getting sick of me saying that yet? Again, if you're interested in joining the TWIG, talk to, let's see, who all is in the TWIG? Raise your hand. All right. There's about four or five in here.

If you're interested in joining the TWIG, participating, talk to any of us, or there are several folks around that, if you're not a member of SNEA, that would love to talk to you about joining SNEA. Again, swordfishforums.com, ask any questions you want to there, or there's also a mechanism, storagemanagement at SNEA.org, you can send us email. All of this information is on SNnea.org. And if you have specific feedback about the SPAC, you can also send that to snea.org. Okay, questions? Thanks for listening.

If you have questions about the material presented in this podcast, be sure and join our developers mailing list by sending an email to developers-subscribe at sneha.org. Here you can ask questions and discuss this topic further with your peers in the storage developer community. For additional information about the Storage Developer Conference, visit www.storagedeveloper.org.