Storage Developer Conference - #21: OpenStack Swift On File

Episode Date: September 8, 2016

Hello, everybody. Mark Carlson here, SNEA Technical Council Chair. Welcome to the SDC Podcast. Every week, the SDC Podcast presents important technical topics to the developer community. Each episode is hand-selected by the SNEA Technical Council from the presentations at our annual Storage Developer Conference. The link to the slides is available in the show notes at snea.org slash podcast. You are listening to SDC Podcast Episode 21. Today we hear from Dean Hildebrand, Master Inventor and Manager with IBM, as he presents OpenStack Swift on File from the 2015 Storage Developer Conference. Hello, welcome, and thanks for coming to the...

It's going to be somewhat security, but I would say a little of light on the security and a lot more on general access of objects in file, and so how we're mixing those two together and how all that's going to work. So I'm doing this with Sassakant Ida over here as well, so I'm going to give a high-level overview and go through some of the challenges that we're tackling, and he'll give some of the specifics on how we're actually merging user identity between file and object. So at a high level, we're going to talk a little bit about OpenSAC Swift. I don't know just the kind of level sets, people's knowledge about the system, why it's

useful to combine Swift and other object systems with file systems as well, all into one big system. The work we're doing with the community, specifically Red Hat in this case, primarily around integrating file with OpenStack Swift, and as well as then talk about our architecture, IBM Spectrum Scale, formerly known as GPFS, that we are now building as an object solution, and it's out there and people are using it. And then talk again about how we're mixing user identity with file and objects. So, you know, again, just at a high level, what is OpenStack Swift? That's the line from their web page, but effectively it is a cloud-based object store

that you can deploy on your prem, or you can deploy in your cloud, and you can deploy it anywhere. There's many deployments of it out there now. IBM's cloud is based on OpenStack Swift, HP's cloud is based on it, and many other vendors are out there selling them as private clouds. Again, it's good for everything that object store is good for. It's good for web content, images, static content, but even these days it's merging into other areas such as scientific computing and many other areas, including archive, banking, and others.

For a specific level, it's written in Python, like most of OpenStack is written in Python, and there are other efforts to merge them into other languages as well, but still today the primary language is all Python. It's relatively small, I would say, in terms of how software projects go for cloud and for storage. And Swift in general has a very vibrant community. I encourage anybody to come to the OpenStack summits and meet the developers firsthand. So at a high level, what are the features of OpenStack Swift?

Well, they're the features that you would expect and why you would want to use object storage. So you have secure REST-based access. You have active cloud access where you can have multiple sites and distribute objects all throughout them. The high availability, the flat namespace features that not only make your objects easier to store, but it's easier to manage as well. You don't have to worry about exploding directories and things like that at a high level. The architecture here is very flexible. It's open source of course but there's a big middleware infrastructure

and we'll talk about some of the things that we've written to add into Swift and so it's really easy to customize for your needs many customers each write little modules for the things that they need to do for their environment plus has all of the other aspects

that you might expect such as versioning, quotas, many other aspects that you might expect, such as versioning, quotas, right, meaning many other aspects. Yes? Sure. So sort of on the middle box there, it's Swift and S3. There seems to be a lot of, I don't know, deliberate accidental completion there.

Right. So Swift is an infrastructure that supports the Swift protocol. So think about that supports the Swift protocol. So think about that as POSIX. Our file system is an infrastructure that supports the POSIX protocol. Then you can layer things on top of it. One of the layerings that has been done and that we support is the S3 API module that's been done on top of it. There's others out there that are less robust. I think NetApp did a CDMI one,

and there's other little pieces of things that all build on top of Swift as well. Any idea what proportion of people are using S3? So this is a very interesting question, but I would say that a large majority of people are asking for S3. I think a lot of it's come from there's a term that a customer told me once called repatriation. So many people are in Amazon S3 today. They're sick of it cost-wise, management-wise.

They want to build. They're finally able to now deploy private clouds inside of their sites, and they want to repatriate their apps, except all their apps are written for S3. So now that's become a really important API. I think that's changing over time with the Swift API. Again, you know, IBM and HP's and others' clouds are based on Swift. People are writing apps for Swift.

But it's, you know, it's an emerging API versus a more, what do you want to say, not legacy, but established API. So looking at really high level how the whole system works, but the Swift user is putting in objects and retrieving them. It goes into a consistent hash ring where it's distributed across the systems. And there's two things to notice here. One is when it actually ends up. So, you know, as always, right,

the dirty secret of every object, not every, most object stores is that there's a file system. So how do we look when we look at the file system? And we can see here that where this object ended up, it's not really legible. If you were a file user, you wouldn't go and say, hey, I know how to get to that. So it's based upon many different aspects about the ring and how they're distributing files across the entire system. The other thing to notice is that it's owned by Swift.

So there's a single user in the system. All files are stored by that user. And then higher level systems such as the Keystone Identity Authentication Server is used then to do all of the multi-tendency authentication. But down at the lowest level inside of the actual file system, it's all owned by Swift. So if you wanted to just take this file and export it via NFS or something, you'd have multiple challenges for doing so. One would be the user. Two, you're not really hooked into the authentication service.

And B, your file looks like a bunch of random characters. So not directly easy to do today is the point. All right, so we have this great, in OpenStack, they have a great object system. Turns out that actually some people still want to use file. So why do you want to use file to your object system? One is there's very few apps written for object systems today. Like I said, many are written for S3 and they're in the cloud, but any new apps or any legacy apps are all written for File API.

So if you are working inside on-premise locations and you already have file apps, your system administrator said for various reasons they want to go off and use object storage, you're having a hard time transitioning. You have to either rewrite your apps, you have to use some sort of middleware, there's many other things you have to do to get that up and running. So using file as a transition towards the object API is a very common use case.

Another one is, you know, many of the apps out today are generating data via file, but then you want to share them in your cloud via object. So file in, object out in many cases. And it actually works in a more of a circular manner as well if you want to analyze that data. So much data can be ingested using object because of its strong authentication schemes and worldwide access. You then want to run, let's say, an analysis on that data, generate results, and then reshare those results back over file again. And so a couple things to note here are, in theory, you can run all of your analysis with Spark and Hadoop and everything else that you like on your objects directly. But there's a reason that you have object stores and you have HDFS.

They are not the same, they don't work the same, and they don't perform the same. So if you want to run and get good performance, you still want to continue using your file APIs and through HDFS-based systems in order to do that. And so it's still an advantage to switch to using the file API when it's necessary and then switch back to the object semantics. One small thing there as well is that the object APIs don't support

append. So some of the systems, Hive, HBase, others, require append as part of their system. You just can't even run them on the system. It's just not possible. And then lastly, most data centers are now going to be filled. They have block, you have file, you're now going to have objects.

The block's not going anywhere, the file's not going anywhere, and now you've got a third. All the better if you can do it in one management pane. If you have to put up a whole other management system on the side, you know, you're just creating more headaches for yourself, right? So the goal of integrating them is to try to relieve those as much as possible. All right, so how do you do this today? There's many vendors out there that are selling gateways, all with varying success. The key sort of sharing thing that they all have in common is that they're all sort of in front of the system, so adding another tier to the entire

system. And in addition, same as in the analytics aspect that I just said, they're still having to work through the object API in the backend, meaning objects are immutable. Everything you want to do, if you want to update a byte, you still have to reread and rechange the entire file.

And so if you ask object vendors, how are these gateways working for you, they will clearly say it's great, it starts off well, but then eventually the user thinks he has a POSIX file-based system and he starts using it like that,

and then they run into trouble. So it was great as a transition, but it's not great as a long-term way of supporting file into the system. And so that's where this work that we're talking today is about how do we get to somewhere where we can use file like it's supposed to be file and object like it's supposed to be object.

Please feel free to interject any questions throughout, by the way, just as a side note. All right, so the work that we as a side note. All right. So the work that we are doing is called Swift on File. Again, this is an open source project. You can go to the URL and check out the work. It's implemented as what's called an object server implementation inside of Swift. It's otherwise more colloquially known as a disk file.

So this is the part of Swift that does the otherwise more colloquially known as a disk file. So this is the part of Swift that does the interaction with the lower layers. So we've changed this to customize it to make objects to be more like files, effectively. Just as a quick note that this module then is very flexible. So this same module is used to support kinetic, for example, drives and many other aspects because it completely substitutes how Swift writes data down. Instead of using POSIX, it can use any API that you would like it to use. And so we're taking it and then making it make file, again, objects look like files, which again, they're already stored in a file system, but we have to solve those two problems that I mentioned earlier. One is to get the user correct,

and one is to store them in a way that someone would actually recognize their object as a file. Yeah, sorry, in the back. So if you say, how would the data coding work with Surface? Would you have data coding on top of the file? Yeah, I'll go through what we did, and then everyone can do their own, any vendor group that wants to do it the way they did.

So we do it all inside of our system. And so Swift just stores data, and then we take it and erase your code and do all those good things at the lower layer. But the API that you created is for Swift? So this is just a pluggable module, and they have the default version that does what I said earlier, where it creates the Swift Swift user,

and it stores the file as a long path name for specific reasons according to the ring. And we're just taking that and then changing two main things I'm going to talk about here is we change how it's no longer using the Swift user anymore, and we store it as a regular path. But we're not changing any of the APIs or how things are written or things like that. If they... Sorry, go ahead.

Do you use the library aspect of the Swift to write your integration code? No. We, again, at IBM using Spectrum Scale do not. Scale does everything. Once it passes the data off to us, then we handle all sort of what you might want to refer to

as enterprise storage functions based on that data. All right, so the disk files now can be enabled as what Swift has as storage policies, which allows you to configure on each container a different type of policy. So you could store it in different disks, different tiers. You could choose at the Swift erasure coding, for example. Or now you can choose Swift on file, which is a way of now storing your files and objects together. The one requirement here, which I'm kind of sneakily bringing in,

is that you require a cluster file system, scale-out file system, in order to make this work. You can't have scale-out file access if you don't have a scale-out file system. I don't really care how you might try, but that's the requirements. And so today we have it working with StepStreamScale and Red Hat has it with

Gluster. But this way the file system itself is presenting the native file access for any users that want to use the file side. And then you can use object side through the Swift implementation as well. And then you can add in, we'll go through the architecture, but you can add in as many servers as you would like. And that's part of the scale-out cluster file system aspects to the system. This would all still work with a single server, you just would be limited to this single server. So if you didn't have a cluster file system, you just couldn't scale out your front-end set of servers. But the key here is that the objects are now going to be stored in the same path as the object's URL. So let me go through that in a second.

Are you going to cover... So in a file system, every level in the path has permissions. In the object in space, the whole name is what's got the permissions. Yeah, I'll go through... Do you have any sort of mapping to what happens to those intermediate steps? Yeah, I'll go through a little bit on that, and you can ask questions if we're sort of sparse on your questions.

So just to go through again on how it works, the Swift user is still putting in their object instead of seeing the original ring. It's now using the Swift on file policy to go into the Swift on file ring, and then it's stored inside your scaled file system, which again, then in the typical case, and in our case, is taking over then all of the functions, the eraser coding, the data protection, encryption, whatever other aspects

you want to see is all being done at this layer. One thing to note is that the Swift user is still Swift in this case, so by default today it still uses that, and I'll talk about what we're doing soon. And then the other thing, though, and the key here is by default, it's now being stored in a path name that is much more recognizable to regular humans. The general path is to look at it from an account level, and then every account has multiple containers, so then the container name is underneath that, and then the object name goes underneath that.

So it's a three-level hierarchy of account, container, and object. And in the Swift S3 case, it's just account, bucket, object. So it just goes in those three levels, and those are the three levels that you have. So you have that one, the account, the container, and then the object itself. And so now when you export that via NFS or however you would like to access it, when you still need to know what account you're dealing with, what a container you're dealing with, but the data is then immediately accessible and recognizable as a real name. You mean that you eliminated metadata which is

usually stored in some SQL or some database? Nope. All that's there as well. Yeah. Is the consequence of that that when you store an object through Swift, it can be accessed by file system users, but if a file system user went and created a file in those

directories, the metadata would be missing from Swift and therefore would not be accessible via Swift? So it would be. And then what Swift on file does by default today is... So let me just back up for a second. With Swift, one advantage is that there's no lookups to determine where the file exists. It either exists or it doesn't exist.

So with Swift on file today, it just goes and accesses the file. If it's there it's there and it retrieves it for the user. Dynamically to your point, all of the metadata that needs to happen then is dynamically added onto the object and stored into the database. So that's all that happens.

Right. And in the work we're doing right now is where you can control we call it objectizing files. So if you want to work with your file system for is where you can control, we call it objectizing files. So if you want to work with your file system for a while, you can do that. And then you could then manually make it visible via the object API. Or again, you can just do it dynamically where people come in and as they ask for files, they're just automatically added into the system and then exported to the users.

All right. So just to go over this point again in nauseating detail, the object, again, that has a count container and object, instead of being stored like this, is now stored like this. So a consequence here, I mean, just again, for very things, there's tradeoffs in everything you do. One of the reasons that it was nice to have this crazy path name was that if you had a billion objects, it stored them across a million directories, let's just say, and reduced the number of files per directory. So a consequence of using a system like this is you need to pay attention to your files again.

So, you know, as always, you wouldn't want to put a billion files in the same directory. Now you're using file and object. You want to be a little more cognizant of how big your containers are getting so that when people come in from the file side, they're not all of a sudden doing an LS of a billion, a directory with a billion files in them. So you're back to thinking file again, but that makes sense because you're doing file and object.

You're not just doing object. So one quick note is that while we want to work towards, in one way or another, making this our default policy of always using the Swift on file and being able to store your objects in the system in a way that's accessible via the file interface. Currently, there are some limitations, and so it can dynamically support both of the

rings, both policies simultaneously. So for containers where you want to think about file and you just want general scale out access, you can choose that. And for other containers where you want file access, you can dictate and use that. And they're both running in the same system. All right. So use cases, again, just to go over some basic ones that we think are really useful. So for example, camera feeds from many different video capture from many different fields that are coming online now. Object is a real good use case for those ones due to the global access of an easy HTTP puts of all the data

that can be streamed into the object system. Once you have it in, though, you may want to do something more post-processing, searching other aspects and other post-production on those video feeds that are coming in. And typically today, anyways, all those are using file interfaces. So you can then ingest it via the object and then access it again via the file.

Another use case that we think is pretty valid and we're seeing emerge and actually I think this one is actually more that people are doing today is on the scalable side especially in the scientific computing but in other areas people are still generating all this data via file it's really the only protocol that still has the throughput and the performance

that can meet the performance requirements of many of these areas. But once that data has been ingested, again, it may want to be further analyzed or other post-processing done on that data. And then at the end of it, again, maybe you want to be able to share it. I mean, one problem, you know, it seems like since the beginning of time with NFS and Samba

and other distributed file system protocols has been security and authentication and global access, especially replication across multiple sites and being able to build that cloud up of the data for your file data. And now that's what Object is giving to this. So when you objectize your data, for example, you're giving that access to the data. All right, so just to go very quickly over what we're doing at IBM. Spectrum Scale is a

scale-out file system. We already support many different protocols, POSIX, HDFS, NFS, Samba, on top of this all in a single system. Now with the object support, we're supporting the OpenStack Swift

on top of Spectrum Scale, building it all into a same system. So with the goal of saying, well, we already have all of these enterprise features as part of the system, and now we can also support object as one of the protocols into the system. And I don't want to go too much over

all the different features, but the key, I think, is that more use. So we use the standard code base. We make practically no changes other than a few module changes that are supported within the infrastructure. But at the same time, we're using most of the features of what we have already today, because those are the ones that our customers already

know and love, and they're already written, I guess you could say. Looking a little more at the architecture, there is a growing sort of, I guess, different way of configuring Swift called the Paco architecture.

What just that means is you've got the proxy account container and object servers all running on every node. So when we take our file system on all the different servers, we just run all of the OpenStack Swift processes on every server as well. And they have an upper layer that is handling all of the object work and then, you know,

spectrum scale is then used as basically the storage layer. For those who are curious, this can then be, it's really an independent, so you can use commodity disks and drives, you have specialized storage devices, you know, many other choices that can be used

at this layer, then tiered, have your objects tiered between all those layers as needed based upon policies. Any questions then before I'm going to pass it off to my co-speaker, Sasi, on some of the details of the user identity integration? So you mentioned tape a couple times through there. Yes. Any thoughts on connecting this to LTFS? Yeah, so... Sorry, maybe you have to describe what LTFS is.

Sure, sorry. So LTFS is the linear tape file system. IBM and others have worked towards this open source standard of creating, using tapes as a file system. Yeah, it's not just open source. It's actually a real thing.

It's a standard. It's not just going to be an ISO standard. Right. So then many different vendors sell LTFS solutions. IBM sells one called LTFS, but then they also have Enterprise Edition, which is actually spectrum scale merged with tape. In that architecture, spectrum scale becomes the disk cache in front of your tape archive. And so that is a very common scenario where people want a few hundred terabytes, let's

say, of a disk cache fronting 50, 80, 100 petabytes of disk in the back end. And so using the object service on top of that, it works exactly as file would in terms of how you use policies to tier data up and down between the disk cache and the tape. And in Spectrum Scale, we use DMAPI in order to do that, to automatically tier data down or up again. The challenge, if there was one, which I think there always is,

is that objects are typically smaller than files. And so in the current use cases, I think that once object starts becoming more of an archive, it'll start storing archive data, which I think is going to start being a lot bigger. But in many of the more web, online- start storing archive data, which I think is going to start being a lot bigger. But in many of the more web online type image use cases, the files are relatively small. And so I think creating appropriate tiering policies to your tape subsystem, I think, is going to be very important.

And that's something we're working on as well, is to make sure that when you're tiering containers and things, they might be co-located on a set of tapes. But for now, no direct connection between Swift and LTFS. You're going just to your other classes. Right, yeah. There's some discussion with some other vendors around there around doing it directly on tape, but I think we kind of all, in my opinion,

we, we meaning some people, agreed, we meaning we agreed that, like, I just don't think that it's viable. Given the object size that we've seen today and the throughput rates that they want to see, using tapes is directly, is just going to cause huge delays in the system. And you need some sort of buffering mechanism in order to make that viable. Another thing that we're big on is prefetching. So if you know you're

going to use a container, you can prefetch those containers onto disks and then use them based on acid disk policies. One thing about LTFS that may actually help here is LTFS standard recently had added extensions that allow you to store Swift metadata, for example, in a standardized and interoperable way. So that'll help to providing that round. So the one upside of our support for this, and this is just coming from DMAPI as a standard, is that all of the XADDRs,

so all of the Swift metadata stored as extended attributes inside of the files, none of that ever hits a tape. So all of that stays in the disk layer. And that's important for a couple different reasons. One is that people might want to do a lot of analysis on their metadata and start using that and doing searching that. So that doesn't cause tape recalls. But the other thing is that Swift inherently does a lot of stuff in the background

where it's looking at your metadata, and you wouldn't want those things to generate tape recalls either. And so all of the extended attributes continue to stay on disk, and that actually makes tape much more usable. Whereas, again, back to using LTFS directly, I think maybe with some of these enhancements it's okay, but it would still be kind of a slow system. You'd have to really watch out for unnecessary recalls.

Sasi, I think you're up. Thanks, Dean. Coming to the next part of the talk, I would like to start with how OpenStack Swift semantics are, ACL semantics especially. In the Swift, kind of ACLs would kind of set the owner would be able to set those to do read-write access to the other users of the keystone. And they are actually set at the container layer and they are stored on the SQLite

database and are not actually translated to user ID or group ID or as we see in the file world. And they are primarily controllable via HTTP headers. So this is just a sample illustration saying that how does a read ACL look like in the Swift world where we can see the tester2 is a kind of user who is assigned to with a read ACL whereas tester2 will have all access to the objects underlying to the container 1. Similarly, we can see the write ACL where tester2 has been given write ACL where he has entire write permissions to the objects which are lying inside the container. On the other world, coming to how file ACL semantics look like, basically in the file

world we kind of predominantly see three kinds of ACLs which are POSIX, NFS v4, and SAMBA ACLs, where POSIX ACLs kind of help users to set three kind of permissions, which are read, write, and execute, whereas NFSv4 and SMB ACLs kind of provide more fine-grained control in terms of permissions than when compared with POSX. So if we have to summarize how file ACLs are different with object ACLs, we can just point out them as three points, which is the file ACLs are kind of more granular and comprehensive. It supports inheritance, like the file inherits its parent's directory ACLs

at the time of creation, as well as ACLs can be set at both directories as well as at the file level. With this thing, with these two worlds in mind, if we have to support user ID mapping, what's the primary use of supporting user ID mapping in the current day Swift on files use case? It's like traditionally multiple file related features such as backup quotas and ILM features

like information lifecycle management where a particular user files has to be moved to a lower tier or has to be brought to a higher tier. These are all primarily dependent on the user ID whereas that kind of misses in the current day OpenStack Swift world where the file is kind of owned or the object is kind of owned with Swift user. And really we can't apply these user ID related functions to those. So there is a whole journey that has to be started because these are kind of two ends of the spectrum,

which can't happen that we have to support 100% compatibility, which even today NFS and Samba are not able to provide in that sense. So for now, we are kind of limiting ourselves to user ID mapping, whereas in the future work we would be concentrating on how ACLs can be ported, or how can we support compatibility between these two worlds. So as I said, it's the first step.

So in order to achieve that, we kind of figured out that it can be achieved in two kind of modes. One is using non-unified identity, which is like not really user ID mapping, but kind of enables access to the file world to access objects as well as to the object world in order to access the file, to the files. And the second mode is unified identity

where we kind of maintain user IDs mapped in file as well as with objects. To the mode one, we can actually see that the same clustered file system is actually used for object access as well as file access. To do object access, we have a different set of object users, which are kind of backed with Keystone and a different ADL Dapp can be. Whereas in the file access, there can be an application sitting over there or direct file users can play with the objects and it kind of is using a different ADL Dapp over there. In this kind of scenario, we can see primarily two sets of data are kind of

generated. One set of data will be generated by the object access and the other set of data can be generated using file access. So if we see, if we have to say file access or the application sitting at the file mode to access the objects, there are kind of three level of permissions we have to specify for that. One could be just make the application user as root or define a newly special group or a user which has the access to Swift-owned files. Similarly, give explicit ACLs to the files owned by objects so that it can be accessed even by a file interface. And similarly, if on the other side of the spectrum where if we see how can a user, an object user access the files which are created by the file interface.

In that case, we kind of elevate the Swift user permissions using DAC override capabilities, which kind of set the capabilities to a minimum lower level. This is just a sample example of how exactly we're trying to see. Here John is a person who is kind of the end user and he kind of approached it to Keystone in order to authenticate to the OpenStack Swift. And this is the kind of object put that happens with the user and group of Swift. Whereas if the file kind of has been written by the POS-X or with the user and group of Swift. Whereas if the file kind of has been written

by the POSIX or with the file interface, those can be of any user, but for now we can look into as root user. In order to make Keystone user read the file which is created by POSIX interface, we kind of elevate the Swift user permissions. So with this mode we are kind of able to see it's not ID mapping, but we're kind of able to

succeed on the access permissions. Doesn't that mean that anybody who's got permissions can put content into that container, the object container, then any user who has access into that container, the object container, then any user who has access to that container, even if it's a different user, can get to that object. Right, right, yeah. But the primary problem that we are looking into

is how can a file user look into that thing, rather than looking at it from the Swift perspective. Okay. Coming to the mode two, it's kind of unified identity between file and object. Here we have one-on-one mapping with the user, object user who is making the request, and with the file user.

Here we are kind of using common AD and LDAP for both these two worlds. And there are kind of few design points that we are kind of taking into consideration for now. That is like if the object already exists and the existing ownership of the object will be retained. Say if there are two object users, say John and Bob, kind of, and John owns an object over here and he kind of gave read permissions

or write permissions to Bob in this scenario, and if Bob makes another upload or a post operation for the same file, we will not adhere to the Bob, we don't map Bob's user ID onto the object, instead we will retain the actual owner who is John for in this case. Yeah, and similarly on the other side,

if an object update is performed using the file, we kind of retain the file ACL itself. And to the nested directories question, for now, we are kind of mapping the user to the object, but we don't look into the directories that comes in between. So the nested directories will be still owned by a different user. So that's kind of fuzzy over here of what to do but that's kind of a design I'm open to have a call on that. So this is just a sample illustration to understand that if John is the keystone user in this mode, the object will be owned with jog itself rather than having Swift or

a single user in that case. Coming to how we implemented that, we kind of chosen the middleware approach to do that thing. Middleware is kind of a Swift's component which kind of helps us to play in terms of how to inject code when it is a request path or the response path, et cetera. And this can be even implemented in the disk file layer itself, where it kind of abstracts the entire feature. But for now, we've kind of chosen the middleware because it has on and off features,

and it can be turned on when it is required, etc. So coming to accessing objects via file without ownership implementation, which is mode one, we kind of return a proxy server middleware, which kind of collects the username from the request, from the end user's request. If it is not able to identify the username and if only auth token has been issued to perform the request, we kind of query back to Keystone database and get who is the actual user who has made this request.

And we will make sure that it is passed to the object server's pipeline so that coming to the second point, the object server will get whatever user has been making this request. And we do an ACL inheritance or append operation with the obtained username. So this kind of is a high level flow where Bob is a user who is kind of putting the object. And it passes through the object server one ring which is the Swift's on file part. And we write that first initially with the Swift user,

gets its ACL template and append it with the username we kind of obtained over there and we apply that to the ACL template onto the object. So after that, we could see that even though it's kind of owned by Swift, still Bob has access to it because of the ACL inheritance that we kind of done over there. Coming to the mode 2, where we have a unified identity between both worlds. First step is to elevate the Swift user permission so that it can read the objects which are owned by different users in this case and the same proxy server which will collect the username and pass it as a request and the object server middleware which kind of

does chown operation of the user which we got. So this shows the same picture but we obtain the current user and write the file and do a chown with Bob user in this case. As Swift user has been already elevated, the whole communication within this OpenStack Swift happens via Swift user so it will be still able to read and write files which are owned by a different user, in this case, Bob. Yeah. Here is a whole picture of our roadmap related to a spectrum scale,

which kind of supports the entire spectrum of analytics, entire OpenStack world, its kind of bridges gap between both. Yeah. Thank you. So is all the metadata that Swift needs to serve the object actually

down in the file system? So the file system was stored up somewhere else with a separate instance of Swift. You can basically reconstitute the object. Apart from the extra directory. Right, yeah. Current object

information is typically stored in the XADDRs of the files. But which objects are in which containers is actually stored in the database. So those two features are still... Yeah, so the hierarchy...

Yeah, so the hierarchy is basically inside the database, and the files store... Files at the front. Files at the front, yeah, the object name. But now, as you said, in this file, that's actually no longer true. We have the hierarchy inside the file system, and we have all of the hierarchy inside the file system and we have all of the information inside the file system. So there's still a few things that are remaining to be stored in that database in this case

but it's actually becoming much less useful. The administrative procedure to rebuild the container database based on the file system. Right, which I don't even know if that's possible today. If you lose the container database, can you rebuild it?

Nope. But you can't. Yeah, data still resides. This is something, you know, in the file system world, they say protect your metadata like it's gold. Now in the Swift world, they say protect your database like it's gold.

And it's basically become the same thing. So a question about namespaces. What do you do when you have objects that have reserved characters in their name that the file system won't accept? Yeah, so there's limitations here. One of the other ones is

that the URL itself is not as long, the supported URL is not as long as a supported POSIX file name. So file names have to be shorter in this case. So if you're asking for it through the object side, then it becomes a problem. Especially because Swift supports nested directories, virtual directories they're called, but they just add into object names.

So there's object names with slashes in them, which makes the name can be really long, and if you create a hierarchy inside the file, that would weigh beyond what a typical URL could support. So how would you store on this an object in A slash B slash C? As you said. Would you create directories? all its important. So how would you store on this an object in A slash B slash C? As you said. Would you create directories? Yes. Right here? Okay.

That's the same approach that the middleware, the CD-Line middleware layer for Swift did. Correct me if I'm wrong, but I thought that with Swift accounts that was actually a namespace for users and groups. So when you're translating from the file path, this file path under account one can only have some set of UIDs and GIDs. This file path under account two can have a completely different set of UIDs and GIDs. This file path under account two can have a completely different set of UIDs and GIDs that might

be identical but not be the same UID but not be the same user. Do I have it correct and how do you guys handle that case? In terms of accounts

in the OpenStack world, accounts are also databases, but are actually translated. If we look into Swift on file perspective, the account database, we kind of create the first level of hierarchy. So the first parent directory would be auth underscore something which maps with the account slash container name slash the object path in that sense. But truly we have not even looked into supporting different UIDs and GIDs per account level. So it's now, it's uniform account.

So we can't. But I think the further answer, I mean, one of the keys here was that common share LDAP or ADD namespace between the two. And that's really the binding. That keystone in the back end is supporting that as their set of users. The file on the other side through Kerberos or whatever mechanism you're using

is supporting that exact same set of users and then the access is the same. So it sounds like ignore that requirement and... mean, it doesn't map well, obviously. Without the LDAP, you mean? Yeah, I mean, if you're going to have one backend, then you're really... I guess you're not ignoring the requirement. You're saying every single account has a different user namespace, but it all actually points to the same LDAP server. So it's configured as a separate one, but they're

all configured as the same. And then you can sub, then you provide, as you showed, apples and whatnot down at the container level. So you have the same set of users, but then you can control container access through these apples. Interestingly, the thing is back to S3 was mentioned before. S3 is a much more rich object API and

it does have object apples as well. This isn't something that we support

yet in the Swift version of S3. This is something where we want to get to

is now expanding that to provide a much more richer interface.

It will map back to. Yeah. And so it'll map back to the ACL mapping problem. There's always an ACL mapping problem in the world. But nonetheless, the S3 one will be a lot easier than the S10 to MSFS4 and based off of ACLs. But in the Swift world, where effectively if somebody says, well, I want to have a common set of ACLs across

files and objects and everything, and you want to use Swift's ACLs, you're going to get a very basic set to get a good little close comment. There's just no real way around it. So this is where we haven't really tackled that job yet in terms of doing the

ACLU. So today they're still separate in the Swift on file. You're kind of doing the ACLU to basically blast past any restrictions that got put on files if they were landed through files system API, right? Right. Because they could have been protected several different ways, but you're not going to have to put the object into place.

But they will be respected. So, for example, if you use Samba and create ACL on specific objects, even though your access over here says you can access it, you create the ACL. Okay, so you'll enforce the ACL on the file system level? Because at the end of the day, it's still...

So the ACL is the script user, so the ACL is the limitation. So you were pointed with the Swift user. Now in the last thing you just talked about, you're no longer the Swift user. You're the Swift user, but you're pretending to be that user. So that the ACLs are respected because, again, all the ACLs are supported at the lowest layer, which is your file system. You're changing the UI? You're changing the UI? You mapped that into the kernel? I think it's just checking out the ACLs that are set. So the GPFS spectrum scale is setting the ACLs on that file as well.

So they're all within extended attributes. I'd like to follow up with the, once you traverse into the kernel space, you're dealing with the kernel's idea of who the user is. So it depends on the identity of the script set, right? Yeah, none of this is in the kernel. So it's all on the user space. All right. That's one.

And with CD-Line, because it uses NFS before Apple, it's not in the studio. Yeah. I don't know the time, because I don't have a question in this room. Yeah, we have that. Okay, thank you very much. Thanks for listening. If you have questions about the material presented in this podcast,

be sure and join our developers mailing list by sending an email to developers-subscribe at sneha.org. Here you can ask questions and discuss this topic further with your peers in the developer community. For additional information about the Storage Developer Conference, visit storagedeveloper.org.