Grey Beards on Systems - 60: GreyBeards talk cloud data services with Eiki Hrafnsson, Technical Director, NetApp

Episode Date: May 15, 2018

Sponsored by:In this episode, we talk with Eiki Hraffnsson (@Eirikurh), Technical Director, NetApp Cloud Data Services.  Eiki gave a great talk at Cloud Field Day 3 (CFD3), although neither Howard nor I were in attendance. I just met Eiki at a NetApp Spring Analyst event earlier this month and after that Howard and I had … Continue reading "60: GreyBeards talk cloud data services with Eiki Hrafnsson, Technical Director, NetApp"

Hey everybody, Ray Lucchese here and Howard Marks here. Welcome to another episode, sponsored episode of Graybridge on Storage podcast, a show where we get Graybridge storage and system bloggers to talk with storage and system vendors to discuss upcoming products, technologies, and trends affecting the data center today. This Graybirds on Storage podcast is brought to you today by NetApp and was recorded on May 8th, 2018. We're very glad to have with us here today Aki Raffneson, Technical Director of NetApp Cloud Data Services. So Aki, why don't you tell us a little bit about yourself and what's new in cloud data services at NetApp? Hey, thanks Ray. No problem. Well, we've been working day and night now to get a few services out that are really relevant to the cloud-first community and in general to enterprises. So the new cloud stuff, I heard yesterday or today,

there's some new functionality coming out in the cloud data volumes, or cloud volumes, I guess, right? I guess, I mean, for people that are listening, they may not know what cloud volumes are in general. We call it NetApp cloud volumes. Essentially what we've done, we have deployed file-based services supporting multiple protocols like NFS and SMB to the public clouds in collaboration with the public cloud.

So yesterday or today, we announced that cloud volumes is now available on Google Cloud. So Google Cloud Platform, and you're already available on the other two major platforms, isn't that correct? Yeah, last year, around September, I think it was, we announced that we were partnering with Azure to bring Cloud Volumes, which is now, it's branded in Azure, it's branded Azure NetApp Files, but essentially it's the same technology. In a similar way to Google, you can use their APIs directly

to create volumes based on NetApp technology, but it's transparent to the user. He's just using his account, his keys, and all that within the public cloud. And we're also available on AWS. But on AWS, it's purely a NetApp service, and we call it Cloud Volumes for AWS. And so on the Azure and the Google Cloud platform,

you would be using normal Azure APIs to establish a volume and that sort of thing? Yeah, exactly. Oh, that's pretty impressive. Deeply integrated. We worked side by side with the engineers, both at Microsoft and at Google, to make sure that it's a seamless experience for the end

user. So this is a file service and I say I want volumes of x gigabytes megabytes terabytes and they magically appear? Yeah exactly so it's a you have a RESTful API. For the developers out there, now you can actually create NFS or SMB volumes per region in your accounts. And it literally takes about eight seconds to create a volume up to 100 terabytes in size. Well, in most data centers, that takes about a month, so that's good. Yeah, I'm sure. And we're not even talking NetApp. We're talking anybody. That's just process. That's true.

The underlying technology is something that NetApp's been developing for the last 25 years. My background coming from cloud services and creating hybrid cloud platforms kind of lent itself and our team worked on this for a long time to sort of make that into a cloud service so that it really is geared towards you know automation so we had to not only integrate on the technology side but also on the business side this means that if you got accounts on azure or you got accounts on on google you can actually use that same billing account to to ingest these services so anybody that's got an account on azure Google, even AWS can just tap into this pretty much with a couple clicks kind of thing? Or API is even better than that, right? Yeah, exactly.

And the only difference between these platforms in terms of features or the way that it's consumed is that on AWS, you go to the service and you subscribe to it through the marketplace. And then the control of it and the user interface for that is on cloud.net.com. Okay. While on the other two services, it's a native user interface and native experience,

meaning that the user interface is actually just part of the Azure portal and it's part of the GCP portal. That's impressive. And that's kind of the same reason why I say it's a native API. It's because it is really a really deep integration with those cloud providers. And since it's being built through the cloud providers as well, you never actually see a bill from NetApp. It's a service of the public cloud providers.

And that also means that you can use your credits if you've got credits on these services. And it also means that partners can sell this through the normal sales channels of the public cloud providers. They don't even have to be NetApp. If I wanted to provide a database service that utilized this sort of storage technology, I could just go ahead and do that right through the marketplace itself. So I could build on top of this just about anything I want.

Yeah, and I mean, it's super useful for SaaS providers and just a lot of, not just enterprises, but also developers thinking about containerized applications, you know, because there's some features in there that are unique to NetApp, like instant snapshotting that you cannot get in any cloud service today. Oh, no. An EBS snapshot is very different from a NetApp snapshot. Oh, God, yes.

And for enterprise applications, I'd much rather have the net app snapshot so the bill for this is per gigabyte per month yes exactly there there's a few well we've got both um sort of metered on on-demand model uh but there's also the uh availability of uh sort of pre-bought or reserved space. And then the cloud providers can bundle this into anything they want. Okay. Are there performance tiers? Can I set up an all-flash volume? So as this is a service, we had an analyst day yesterday. One of the most common questions was, so what's running behind this?

And my default answer is, you know what, the end user doesn't care because we guarantee service levels. But we're analysts and geeks. We care. You'll get a high performance volume. That is, you know, just to give you a comparison, the only public cloud provider that actually has a file service today at scale is Amazon and their EFS service. If you compare this to that, it's about 20 times faster than EFS and cost 20% less. I think the other problem with those sorts of other services is that there can be quite a lot of, I'll call it noisy neighbors and stuff like that that you have no control over when you're working in that environment. I'm assuming that with the NetApp cloud volume, there's a lot

more control over how much activity is going on and whatever the storage that's actually behind it. Yeah. And then from our side, I mean, that's what part of the service does. We've completely automated that, that those parts so that we don't get into the issue of noisy neighbors. We've figured out, you know, a whole bunch of ways to automate these things

within each cloud provider because each cloud provider is different and so the service levels are pretty comparable across all three so you should be seeing flash performance in the top tiers and it's interesting in the top tiers. And it's interesting that the lower tiers,

right now we're actually kind of limiting the speed to get to those lower tiers, but we might implement even lower tiers in the future, like let's say if you wanted to have it for secondary backup. There are lots of relatively cold data applications i could think of where you know i want this available to a lot of applications and therefore i want it to be file based yeah well i think it's it's it's interesting that you're you're sort of uh you're you're sort of gating the performance to get to the lower tiers because you've got such a high performing storage system that you can't, you can't take it down that level. That's, that's an interesting component.

Yeah. I mean, we build this with the very best stuff from now on. No doubt. No doubt. And you know, we're not just thinking about getting to this first stage of file storage, but this is really a multi-part play we're rolling out. So Cloud Volumes is, you know, an improvement over something like ONTAP Cloud, where you have to manage the control yourself,

but it's still easier to do that and more cost-effective than something like EBS. I was just thinking how much easier something like this is than setting up a software-defined storage system to build a file system on top of EBS and having to run all those instances. That would be the ONTAP Cloud, I guess. So this is one of multiple solutions in this space that you guys offer.

OnTap Cloud being one, I'll call it the cloud volume being another, but you also have NPS, which is this NetApp private storage co-hosted that can access from the cloud, I guess, right? Yeah, NPS is what we call cloud adjacent. It's not really part of the cloud services, but it's one of the technologies that enables companies to be close to the public cloud to use the compute. But it really is not a managed service

in the way that we think about cloud volumes. And I mean, NetApp actually has a few cloud offerings already. Like, for example, you said onTAP cloud or cloud ONTAP, which way you want to say it. And that's been around for two or three years now. And that and OCI and Cloud Sync, which allows you to... OCI? Yeah, what is OCI?

There's an acronym I'm not familiar with. OCI is the metrics and analytics platform that NetApp has created. So it would be like On-Command Cloud Insight or something like that? On-Command Insights, yeah. Okay. And then upcoming, we'll have a cloud version of that called Cloud Insights. Okay. And then upcoming, we'll have a cloud version of that called Cloud Insights.

So Cloud Volumes is really like the first in the line of these multiple things that we want to do in the public cloud. Right. But it's really a building block. So just like Amazon had to build S3 and EC2 first, this is really the first core service that is pure public cloud uh it's highly integrated into the public cloud ecosystem and then what we build on top of this is multiple or down the line are multiple things like application support um we've got a number of

data connectors that are already out there. You can use public cloud analytics like HG Insights on Azure if you want to do Hadoop-based analytics. For developers, we've got a plugin for Kubernetes called Trident. I was talking to Josh last night and he was pretty positive about what Trident. Yeah, tell me about Trident, because I was talking to Josh last night, and he was pretty positive about what Trident provides. So, I mean, today, you know, if you're a Docker container user, if you want to use persistent storage, it takes a little effort to get there and stuff like that. But with Trident, it's a little bit easier. Is that how this all plays out?

Pretty much. So, the way that container persistent storage works is that how this all plays out pretty much so the way that container persistent storage works is that usually most people are using kubernetes now like even docker's um decided to you know support kubernetes as their uh scheduler um the same as mesos or mesosphere doing that as well right bye-bye swarm is getting on organ and the persistent volume management in that it is based on a on a eventually eventually consistent model meaning that meaning that there's a separation of roles there so the storage admin for example still has a role to play there where he can set up the configuration for something like Trident. And it's actually really easy to configure.

To tell it, if it's on-prem or in the public cloud, essentially he's saying, use these credentials and you can make this type of volume. And then he's really done. Then the developer comes in or the application admin, he submits an application template or even just a pod definition, which is almost like a VM in a container sense. That definition says, I have a persistent volume claim, which means I want a volume of this size with this service level, this speed,

and please get that ready for me. And Kubernetes takes that, matches it with a persistent volume, and then automates that. And Trident is the key there. Trident takes that volume claim and figures out where to actually create the volume and mounts it to the application or to the container. And you mentioned it was an eventual consistent model. Even with a cloud volume or an ONTAP storage volume, it would be an eventual consistent?

Eventually consistent? Eventually consistency means that anything that you request in something like a scheduler like Kubernetes is always eventually guaranteed. That means that it doesn't, half doesn't matter in which order you ask for things because they will be eventually consistent when all the claims or all the configurations have been met,

if they can be met. Okay, that makes me feel a lot better. For a second there, I thought you were talking about that the data I write to that volume will eventually be consistent, and that's disturbing. Yeah, that's not what I meant. So it's the same term for different technology. Yeah, you're really talking about eventually provisioned, which I'm cool with. Yes, eventually provisioned might be a better word for it.

I'm not sure I understand what you guys are talking about. So I'm a container developer and I want to use storage and I build this, you know, I'll call it procedure that defines all the environment that I want. And so Kubernetes fires up this container on one of the pods someplace and it's going to go out and eventually provision the storage? Yes.

Yeah, but eventually here is seconds, not millennia. Yeah, yeah. But, I mean, the container can't actually execute without the persistent storage is the problem. So at some later point, the storage will be provisioned and the container will execute concurrent with that. Exactly. I got you. I got you. After it's been mounted and all that. Right. of abrision and the container will execute concurrent with that exactly i gotcha i gotcha

mounted and all that right so there's actually like a really good use case like there are multiple use cases obviously for for kubernetes applications in public clouds uh and a lot of those applications need a lot of volumes and there's a dirty little secret in the public cloud um and also a little bit on-prem in that there is a finite amount of block storage you can mount to a vm and that limit is not there for something like nfs or or smb i'm sorry you're saying there's a finite amount of ebs i can i can have for an ec2 instance yep're saying there's a finite amount of EBS that I can have for an EC2 instance? Yep. Yeah, well, there's a limited size and then there's a limited number. We're talking tens of terabytes.

I'm not talking about the amount of storage. I'm talking about the number of devices you can connect to. Right, right, right. Which is a problem when you're doing multiple microservices that might need volumes and multiples of those separate volumes. Yeah. So there's that case for using network-attached file storage. Obviously, on our side, I mean, you're getting way way faster storage you know even if you're using something like faster is always better usually i mean if it's cost effective

then yes definitely i mean i did some like file tests um compared it to like general purpose ssds and you know on on large EC2 volumes. And we were seeing like 10x difference in using cloud volumes versus a general purpose SSD. With the same EC2 instance and that sort of thing. That's interesting. Same instance. So, I mean, there's definitely a huge uh in terms of services but there's also new things that could be done with this new technology because you haven't had multi-read multi-write in public cloud before so that's when you say multi-read multi-write you're talking multiple ec2 instances reading

and writing to the same file system exactly Exactly. And think about some of the problems that people have been trying to solve here in distributed computing that become a whole lot easier, at least for some of the use cases. Think about something like distributed key value store, like EdCity or something like that, which is a core component in many of

these things if you had a multi-read multi-write highly available volume underneath you know you don't actually you might not actually need to have that um distributed computing model for that so it makes interesting like there are opens up interesting things in development in particular, but there's also just a ton of use cases for business. I mean, for applications, for... We've all mashed together various platforms and applications and just used a file system as, well, I'll dump the output from application one here and application two will pick it up,

which becomes difficult. It's one of those things that one of those big differences between developing code in AWS and developing code in your data center. Absolutely. I mean, some of those changes have been beneficial. Like, I mean, object storage definitely has its benefits, not for high performance computing or, you know or that sort of thing, but archiving and sometimes even as a transport layer to use that as an intermediate. But for high-performance applications or moving applications that are file-based or doing containerized things,

I think we're very well suited for that. Yeah. And plugging into the cloud is just the biggest problem. You've now got cloud volumes in the big three public cloud providers. I'm assuming that when I create a volume, it's local to one cloud provider today, right? I'd say it's local to a region too, right?

It's local. So the failure domain is a region. So any SLA or availability or durability is linked to the region. So it's not shared. The volumes are not shared between cloud providers. We're actually in the data center. I wanted to mention the names.

All three sites. All three cloud vendors. Yeah, the usual suspects. Yeah. Okay. Because I started getting into the blue sky of, well, I could lift and shift my applications to Azure and share data with the new applications guys are writing in AWS.

That would be an NPS thing. I guess I'm a little ahead of myself. Yeah, I think so. But, you know, you can replicate. So if it's an ONTAP volume at both locations, you could potentially asynchronously replicate between one and the other. I'm not sure that you've got that feature in there yet uh achy but uh that's certainly something that could come down the line that's an interesting use case so what we do have today

is cloud sync which is you know it's sort of like you know uh an enterprise version of rsync i mean it does file replication very uh efficiently and you can use it to pump data from on-prem to cloud you can you know it doesn't matter which vendor is behind the target and source but people are using that to get stuff to cloud volumes but you can also use that to do to do auto sync or continuous synchronization between a couple of cloud volumes quite easily. So that use case could, you know, you could do that today between cloud volumes on the different cloud providers. Yeah, yeah, yeah. That'd be interesting. If that's really useful right now, I don't know, but when we start talking about

our application orchestration side, there will definitely be use cases that we will think about making easier. I think we're dreaming and talking about multi-cloud a lot more than any customers are actually doing. I mean, all customers would want to do it, but whether they even consider it at this level would be a different question.

Yeah, I understand. Not yet. The data gravity thing. Yes, exactly. There's that. So if you were thinking about instant failover and all that, you would want to have a feature like Snap Mirror enabled between the services. We don't have that quite yet, but it's on the roadmap to enable that. And most likely, eventually, we'll have something we'll call file sync, which would automatically detect if we can do Snap Mirror instead of files.

Right. Hey, well, this has been great. Howard, any last questions for Aki? No, I think I got it. All right. Aki, any last comments you'd like to make to our listening audience before we sign off? I just would say that we welcome any input into what features they would like to see on Cloud Volumes and um we do have a really good uh site to go to to learn more about our cloud services at cloud.netapp.com right uh it's called cost central and we're we'll be publishing a number of things there that will actually you know help users assess um how well

they're fitted to go to go to cloud to cloud volumes, use our different services. And I encourage them to check that out, cloud.netapp.com, and try and get some feedback to us if they have some burning features that they really want to see in the public cloud, especially from their experience with NetApp or experience with other vendors. Yeah, yeah.

So we will certainly link that in the podcast post that we published associated with this. And we thank you for keeping the URL simple. Yeah, you should see our URLs. You don't want to talk about it. Well, this has been great, Aki. Thank you very much for being on our show today. Oh, thanks, guys. My pleasure.

If you enjoy our podcast, please tell your friends about us and take some time to review us on iTunes as this will help get the word out. That's it for now. Bye Howard. Bye Ray. Until next time.