Grey Beards on Systems - GreyBeards talk backup with Rick Vanover, Product Strategy Specialist for Veeam

Episode Date: June 11, 2014

Welcome to our 9th monthly episode where we discuss data backup with Rick Vanover, Product Strategy Specialist for Veeam Software. The GreyBeards just talked with Rick and Veeam at last month’s Storage Field Day 5 (SFD5) in Silicon Valley and once again we would suggest everyone who wants to know more about Veeam backup and … Continue reading "GreyBeards talk backup with Rick Vanover, Product Strategy Specialist for Veeam"

Hey everybody, Ray Lucchese here and Howard Marks here. Welcome to the next episode of Graybeards on Storage, a monthly podcast to show where we get Graybeards storage and system bloggers to talk with storage and system vendors to discuss upcoming products, technologies, and trends affecting the data center today. Welcome to the ninth episode of Graybeards on Storage, which was recorded on June 6, 2014. We have with us here today Rick Vanover, Product Strategy Specialist, Veeam. Why don't you tell us a little bit about yourself and your company, Rick? Hey, Ray. Hey, Howard. Thanks for having me on, first of all. Yeah, product strategy specialist is kind of an interesting title, but along with the short, what do I do? I'm an evangelist. Basically, I find people, talk about the brand, and do stuff. You know, you might see me at a major event event. I do a lot with social media.

I do a lot with community stuff. I can write. I'm not really that good at it, but I can do my own podcast, the Veeam Community Podcast, voted number two at the recent round of vsphereland.com votes. I'm on

Twitter at Rick Vanover. I can endorse the Veeam Community Podcast podcast i have been a guest there on uh yes that's right actually and ray's been a guest he doesn't even know it because the one time we did a podcast with uh calvin at storage networking world in dallas and i said hey calvin give me the mp3 and then I redistributed it on the Veeam podcast. Yeah, yeah, yeah, yeah. I understand. Well, that's good. That's good.

So what's new at Veeam these days? Well, new at Veeam, well, it's kind of interesting. It's always something new, first of all. So today, at the moment right now, we are promoting our Veeam Availability Suite version 8, which is including the new version of backup and replication. And in the storage realm, there's actually some very interesting stuff coming around the pike, especially for people who use NetApp storage or maybe people who use data domain storage. We have some great features there. You know, we're just trying to do everything we can to support the needs of that modern data center today. I don't have to evangelize people to be virtual. That's easy. But basically, they want things faster. That's

what I'm helping them with. Why is backup so darn hard to do well? Well, I personally – It's only a podcast, Ray. Yeah, I know, I know, I know. I thought we'd get a lead in here and start driving down below that as we get into it. But, you know, I find customers have, A, very seldom been satisfied with backup in general because of the complexity involved, the time it takes, and some of the failures that have occurred over the past. Now, I have to say, I have not talked with any Veeam customers, so I'm sure your product everybody is very satisfied with and love it.

And I know that you have quite a following in the VM space and are moving into Hyper-V. But what is the big problem with backup? Well, it starts with problems. Everybody has a problem. In fact, the best blog post I can never write, Ray and Howard, is the top five reasons why I hate tape. Since we started supporting tape last year, I can't write it now. All right?

But those top five problems almost always come down to it wasn't the right technology. Well, it came down to things like people smoking in the computer room, people stealing, actually selling the parts, those types of things, instead of keeping it around for restores. It's a great blog post. Catch me in a bar. I'll tell you the story. But everyone has a problem with data protection and backup that they don't really know about until they go to restore. And I think that that fundamental – everyone's got a story. I have stories even with disk-based backups or lack of backups even where I've had a problem.

Or backups of parts of things. Yeah, yeah, incomplete. My favorite is, you know, of course we backed up the Exchange server. Did you back up the domain controllers? No. Oh, but you actually want to restore email. That's totally different, you know? Yeah.

So everyone's got their story. And what we try to do is approach it differently. And whenever I – it's a tough space, right? People deal with a lot of things, everything from the storage side of data protection to keeping users happy. But what I break it down to a lot of times is I say, look, our backup engine, honestly, what we have coming with NetApp and what we've done with HP, it's got some front-side smarts over there. But our backup engine is honestly pretty boring.

We put all the legs on the restore. And what we have coming with version eight well the current s the current count that i'm gonna stick with is 39 restore scenarios from one agentless backup so when you think of kind of don't put all the effort on the backup put all the effort on the restore and then i can get you out of trouble 39 different ways i i don't know. I think that speaks to the needs of what people want. You know, when you go down the restore path, I want to make sure that, you know, those risks are mitigated and people know what they're getting into. And it's got to be easy to use.

So that's just the way the product thinks. And it kind of starts to address those problems that we've all had over the years. Does Veeam offer some sort of restore test capabilities? Yeah. So in terms of restore test, four years ago, this is one of those 39 ways. But we have this technique called a virtual lab that will emulate the recovery of the virtual machine. And what I mean by emulate is not just, okay, I verified it. It'll actually power it on and ensure that in the VMware side, you get a heartbeat like VMware tools and IP stack loads correctly.

And then also it'll check to say, hey, Exchange is running because I can see it on the TCP port, stuff like that. And you can bundle a multi-tiered app. To Howard's point, yeah, Exchange is running because I can see it on the TCP port, stuff like that. And you can bundle a multi-tiered app. To Howard's point, yeah, Exchange is useless until you have the domain controller to let it run, for example. So, yeah, we do have an emulated restore type that people put on their most critical VMs. I mean, we say do it on everything. But honestly, that's the most complicated part of the product. But in the end, it's only a seven-click wizard to do that. Yeah, but that's the biggest advance in data protection in the past 20 years,

to be able to do both instant recovery and temporary recovery. I just am convinced. The biggest advance in 20 years in data protection? What about deduplication? What about storage libraries and stuff? I said 20 years, so we already had tape libraries. You could say backup to disk was a bigger change

and backup to disk is necessary because you can't run a VM from tape. Right. But those were really not conceptual changes. The biggest conceptual change is this instant recovery.

It's that I can have a VM fail or have a host fail and have the data be unrecoverably corrupted and bring that VM up without waiting however many hours it takes to copy the data from the backup repository back to primary storage. And that's one part that I kind of glanced over, and that's the sure backup and instant VM recovery. Basically, we take this disk-based backup, and then it's a window into the vms from that backup restore point so that when we need to use them instant vm recovery virtual lab and some other things we don't have to transfer that data back we actually run it there so it's a absolutely reversed perspective a lot of times recovery is move the data back and therefore it's available. No, no, no, no.

Let's make it available, get the users off my back, aka boot it up in two minutes. Then because it's a virtual machine, I can do all kinds of interesting things like migrate it back. So if it's my ERP system, sure, it's going to run slow like pig from the backup repository, which is four terabyte, 7,200 RPM drives when it used to be running on a hybrid array that was 90% flash hit ratios. But I can get the five people in accounting up and running in 20 minutes, start a storage vMotion or a storage live migration. And then in the morning it'll be back on primary storage and we'll be fully up and running but limping now is way better than waiting hours for it to be back later yeah what would you rather do be uh walking down the street in

crutches or be laying on the ground waiting for someone to pick you up. Neither, actually. I understand. So it's the instant recovery that's one of the best things that happened the last 20 years of data protection? That's four years old. That stuff is four years old. I know, I know. Well, it's four years old from you guys. Yeah, yeah.

But it's just starting to hit the market from everybody else. Yeah. Well, you've got to be careful there. And I'm not going to name any names. But this notion of an instant restore technique, I want to highlight this because there's a very important minor technical detail that can make a huge difference to how people say, oh, yeah, I can do that too. Now, one of the – especially in virtual machines, let's be very specific, in VMware in particular, there's this notion of something called a reversed CBT. So VMware looks at blocks of virtual disks.

Change block tracking. Change block tracking, exactly. That's a great kind of forward-looking technique when you do incremental backups. Hey, what's changed since the last time I was here? Okay, I got a 3% change rate. Okay, scoop it up, move on. And it takes me no time to pick that up. More importantly, it takes you no time to figure out which blocks are the 3%. Yeah, exactly. Back when I taught the backup school seminars

for TechTarget, the one problem people kept presenting me with that I didn't have an answer for was we have this NAS and it has 7 million files on it. And when we run an incremental backup,

it takes two days to walk the file system to identify which files have changed. It's not the backing up that took time. It's the identifying. And by building change block tracking into the architecture, that all gets eliminated. Now, the caution here, though, is there's a lot of instant restore techniques that take that change block tracking data and work it in reverse. So, hey, I'll just manipulate the files back from a block level on the disk, and then you're back and running. And just restore the change blocks? Is that what you're saying?

Mm-hmm. Okay. But the problem is, what if you delete the VM? Or what if your SAN fails? Or the other data is corrupted. Yeah, so you can quickly – or you lost your VMware host or you lost that host or that cluster, that storage network. All those situations. That really only works for somebody fat-fingered and deleted a database.

Let me bring it back. Yeah. Or a bad Windows update. It's a narrow use case. Yeah, there's lots of cases where that works. But not in anything you would think about using the word disaster case with. Right. It's definitely user error stuff. But component failure.

Right. But the problem with reverse change block tracking is that it's faster restores, not eliminating restore time. That's true. Absolutely. It is a faster restore for a situation, but that's why I like to kind of fully think about what could go wrong. You could lose your fiber channel switch. Then you're screwed because you can actually do this in four ways that we want to count. Right, right, right, right, right.

Okay, so a lot of these – the 39 scenarios that you've talked about, and Instant Restore and the Virtual Lab and stuff like that, are those sorts of things available for Hyper-V as well as VMware? Good question. So when we hit version 7 last year, we pretty much had feature parity. I believe the only thing that we didn't have in version 7 was vCloud director support when you kind of add those numbers up, of course,

because that just doesn't apply to Hyper-V. And then we had some replica virtual lab capabilities that didn't apply to Hyper-V. Those will come in version 8. But if you want to say, I want to pull out an email message or a SharePoint item looking into version 8 SQL Server record or maybe just a SQL database or an Active Directory user or, oh, snap, a note item or a calendar item out of somebody's mailbox. Those types of things, it adds up. Of course, files out of the guest VM. And then additionally, maybe just – in fact, this is one of my little favorite sneak tricks is just give me the VMDK.

And then from there, I'll let the application admin do their thing. So a lot of times, I have to have different conversations with different people. If I'm talking to an infrastructure person, I kind of think below the app, you know, okay, here's the whole VM, or here's everything that was on that data store. If I'm talking to an app person or the generalist, maybe, then I'm like, okay, this Veeam Explorer for Exchange thing, this might be exactly what you need. You don't know all the Exchange command line tools or those types of, you don't know how SharePoint databases work. Let me just show you how to make it easy for you to restore. So depending on who I'm talking to, I really deliver a situational message because it's a lot of stuff when it comes to backup.

Because just to say you have a VMware or a Hyper-V VM really doesn't tell me anything. What's inside of it? How many of them? How big are they? And Howard's point about those very large file servers, when you come underneath the infrastructure at the virtual disk level, VMware or Hyper-V, now that they both have virtual disk formats larger than two terabytes, you can have incremental backups comparatively just fly. I remember when I was at a place I was at before, we had like a seven terabyte or maybe it was four terabyte file server, but the incrementals were only like three hours. And that was before it

changed block tracking. So, you know, it's it's it can be awesome if you get it there now does hyper-v support change block tracking kinds of things or trick question so hyper-v does not have change block tracking in the form of an api like vmware has now that's not good or bad. It's just different. Now, System Center has a change block. System Center Virtual Machine Manager, when you manage a host through that, has a change block tracking mechanism, but you lose it if you move the VM to different storage. Or, I'm sorry, to a different host.

You'd always lose it on different storage because every block would change. So what we've – go ahead. And just in terms of general incremental backups, NTFS keeps a change journal. So if you're doing an incremental backup of an NTFS volume and your backup software is smart, it can ask NTFS what files changed and get a list and not have to walk the file system. Yeah, but the world is moving to SMB3, right?

So the question is, you know, in this environment where I'm using Hyper-V, VHDs, SMB3, and all that stuff, and God, even SQL Server and SharePoint can go directly to SMB3. You know, how does this play in this game? Well, so we did write specifically a file system filter driver that we install on Hyper-V hosts, not on the guests, but on the hosts. Because it's a different – we don't want to get in – we could, I guess, get into the business of making Vibs for VMware. But the API framework is so different. And then if you think about hyper v as a host it's easier to develop around that so the service we built the windows

service the part of the on host proxy is what we call it has a file system filter driver that will track those blocks of vhds and vhdxs. And honestly, it's pretty slick, and it does support migration to different hosts. So we went that way, and actually, just last week, just this week, we got a note that that implementation hit certified for Windows Server 2012 R2 and 2012 and 2008 R2 for our Hyper-V. I mean, we've been Microsoft partners for a long time, but we've been working for additional kind of designations, and we got a certified level for that technique.

So it's good stuff. Yeah, yeah, yeah, yeah, yeah. That's interesting. The 20-year best thing that happened to data detection in 20 years. Instant recovery. I am impressed. All right, so now I've got to bring up the downside of some of this stuff. Being able to recover an email or a note or an email folder or something like that,

I think all that makes a lot of sense and is relatively painless when I screw up my emails and stuff like that. I really want that sort of capability. The challenge I had, and I wrote a little bit about this in the post I said, was doing those sorts of things for Active Directory or SQL Server or SharePoint, elemental restores, like restoring a table or restoring a row in a database seems to me very dangerous. Well, it can be. And

we made these tools for people who may or may not be. Like, for example, Veeam Explorer for SQL Server. If I'm a SQL Server DBA, I don't know if I'd use it yeah i don't think you need it because i get the logs and journals and i can you know i can go play it

back and forward and i don't need the problem i don't yeah and if i'm a top dog exchange engineer for a large environment um yeah and i know how to use ese util in my sleep maybe i don't need that either but i'm not saying you have to use that. I'm just saying it's there. There's 39 ways out of a problem. As a former Exchange monkey, I can tell you you still do individual item restores. Using the backup capabilities or the restore capabilities rather than the utility. First of all all you get users who don't handle data well yeah the technology is getting really smart but i can't

say the users are either no no no every place i ever administered exchange i would come in and discover that they were never doing garbage collection. And I would set up a rule on the Exchange server that said anything that's in the deleted items folder that's more than 90 days old should be deleted. And then I set deleted item retention on the Exchange server to 90 days. So it's 180 days for the data. Before it's gone. Yeah, there's plenty of ways to do these apps wrong. Don't get me wrong. Every time, without fail, some senior executive comes around after the data has been deleted.

181st day, mind you. And says, but I was using deleted items as an important part of my filing system. No way! Howard! My folder's gone. Which folder? Same on him! Some folders of deleted items. And we send out memos It was subfolders of deleted items. And we send out memos, you know, 30 days before the first job runs and 15 days before the first job runs and four days before the first job runs. I ignore most of those emails. We're going to do this.

Yeah, yeah, yeah. I've got a folder of my deleted items for those messages, Howard. I don't read them. Exactly. got a folder of my deleted items for those messages, Howard. I don't read them. Exactly. They go directly to the deleted items folder. I was working at a large advertising agency in New York who had the contract for a major computer vendor who shall remain nameless. Oh, God. And the vendor put the agency in review and decided to assign the account to a new agency. But the new agency wasn't going to be ready for 60 days. So the agency that I was working for got a 60-day extension. Oh, God. $2.3 million. Which they deleted all the information about, yeah. And they deleted the email that authorized the $2 billion.

$2 billion? Million. Million, yeah, okay. This is a very valuable email. Yes. And a year later, a request came to my desk. Can you find this email? Oh, gosh.

So, you know, these applications. And I found it. Good for you. And then I could restore that one email back to the guy's mailbox. Yeah. I won't talk about how many weeks it took me to find it. Yes, no, I understand. But, you know, this story comes to mind every time I hear people say, we have a 30-day email retention policy. Yeah.

Well, it goes back to this elemental recovery or elemental restore. You know, I understand emails. I understand emails. I understand emails. Database tables, database rows active directory at units and those are in my mind more dangerous than an email oh they're very much dangerous i agree but um it's not like it's going to do it automatically for you no no i understand this is the best analogy and i've been using it a lot, these applications and these infrastructure solutions

that I play in, great idea, right? Exchange recovery, SharePoint recovery, but where and how you use it is much like a bicycle. And what I mean by that is it's very dependent on which the terrain you use it. So if you take your bicycle on a nice, clean, smooth street, it's going to be great. But if you start riding in the grass or in gravel or try to cross a creek, it's not going to be so fun. Different infrastructures can be

that very much same experience. You take a tool like VMware and Hyper-V and then back it up with something like Veeam. But what are they doing inside of it? What are the users doing? It's not necessarily Exchange's fault. It's not necessarily, you know, let's let's say in fact i had an email just yesterday or no it was a spiceworks uh thread of a gentleman who needs to restore a sql server database test database by the

way to 2 p.m of yesterday and he only runs backups every night he doesn't have a dba doesn't have the logs and i'm like oh man i'd love to give you that new vm explorer for sql server last week because then it would help you but right now you're limited to you know the last time you took that image based backup but i i'm going to put it in a category of what i call daily disasters and they happen they reallyeting a file. Deleting an OU out of Active Directory. Deleting... Imagine if someone... Imagine an upset,

disgruntled employee time-bombs some sort of script that disabled all computer accounts with a service account the day after he leaves or something. I mean, imagine how bad that would be. They're knocked out the OU of users. I mean, I don't want to scare people, but I'm just saying.

I haven't been that bad, but I have. Howard! Howard! No, no, no. Yeah, allow me to rephrase that. I am never that bad, and I have not seen anything that bad. Yeah, that's good. But I have seen administrators delete a group and then discover that that group had the only permissions to some folders. Oh, yeah.

And the easiest thing to do is just restore the group because you get the same SID. Yeah. The SID is actually very important because our tool, the new tool, VPSplorer for Active Directory, does maintain SID identity. So little details like that can make a huge difference in that, like I said, those daily disasters, everything from users to infrastructure problems to bad apps to bad scripts, real problems, malicious problems, plenty of things can go wrong. That's why I like having that large arsenal of ways out of a jam, and it includes, of course, bringing the whole VM back or maybe just a file. I mean I don't have a breakdown of different restore scenarios, but I bet. Well, it just turns out that I do. Okay.

Really? Well, I just finished writing for Information Week Analytics our annual state of the backup report. Oh, my god. Which is coincidental since Ray chose our guest this week. And so we send out a 40-question survey that asks people how they make their backups and how they do their restores. And every year when you look at, you know, what kind of restores do you do,

almost everybody does single file restores all the time. Yeah. When you start talking about full system restores, only 8% of the people in our survey do them at least once a week. And only 35% do them occasionally. So a test kind of a scenario? Well, I mean, if you have a big enough data center, you'd probably have systems failing weekly.

Okay. But 12% say they do single file restores on a daily basis. Yeah. 19% say they do single file restores on a weekly basis. I include myself in that latter category. And for entire systems, that's 2% and 6%. Yeah, yeah, yeah. That's true.

You know, it is a huge, I'd say that even, then you get more specific like application stuff and it's a very long tail beyond that probably. Right. Well, we ask about individual items

and mailboxes and stuff like that too. But, you know, too frequently when we talk about backup, people think about the full system restore because that's what they're afraid they're going to have to do. Worst case. But that's not what they do all the time. And so they're protecting themselves against that worst case scenario while making their lives more difficult on the things they do frequently. But that's where these 39 different restore scenarios pay vast dividends, I guess, right?

Exactly. Because you're backing up the whole system, but you're giving them the ability to restore a virtual machine or restore an email or restore a table or restore an OU or whatever, things of that nature yeah the only the only catch point is though that too many times again people have those bad historical stories with their own professional experience with backup and it sometimes is an afterthought and what i mean by that is a lot of times they put backups on inferior storage and you guys will agree with this that you can always find lesser expensive storage right and so if you keep looking you'll find a oh wow that was half the price of this one oh look at that that one's 30 less than that you can just keep going right and the performance usually goes

with it and then you end up at some guy who bought an quanta server and filled it full of Western Digital drives and installed Linux on it and said, here, it runs NFSD. You can back up to it. Exactly. And then, again, remember the bicycle. Your experience will depend on the terrain you go. And when we talk about

larger data profiles, especially VMs, it's not necessarily fun. And I know Howard and I have been talking about some different disk targets and stuff, and I'd love to see people put production class stuff to hold their backups. Granted, I'm not paying the bill when I'm making that recommendation. Well, I mean, there are substantially different requirements. Oh, God. The IO requirements for backup versus

normal activity? Absolutely. Yeah, I mean, if I was designing the backup target of my dreams, it would have just a tiny bit of flash for metadata. But since everything, since all the real data is

primarily sequential, spread it across a bunch of spinning disks. That's going to work fine. I was going to say spread it across a bunch of tape drives and stuff with a tape library and stuff because it's all serial and sequential. High throughput, you know. Except on the stores. Yeah, yeah, yeah. for anybody who's using Beam, I will say this, and I'll tell anybody this. Our highest performing environments are always

a physical server with a JBOD attached to it. And the only feature I want on that storage is RAID. JBOD? What? I'm serious. It's the fastest target. Backup target is a JBOD on a server that's running this? Yeah, like a

large shelf of drives? Yeah, yeah, yeah, yeah. On a SAS array or maybe even direct fiber to it? It's always the fastest. Oh, my Lord. It's just got scale problems. Yeah. As many – like I'll give one example.

The Cisco UCS C-Series, they have one that have 24 local drives on it. Right, right. Even for rotational storage, that's a lot of capacity and a lot of spinning drives on one array. That works pretty good. Are you somehow optimizing your access to those drives to continue to spread the workload across them and stuff like that? Well, if it's one big volume, would leverage the uh the operating system to do that so no we don't have very specific individual drive uh well that's that's the plight of agnosticism i'll be honest with you because we are technically storage agnostic and when i mentioned that design there of being a jbod that would be a windows system with like an ntfs volume

so it's up to windows and to service it better, yeah, yeah. Yeah, yeah. And now that we've got 24 cores in that thing, Windows software RAID looks a lot better than it used to. Yeah, okay. True, true. We are almost at the end of our time together. I have a couple of final questions, and then I'll turn it over to Howard. So, you know, all this focus on virtual

machines and Hyper-V and KVM, those sorts of things. I understand that, but do you guys do just plain backup for physical servers that are not virtual machine oriented? Not with Veeam backup and replication. You know, we get that. That's the number one most requested thing to support physical servers. Yeah, yeah, yeah. But just the logic that the product has right now, no, is the answer. Okay. I think that's my last question. Howard, do you have any last questions for Rick?

You know, the biggest question I have is really, does Rick have some explanation as to why when we keep doing surveys, and it's not just our surveys, that most people, when you say, how are you backing up your VMs, are still using an agent in every VM? I mean, I understand why you only want one backup system so you're going to keep using NetBackup for everything, but I don't understand why you're not going to use the vStorage API for data protection. Well, I have to fight that battle every day. And then sometimes, you know, not that people have made bad decisions in the past with those products. But they have. I mean, it's just like, I spin it this way.

They might be missing out on these frameworks. If you understand vSphere APIs for data protection, if you understand Hyper-V and VSS on the host, if you let me explain that to them, they have a chance of saying, oh, wow, that means my file server of 15 million files or 15 billion whatever files and 8 terabytes might speed up. Oh, yeah, yes, it would. So if I get that through their head, then I have a good chance of letting them give it a try. And that's what it comes down to. Backup is one of those things that, you know, people put their own kind of career on it in a sense, because everyone knows a failure that you can't restore from is indeed a resume

generating event. So I always tell people, don't listen to me. If you have a question on how something works, I'll tell you. But I always tell people, download it, to me. If you have a question on how something works, I'll tell you. But I always tell people, download it, try it. In fact, I get the question all the time. This product versus Veeam. That product versus Veeam. I say, you figure it out. Download it here.

Put them side by side. Try to do this. Try to do that. Then let me know if you have any questions. So, yes, people still go a different route. And I get that. I do. It's a big market. Honestly, we have a lot of opportunity even still. I mean, we have some, you know, impressive milestones that we've had, but I don't think we'll ever be done reaching out and finding those people that can do, you know, protection better and really deliver

an always on business. That's what it's all about is keeping those users away from us. Okay. Well, that seems like a good point to end this discussion. This has been great. Thank you, Rick, for being on our call. Sure. Next month we will talk to another startup storage technology person. Any questions you have, let us know.

That's it for now. Bye, Howard. Bye, Ray. Bye, folks. And thanks again, Rick, let us know. That's it for now. Bye, Howard. Bye, Ray. Bye, folks. And thanks again, Rick. Until next time.