Grey Beards on Systems - 147: GreyBeards talk ransomware protection with Jonathan Halstuch, Co-Founder and CTO, RackTop Systems

Episode Date: April 25, 2023

Sponsored By: This is another in our series of sponsored podcasts with Jonathan Halstuch (@JAHGT), Co-Founder and CTO of RackTop Systems. You can hear more in Episode 145. We asked Jonathan what was wrong with ransomware protection today. Jonathan started by mentioning that bad actors had been present, on average, 277 days in an environment … Continue reading "147: GreyBeards talk ransomware protection with Jonathan Halstuch, Co-Founder and CTO, RackTop Systems"

Hey everybody, Ray Lucchese here. Jason Collier here. Welcome to another sponsored episode of the Greybeards on Storage podcast, a show where we get Greybeards bloggers together with storage assistant vendors to discuss upcoming products, technologies, and trends affecting the data center today. This Greybeards on Storage episode is again brought to you today by Racktop Systems. And now it's my great pleasure to introduce once again, Jonathan Halstuck, co-founder and CTO of Racktop Systems. So Jonathan, why don't you tell us a little bit about yourself and what's wrong with ransomware protection today?

Sure. Thanks for having me on the show. I'm Jonathan Halstuck. I'm the CTO and co-founder of Racktop Systems. And I've been involved in data security and offensive cyber for nearly two decades now. And I think that the approach to ransomware has been very much one of recovery and talking about cleaning up the mess after the event has happened. But with cyber, we really want to get ahead of the event and we want to think about how do the mess after the event has happened. But with cyber, we really want to get ahead of the event and we want to think about how do we thwart the attackers. It's a very different problem than when you think about natural disasters and floods or catastrophe hitting

your IT system and having to have a backup or disaster recovery plan. There's no way to stop the hurricane. Right, right, right. So I mean, the big challenge, I mean, obviously backups and stuff like that are all good things to have and stuff like that. But from a ransomware protection perspective, that might work for most cases, but there's more to ransomware than just protecting, I guess, right? Yeah. So what we're starting to see is that it's really becoming extortion where the adversary or these criminal organizations are stealing data first. And sometimes they're not even proceeding with a ransomware attack. Usually they proceed with a ransomware attack to make it more painful to recover as well as prove, hey, we had access to your system.

And so, yes, we stole the data. And let me show you how I have control of your system by encrypting the files there. But now we're starting to see some, even in the news today, lots of, like with Western digital organizations like that, where attackers are claiming they have sensitive proprietary information that they're going to leak and they're extorting both the victim they stole the data from, as well as sometimes if there's sensitive customer information, they'll go to the customer and try to extort that customer as well. I was going to say, I've seen as well on this, and I'm curious to see what you've seen within the industry as well, where these ransomware attackers are getting more and more sophisticated, as in they will infiltrate a system and not necessarily, it's not going to be two weeks that they pull the trigger on that ransomware. It might be, you know, they'll go for, say, an average of a year retention policy on backups, and then they might invoke that ransomware kind of,

kind of, you know, after a year where all your backups are already infected as well. Is that something that you've seen in the industry as well? Yeah. So what we see is on average, right, a data breach is on the network for 277 days before being detected. 277? That's right. And dropping that down to just 200 on average can save you a million dollars. So that's a long time they're on the network. So what are they doing during that time? Well, they're trying to create persistence, right? Some of these organizations are selling access to your network, right? And so they want to create multiple backdoors, multiple ways they can get into that environment. So once they are discovered, once they launch this first ransomware attack, they can actually collect the money and then potentially go after you again.

Because the probability of once you've been hit once with ransomware, the probability you're going to be hit a second or third time goes up. So they're definitely spending that time looking around the network to find the valuable information they want to target or steal or encrypt and creating persistence for the next time they want to get there. Because they could even sell that access to another gang or or entity that's interested in in that uh target network so so backups really aren't a solution to any of this stuff no they're really your kind of last resort yeah yeah yeah yeah so what what what does the customer try to what does the customer have? What can they do? So before an attack, no matter what you're doing, it starts with good cyber hygiene.

So you want to reduce your risk. You want to minimize what could be compromised in the case of an attack. And so with our product, what we do is we have integrated compliance reports that show you the access control list for all the data sets. So you can look by user and see or group, connect to Active Directory and LDAP, what data sets do these users and groups have access to, as well as look at data sets and see what users and groups have access to that data set. And so the data owners or project managers or management can see who has permissions and access to that data set. And so the data owners or project managers or management can see who has permissions and access to that data set. And then they can also, through user behavior, look to see who's actually been accessing the data. Because you could have very sensitive data sets out there that are widely exposed but not being used. And if someone's credential gets compromised that has access to that data set or all the data sets, that's a bad day for the

organization. So you want to really restrict that and limit it to lowest privileged access. So that's kind of what you can do before an attack, right? And help prevent that. And then the other thing you can do is make sure you're not using admin credentials when you don't need to or exposing those because the adversaries typically will try to get into the environment and then they'll escalate privilege to an admin credential so they can create persistence and go find the widest amount of information and steal the most amount of information and then encrypt the most amount of information by using a privileged account. So detecting that admin activity is an effective way of detecting those breaches earlier, closer to the, as soon as they get on the network versus 277 days, right? And so that's another key

part. And so then during the attack, you want something that's going to be able to actively detect that and stop it. And that's really the difference in what Racktop offers with our proactive defense or what we call active defense. We're analyzing how users and applications are interacting with the primary copies of data, right?

The bad guys eventually are going to try to potentially delete snapshots or backups, but they're after that primary copy of data. And that's the one that has been often unprotected. We need to protect that with data security and data protection. And when I talk about data security, it's providing controls to make sure only authorized users get access to it. And it's really the person that says they're them and being able to analyze what they're doing. And then if we detect suspicious or malicious behavior, in this case this case ransomware if somebody all of a sudden starts an account starts reading a bunch of files and encrypting them well that looks like ransomware

we want to stop that as close to uh you know as soon as it starts happening now is data encryption helping any of this stuff or is that something you guys do in addition to everything else or yeah so data encryption doesn't really necessarily help you know lots of people talk about at rest data encryption and there's value to that but in stopping a ransomware attack it's probably not going to help you there having encryption you know over the protocols can help because it it potentially prevents that adversary from getting the foothold in the network to exploit vulnerabilities or to steal credentials to be able to get onto the network to then start to steal data or launch a ransomware attack. But data at rest, using self-encrypting drives is valuable for other things. But to stop a ransomware attack, it's not likely, right? Usually, they get hold of an account that has privilege, and that account would be able to decrypt the data or do whatever they need to do.

A friend of mine, he actually had a company that was doing this kind of on more of an on a networking side. But he had some statistics on when you look at basically nation state attackers and how quickly they get from, you know, could be end user clicked on a spam to the point they get like a second level penetration into something that actually has privileged data access. And it's actually terrifying how quickly they can get in and start basically doing exactly what you said. And it's like a virus that spreads throughout the organization, right? Exactly. Yeah. So you have to be able to act quickly. And that's why people are looking for automated tools, right? We can't, you can't look on Monday to see what happened, you know, last week or over the weekend, you need automated tools that are going to detect and respond. And that's why you see, you know, AI and AI operations and other things and insecurity to help aid, you know,

the humans that are, that are supporting the infrastructure. Well, it's a, it's an arms race, right? Cause it's the same ransomware guys are also trying to use AI to, you know, facilitate ransomware or, you know, spamming or spiffing. I don't even know what it's called. Right. But ways of trying to get people to click on things and stuff like that. So it's a sort of an arms race in that respect. Do you guys use AI in your detection capabilities?

We do. So we train the models and we have assessors, and they're using AI to detect unusual behavior and access over the protocols, both we support SMB and NFS up to the latest versions, and then what's happening with the files on the file system itself. So are you looking at basically kind of the an existing environment at a customer site, and then and then creating a basically a training set upon that, and then using that to, I guess, you know, like basically train a neural net to go through and do inference on

any modifications. So that can be done for some of the long-term behavior, but some of the immediate stuff like stopping a ransomware attack doesn't really require training at the customer site. We know how ransomware behaves, which is it's, you know, reading the file and overwriting it and doing a bunch of other things. So we're looking for those behavior patterns of that malicious software. And so we're doing that and creating that in our so they've got the the rapid response inference engine that can they can they can look at that uh and protect the patterns right exactly got it yeah so it's almost like ransom it's almost like virusware to some extent i mean you're looking at patterns that you've detected or seen in the past and using those to train your more general

assessor capabilities? Is that how it works? That's accurate, right? So for things we know have a specific tactic and technique, we can build confidence that that is that variant of a ransomware attack or something because we're looking for multiple factors. So we're looking at a variety of things and we can build up confidence much more quickly if we know a bunch of things. In general, for like a zero-day type of ransomware, we're using the same general patterns. We know you can't encrypt something without reading it and deleting the original one or overwriting it. So we're looking for those things. It just takes a little bit longer for us to build up that confidence that, yes, this is a ransomware attack, not just some user encrypting a file to send. And so we use multiple points of data to create that confidence.

And that's where the value also comes in the fact that the data sits on our file system because we have more visibility into what's happening than if we were just kind of a bump in the wire or a proxy. You brought up, you brought up like, just like say a user encrypting a file. Have you, is there any really weird kind of false positive things that you've ever seen with this or something that triggered it that was, that was actually a user behavior? So we've gotten a lot better. You know, when we did the initial testing, we had kind of, when we just were eating our own dog food trying to do you know see what would happen and see what would cause a false alert or trigger we realized that we made some

assumptions and this was before we kind of deployed it and so we've been to the point now where we've been able to eliminate those false positives for the most part every once in a while you might run across an application that you know not just a user doing stuff but there's certain you might find someone's custom application that they wrote does some odd things that almost look like ransomware, but that's pretty far and few between. Yeah. So you guys have pretty much have the capability of, you know, handling those false positives and allowing the operator or admin to tell it's okay and stuff like that. Right. Exactly. So once you see that, it can create an incident. You can whitelist it.

You can choose the rules you want to happen to. You can say, hey, I still want to be alerted when this happens, but I don't want to block it. And so it has a lot of flexibility in how you can figure that. But it really is good about giving you visibility, too, with the user behavior auditing to see, hey, who's accessing this data? How often? Where are they accessing it from and then you know

what the pattern of life looks like for the data and and are they we've we definitely right away when we deploy the system start to improve cyber hygiene because there'll be times where they're like oh yeah i forgot you know we're using admin accounts to do that but we really shouldn't be so we'll go and clean that up or other cases maybe there's a reason they have to use admin accounts they can whitelist that in the rules and allow that to happen. Well, that's one of those things too. That's too, from a false positive perspective, it's definitely better to be safe than sorry when you're talking about

ransomware and potentially have your data hijacked. Right. Yep. For sure. There's no way you can't get the genie back in the bottle. That's right. That's right. And you guys are, you know, it seemed like you've always been associated with fairly security sensitive environments and that sort of stuff. So, I mean, these sorts of the technologies that you're talking about have been kind of proven in real world, highly secure environments. Is that true, Jonathan? Yep. We have a wide customer set from the DOD all the way to state and local governments and in the private sector, hospitals, healthcare

organizations, people that have been the target of ransomware. But we are able to provide this security and compliance capabilities without sacrificing performance. So we're able to deliver at scale and at performance the file share and NAS capabilities, but we embed the proactive security. And if you think about it, a lot of times when people are looking at a storage solution, they talk about high availability, right? That comes up typically at some point in the conversation, they choose to employ AJA, right? Well, if you think about it, you know, people want these five nines, three nines, something like that, right? But that's for just typical operations. If you think about something like a ransomware attack where the data is going to be encrypted and not available, you can really go

below no nines in that type of situation if you don't have a way to stop and recover very quickly and return to service. And that's what we're providing. And from a storage perspective, because a lot of storage folks think about HA, I think maybe they haven't thought about security as long, but really when you think about availability for the data, the confidentiality, the integrity of it, you really need to think about it this way. And with our solution, we're able to stop it early, right? You don't want to have to recover from a big mess. You want to make it as small as possible, make the damages as minimal as possible, and then, yeah, and see what was affected and then quickly recover from the versions of those files that are contained in the local immutable snapshot. That's a lot faster than having to even go to a backup that could be stored in the cloud

or someplace else, figure out what file it is and make sure it's clean and bring it up. I mean, if you look at kind of the way some of the organizations are thinking about recovering data and using clean rooms, that could be a multi-week process even to recover a small amount of information. We're trying to get you returned to service in minutes and really keep and protect, four or five nines of availability that you're looking for. And so in your case, I mean, so customers would still deploy backups and that sort of stuff potentially for your solution as well as having this real time ransomware detection kind of capabilities, right?

Yeah. So, you know, we partner with the backup companies in areas, especially since there's data that doesn't exist on us. So we frequently have customers use the leading backup solutions to backup data from their desktops or physical servers or virtual machines in cases and use us as a backup target. And we're a safe place to land that data. Sometimes there's organizations that have that 3-2-1 theory where they'll use us for the primary. They'll also have us do snapshot replication to a second brick store. And then they're sometimes backing up with a third tool, the file shares and either vaulting that data or putting it on tape or doing whatever makes sense

to them. But we support all those use cases. And if you look across our customer base, there's definitely people that do work, leverage backup solutions in combination with Brickstore. So what is your typical deployment model? Do you guys typically deploy as like a hardware-based appliance, a software-based appliance? How do you typically deploy? So that's definitely changed over the past couple of years. So in the beginning, we deployed as a traditional turnkey NAS with direct attached disk. And then in 2020, we started to deploy a software only where you can deploy our software as a virtual machine on your favorite hypervisor, HCI, or in the hyperscalers in the cloud. And it's the same

software. So it's completely interoperable. You can have a virtual instance replicate to a physical instance, bare metal, vice versa. And then we also have a partnership with HPE where we sell our software through HPE Complete. And in those cases, we're fronting HPE Enterprise Storage with a DL360 or 380 running our software on it. And so you can present LUNs from that HPE enterprise storage to us and we can present cyber storage from those LUNs. And now we've taken that even further where we work with, you know,

all the popular enterprise storage providers, Pure, anybody really, to provide cyber storage on existing block capacity. So they can take disparate SANs or one SAN and basically present cyber storage with our software and get that protection.

And that's probably been the biggest growth area of me. I hadn't heard that you guys support other storage solutions, providing block storage, and you're providing a front end as a file service, NAS, ransomware detection, cyber secure environment. How long has that been going on? So I think it's been about 18 months, and it's definitely been accelerating the adoption of that. Yeah, I would think so. Yep.

Because a lot of customers already have block capacity or they love their SAN vendor. This gives them a way to continue to leverage that capacity and then grow it as needed and then get the cyber storage capabilities on top of that. And I think you mentioned earlier, then that's presented as effectively a file-based system. So are you doing NFS, what versions,

and then also SMB, is that correct? That's right, yeah. So we do NFS v3, 4, 4.1, and 4.2, and then we support SMB. We even support SMB1 SIFs, but we alert you that you're using that so you can be sure you're kind of scrutinizing what's going on. And then we have SMB2.1 and SMB3.1.1.

And so we support, you know, signing and encryption for both NFS and SMB as well. Got it. Do you also have any type of tie-in into basically authentication services like ADFS and stuff like that? Yeah, so we work with Active Directory and LDAP, and that's pretty common. We can also work with the Azure Active Directory implementation as well.

Oh, that's great. That's great. That's great. So effectively, the problem with backups today is that it's after the fact. I mean, by the time that your backup is detecting ransomware it's it's occurred it's they've they've stolen your data they could expose it they can hold it at ransom there's lots of things they can do at that point that's already too late to to put the genie back in a bottle is that what you're telling us exactly it's like if you had a car would you rather have collision avoidance or collision detection? I'd rather avoid the accident versus

have the car tell me I've been in an accident. Yeah, that's interesting. That's interesting. You mentioned the cloud. So the cloud solution is effectively a software-defined storage based on whatever the cloud block storage is. Is that how this works?

Exactly. So we would take some of like an AWS Elastic Block storage and attach that to a virtual machine running in AWS. And you could also do the same thing in Azure. You could have cross replication. We actually have customers that have an on-prem instance, and then they replicate to both Azure and AWS for cross cloud replication. And then with our TDM capability, you can actually tier data to an S3 compatible object store. So in the case of AWS, you could tier the data to an object store as part of the TDM capability as well.

If you have, so you detect like something is going wrong, something is going awry with a piece of, of you know, block, you know, with a piece, piece of block data file or something that you think like there's something nefarious going on here. What, walk me through kind of that process of what your system does and how it kind of alerts the, the administrators and users on what's going on.

Sure. Yeah. So let's take, for instance, ransomware, because that's been the major topic. So if all of a sudden we start to believe that files are being encrypted with ransomware, we're going to know where that file access came from. So we're going to basically create an incident within our system. It's going to block that user account from accessing further data. It's going to block the client IP from accessing further data. And it's going to put a hold on what we call rolling snapshots. So we have like a DVR of immutable snapshots for the last five minutes, where around the time of an incident, we put holes on those snapshots so that we can recover, restore any files from immutable snapshots that might've been affected. And so then we can also

send alerts through webhooks. So we can send a Teams chat or Slack or PagerDuty message to the admins, both the storage admins, as well as the security admins, depending on how the organization's set up, or we can fire off an email. They can go and see that information in their SOC tool if they want, or they can go to the Brickstore GUI interface and they can see the incident. They'll see what account was basically implicated, the IP, they'll see the actions we've taken, and then they can also see the files that were affected. So if it was indeed a ransomware attack, they would see the files that we see affected, like the encrypted version of the file, as well as the version of file that's in the

snapshot that we believe they should recover from. So we create a recovery plan for them so they can run that recovery plan. So even if they use, so even if the attacker was able to encrypt five or 10 files before we stopped them, this would actually allow you to very quickly restore all of the files out of the snapshot and remove the encrypted files in just a couple clicks. And then when you're satisfied that the user's been trained appropriately and the machine has been cleaned up, you could then grant access back to that user account and machine to access further data.

So you can really return to service in a matter of minutes. And it's very easy to go through the workflow that the incident management pane puts you through for the admin. It doesn't require advanced cyber knowledge. That's great, man. Almost automating the whole recovery process as well as the automating the detection and, and ceasing and assisting the operations.

So that's great. That's great. All right. Well, this has been great. So Jason, anything, any last questions for Jonathan before we leave? No, I just, you know, hey, wish you best of luck. So this sounds like a very, very cool product, very cool company. And this is definitely much needed within the industry. Thanks, Jason. Yeah, today especially. So Jonathan, is there anything you'd like to say to our listening audience before we close?

Well, check us out. You can see a live demo of us stopping ransomware attacks on our website. And also we'll be at HIMSS this week in Chicago, as well as HPE Discover in June. So come check out our booth if you're going to be there. That's fun. That's fun. Well, this has been great. Jonathan, thanks again for being on our show today.

And thanks again to Racktop Systems for sponsoring this podcast series, actually. Thank you. And that's it for now. Bye, Jonathan. And bye, this podcast series, actually. Thank you. And that's it for now. Bye, Jonathan. And bye, Jason. Bye, Ray. Have a good one. Until next time.

Next time, we will talk to the system storage technology person. Any questions you want us to ask, please let us know. And if you enjoy our podcast, tell your friends about it. Please review us on Apple Podcasts, Google Play, and Spotify, as this will help get the word out.