Grey Beards on Systems - 156: GreyBeards talk data security with Jonathan Halstuch, Co-Founder and CTO, RackTop Systems

Episode Date: October 12, 2023

Sponsored By: This is another repeat appearance of Jonathan Halstuch, Co-Founder and CTO, RackTop Systems on our podcast. This time he was here to discuss whether storage admins need to become security subject matter experts (SMEs) or not. Short answer, no but these days, security is everybody’s responsibility. Listen to the podcast to learn more. … Continue reading "156: GreyBeards talk data security with Jonathan Halstuch, Co-Founder and CTO, RackTop Systems"

Hey everybody, Ray Lucchese here. Jason Collier here. Welcome to another sponsored episode of the Greybeards on Storage podcast, a show where we get Greybeards bloggers together with storage assistant vendors to discuss upcoming products, technologies, and trends affecting the data center today. This Greybeard on Storage episode is again brought to you today by Racktop Systems. And now it's my great pleasure to once again introduce Jonathan Helstuck, co-founder and CTO of Racktop Systems. Jonathan, it's your fifth time on the show and we're glad to have you back again.

Do you think storage admins need to also become cybersecurity subject matter experts in order to secure their use of storage systems? I don't think they need to be a subject matter expert, especially when you have the right products. But I do think that everybody has the responsibility for cybersecurity within the organization. It's almost like marketing. Everybody's responsible for marketing regardless of what their position is. It's kind of bizarre. Exactly. I think a lot of times people think, you know, or the traditional approach has been, well, I'm in can be the backdoor for these cybersecurity breaches, data breaches and hacks. So you got to think about how do we tighten these up? How do we improve our data storage and security posture to make it resilient against attacks. Because a lot of times, you know, if I were to ask you as a storage subject matter expert, you know, about high availability, does it matter and is important? People would respond

instantly, yes. Yeah, for sure. And what they need to start thinking about is data availability, right? Kind of take it up a level and think about, okay, well, the infrastructure, the storage system's up. Now the next thing is I have to serve out data, right? Could be like for NAS, right? I have to be able to serve SMB and NFS and serve files to the applications and users that need the data. So now I need to give them the data intact, right?

We think about RAID, we think about data integrity. We wanna make sure the data is not corrupted from drive failure or system failure. Well, we also now have to worry about corruption due to malicious actors, right? They could be deleting the data. They could be, you know, silently just manipulating pieces, right? We're worried when it comes to AI and ML about people poisoning the data. And so we want to make sure that the data that people are storing is the data they get back and that they can get it back whenever they need it. Poisoning data. How does that play out? I mean, so you're actually modifying the data

subtly so that it's just wrong or incorrect? Yeah, essentially what you're seeing in some of these machine learning models is that they start with some amount of seed data and they should expect to get a certain result. What we're seeing is you could potentially poison those models or change the results by slight manipulations in the data. And if you can do that. Yeah. Yeah. Yeah. Huh? That's interesting. I was going to say, have you seen anything new that these ransomware attackers have been doing over the last year, two years?

It's kind of a different tactic. I mean, you mentioned the kind of the AIML component of that. Have you seen anything that's been kind of drastically different over the last couple of years that these guys have been doing? Yeah, definitely. I think, you know, the big ransomware attacks in the beginning were strictly just encrypting the data on site on premises or wherever they were encrypting the data, and then asking for money to get the decryption keys. Then obviously, people were like, hey, we can recover from backups. But 89% of ransomware attacks today have an exfiltration aspect to them. So now they're trying to steal data as their primary thing, and then maybe do

some light encryption or rapid encryption to basically also disrupt operations. But they all have this concept where they're trying to steal data and use it to do double and triple extortion, where they're saying, hey, even if you can recover from backups, we're going to leak this sensitive data, so you need to pay us. And then they're also going to the, if they happen to steal some of their client's data, go into the client and trying to extort money out of them as well. Oh, this is, this is painful. God. So, I mean, how do you, God, so if they're just stealing the data, it's sort of hard to find out what's going on. I mean, how do you detect something like that, Jonathan? I guess the question.

Yeah, so with the Brickstore, our solution has excessive file access capability. So we can look for things like a data exfiltration attack where a user's reading an abnormal amount of data, for instance, and maybe exfiltrating data. And that would be the theft, right? We're also looking at things like privileged access or admin user abuse. So if you think back about all these attacks, what tends to happen with ransomware and any cyber attack is they get in somehow and they try to escalate to an admin level privilege because that has the most access so they can move laterally, as well as be able to get the access to the most data. And so we want to alert very quickly when an admin credential is being used and scrutinize that type of behavior.

So we're trying to detect that, you know, the threat as early as possible, because on average, the adversary is on your network for nine months before they're detected. We want to get that as close to zero. And so with Brickstore, if that admin credentials used to read data off the Brickstore, we're immediately going to alert that, hey, an admin account coming from whatever IP is accessing data on the system. And also if it's trying to probe the Brickstore, looking at things, we'll alert on that too. So if you go back to the famous Sunburst SolarWinds attack, they got into the network through the software update

for the network monitoring software, and then they up-leveled to an admin credential and then used that admin credential to go look around for data to steal. As soon as they tried to look at data on the brick store, we would have alerted that. So potentially bringing that nine-month detection time a lot closer down to zero. So what's the sort of cybersecurity posture, I guess, that you're thinking that storage admins

and actually everybody in the IT organization has to have in order to be a good player, I guess? Right. Yeah, I think before the attack, right, it's all about cyber hygiene and reducing risk. So I think there's tools that need to be available. And some of it is making sure, you know, the products that the storage team puts in place enables the end users or the data owners to have information to help

make decisions. So like one of the first things that you might think about would be access controls on a share. So, you know, people come and go from an organization, maybe they don't leave the company entirely, they just move from department one to department two, they switch roles, and they don't need access to the same data they did before they need access to different data. In an ideal environment, this information would be presented, you know, to the managers, the date owners to say, on a normal basis, yeah, this, this user is still working, still need that system, or they've moved and they don't. Right now, in a lot of systems, that doesn't exist in any place that's easy to access and readable. It's something that you have to write a PowerShell script for

to go through AD. So by having tools and storage systems like Brickstore that do that, you can reduce the risk that as somebody moves throughout the organization, they just have access to everything. Also, you want to be able to audit what data is being accessed and how frequently and by who because that helps inform if my access controls are way too laxed or mostly way too lax, right? Because you're going to see I have a wide, a widely exposed, sensitive data set, but nobody's actually accessing it. So to reduce the risk, we'd want to take that down and make it you know, either reduce it to just a few people or maybe disable the share until somebody requests access to the data again. So it's those types of tools that they need to be looking for.

So they need to think about how would I prevent an attack? How would I prevent an insider? How would I detect something like that, right? In fact, one in three attacks approximately come from an insider. So when you think about it that way, you know, what would be the things that I would want to be able to do to prevent that type of attack? How would they do it? So they do need to start to become educated, not to the expert level, but to kind of an introductory level about, you know, how do these attacks happen?

How could they get to the data? And then also maybe go through some exercises about, hey, if this happened, what would I do? Because a lot of people kind of like DR plans, right? You set up the replication, you set up the backups, and then do you ever actually go through the exercise of doing it? DR testing. Yeah, yeah, exactly. It's a painful process, but it's necessary to be effective. I mean, I don't know of any DR plan that actually worked the first time in my life, you know, to a large extent. And cybersecurity and recovery from that is going to be even more chaotic because you have an active threat going against you, right?

Like, think about you get hit with a ransomware attack. The adversary could potentially be on your network still looking to see what you're doing or monitoring things that are going on emails, etc. So it's going to be a crazy time. And you know, it's like a bad time to be trying to figure things out. So you really want to, you know, first go through something like a tabletop exercise. Okay, this is what's going to happen. we're going to do this and this and this and then actually try it okay so recovering from a real cyber security attack or you know a fictional cyber security attack just to see if you can do it and stuff like yeah that's that's even when you think about it's a hard one to simulate as well right yeah yeah it can be a little tricky i think you know there's way there's there's definitely organizations that do tabletop exercises and then there's.

But yeah, but it can be done. And I think it's, you know, back to kind of everybody's responsible. I think it's also important to have not just, you know, this isn't just an exercise for the I.T. team or the storage team, but it really has to be a little bit wider. I think you need some of the senior management involved to see how that's going to work because they're going to have to be involved, right? Once you get hit with a ransomware attack, law enforcement could potentially be involved, your legal team, the press, if this is public and you have to have relations and things like that. So it really is something that you don't want to learn on the job. You want to have a plan ahead of time. Right. So, Jonathan, if someone was interested in deploying this technology, what are your deployment methods? How does someone go around putting Brickstore into their environment? So the cool thing is it's software, right? It's an operating system that you can deploy

as a virtual machine or on bare metal and if you do bare metal we have kind of two major deployment options you can deploy as like direct attached disks like a traditional nas has been deployed for a long time or we can sit in front of an existing block storage device so you can take you know a lun Fibre Channel or iSCSI and present it to us, and we can create an HA cluster and present out SMB and NFS shares to give you secure cyber storage. And it's the same software in all these situations, so you can have it completely interoperable. You can replicate between each instance, and you can have one single pane of glass to manage them all. So you can deploy a virtual machine in one of the hyperscalers, as well as maybe another virtual machine on your HCI at the edge.

And then at your core, potentially have a physical instance of Brickstore running as well. Great. Yeah, yeah, yeah. So, I mean, Racktop's been on the security side of storage for quite a while. I mean, you guys have seen probably a lot of what's going on these days.

Jason had mentioned, you know, what's the new types of cyber attacks and that sort of thing. It's becoming, I mean, it's like every week there's another cyber attack. Somebody's exposed information and the data is lost or the data is stolen or something like that. It seems like, if anything, the criminals are winning this game. Yeah, I feel the same way. I think there's two things that I kind of want to say about that. So we, we can, I would say most organizations continue to spend more money and have started to become focused on cyber and keep making investments. Yet, to your point, it seems like

the hackers are winning because the investments have gone up, but so have the attacks of more and more breaches keep occurring. So I really think what that really points to is that the traditional approach and the focus on network security and endpoint security alone is not effective. You really have to change your approach. And remember, the bad guys are after the data. And so let's put the security and protections as close to the data as possible, which is where the data is living, which is where the data is stored. And by doing that, we can simplify the problem to an extent and make it better. I'm not saying you don't do endpoint or network security anymore.

I'm saying you do this in addition to and really solve and fill a gap that's in the security posture today. They really need to think about putting that security and active defense around the data itself because it's too late if you wait to detect it in like your backups. You just think about, you need to detect and protect. You need to protect your primary copies of data, the active copies of data, that users are working with

and actively in real time using AI and things like that, detect that and stop those attacks as early as possible. If you're scanning your backups looking for malware, it's at least hours, if not days, before it's going to get detected. And by then, it's way too late, right? And so it's got to happen there. And people can get to your data without being on the endpoint.

There's other things on your network. You think about hospitals, there's IoT devices. There's all these ways that people can get into your network. And a lot of people think, oh, I have multi-factor authentication, for instance, I'm good. MGM and Caesars look like they had multi-f And so you can't just say, hey, I made this investment. I feel that's good enough. You have to really think about, is this going to stop them? And probably continually evaluate your security posture to make sure you're doing enough to do that. And the second point I want to make is that, you mentioned what you're seeing in the headlines.

You keep seeing all these attacks. That's the visible ones that everybody knows about. That kind of starts with the ransomware, right? There's lots of attacks that are going out there that are silent, like the insider threats. They're not proving I stole the data. They're trying to steal your data and never be found out, right? And so those type of attacks that are stealing your intellectual property that are coming from an insider or a foreign nation state where they want to be able to access your research, intellectual property, and secrets without you knowing whenever they want. Those are the real

threat. It's a harder threat to detect, but it's somebody you still need to detect. And to do that, you really need something like Brickstore with Active Defense that's analyzing how users are interacting with the data or applications are interacting with the data and then alerting on that and detecting and stopping it. Yeah, you had mentioned earlier, too, that data can sit there for a while. And a lot of people look at backups as a way of doing, that's how they're going to protect against it. And I have seen, I'm sure you have as well, a lot of the ransomware attacks are getting more sophisticated to the point where they're laying dormant for a period of time. And they get a lot of kind of that inside information on how long are had been doing snapshots and they had snapshots going back to a year on their system.

And they had a set of ransomware tools that literally lay dormant for a year and was sitting in all of those snapshots. So all of those backups were effectively corrupted and encrypted. Ouch. Yeah, yeah yeah yeah well there's a you know somebody mentioned the dwell time but i think the dwell time of nine months is kind of the average these days it's uh it's a long period of time where these guys are just lurking in your systems looking around trying to figure out the best way in without detecting and that sort of thing. And, and once you've been hit once you're likely to more likely to be hit again, right? It's not like they're, they're like, Oh, I feel bad for them. We took them out once.

Some of these people are selling access to other, you know, gangs so that they get their chance at you. So. Yeah. The whole dark web thing. Yeah. Yeah. Yeah. It's interesting. It's interesting. So the real solution then is, is it's never static. It's a dynamic environment and you have to be constantly wary of,

of what's going on and, and keep your, your software up to date and keep your security posture red. Yeah. Yeah. And I think, you know, as you know, kind of back to do you need to be a cybersecurity expert, you know, to do this with storage? It's not that you need to be an expert, but what you can't do is make trade-offs and say, it's not my responsibility. You need to pass these options up the chain and be able to explain the impact of them to management and above, right? Because just like you may not be an SME or a business expert, you need to put the information about storage and data in terms that management and the executives will understand, right? Because

they need to understand that, yeah, if we get hit with a ransomware attack, this is realistically what's going to happen, or this is what it's going to look like. You know, this won't be available. These are the applications that will be down. This is the expected time it'll take, or this system has this data. If it is stolen, you know, this is what's going to happen. I don't think that the storage experts are relaying this information up and allowing the decision makers to make decisions about,

hey, is it worth spending a few extra pennies here to provide these data security protections around the data? And they're just saying, well, last year, we spent this much. So we shouldn't spend more than that this year. And that's not really the case. They need to think about, hey, this is a few dollars more in a storage investment that could protect tens of millions or hundreds of millions. The MDM Caesar thing has got to be on the order of millions of dollars a day, right? Yep. And so I don't think people think about that. I think at the low level, they kind of get stuck in kind of just thinking about the smaller picture when they need to think about the bigger picture and think that, hey, data is the lifeline of this organization. If this stuff's not working or it's not available, every minute is going to cost us this much. And realistically, you know,

ransomware attacks, they don't, you don't recover in minutes or hours or days normally, unless you have a technology like BrickStore that can provide you that blocking of attack and stop it when it's small and rapid incident management. Yeah. Yeah. Yeah. God, the whole security space is, is, is exploding really to a large extent. And it's, it's, it's like a battle. It's like a war. I mean, every time there's a, there's a solution out there, there's a, there are ransomware people that go after it and try

to figure out how to, how to get around it. It's, it's a tough game. You guys update your software quite often, Jonathan. So we do, we have the ability. So we, we update the software in general for features and, and to, to make that robust, but then we also have the ability to update our assessors, which are what are looking for the malicious and suspicious behaviors. And that can be updated out of band as well. And so we have the ability where you can update the software over the internet and it'll automatically check for updates and download those or for customers that are disconnected on a dark, isolated network. They can download the updates from our portal and then upload them onto the

brick store. Right, right, right, right. And the challenge is everybody's challenge really is how to, how to, how to maintain a security posture. I mean, historically, the problem with security has been it introduces friction in what we're doing. I've got access controls now that I have to follow. I've got control over whether or not an admin can blanketly back up data or stuff like that. It's a necessary evil, but it's, it's, it's, it's something that, that, that needs to be in place.

With the sophistication of ransomware these days, it's, it's, you don't have a choice anymore. Yeah. It's unfortunate. I mean, it's kind of like the airport, right? It'd be nice if you could just drive up to the metal detector and go through TSA, but it's a necessary evil, right? And if, if you don't have those things in place, people take advantage of your, you know, the friendliness and it'll get exploited. And whether it's exploited by one of your own employees or an external threat actor,

it's going to get exploited. And so it is a necessary evil. And it's our responsibility to kind of deal with that kind of stuff. Just like we deal with change management and approvals for that. You have to do the necessary security things or you open up the whole organization to the threat. And it could cost the organization its entire existence. Oh, yeah. Yeah. I imagine that the MGM Caesar thing is having some adverse effects on the IT team, if not the management team in total. Sure.

I'm sure the whole, you know, the stock market, you know, everyone's going to give them a hard time. And when you see this happen, like a big event like that, what you normally see, and gaming's a regulated industry, right, especially in the United States, you're going to say, what tends to happen is, you know, the regulators expect the companies or organizations, you're going to say what tends to happen is, you know, the regulators expect the companies or organizations, you know, the members to basically kind of self-police themselves and

do the right thing. And if they don't, or when they fail, then that's when you see the regulators come in with strict rules and things like that. And we're seeing that, you know, with the new rules published by the SEC for all the public companies where they're going to have to report breaches within three days. No way. Really? Yeah, that's effective in the new rule that goes into force either in November or December. So think about that. One of the big challenges if you look at all these data breaches is they always start off like we only think a fewer small percentage of records have been exposed to access. Then 30 days later, it's like more.

And then usually at some period later, they say like 100% was gone. And so what that says is that these organizations don't have visibility that they need. They need to have visibility into the data and who's accessing the data in real time and historically. And that's another thing we can provide so that when there is a breach or when there is a suspected threat or maybe an employee that all of a sudden turns up to be not the person you thought they would be,

you can go back and look at their history and see exactly what they accessed and understand what your exposure might be. That way, when you're going to report something like to the SEC, you're armed with information that is meaningful and you can make sound statements that you don't have to retract or recant later. So your logs must be pretty extensive to be able to have that sort of information over a course of, you know, a user's life on a system or something like that. Yeah, they're pretty extensive, but it's pretty efficient too. I think a record is about 12 bytes, so you can do a lot. We can store a lot of data, and nowadays it's almost like people are never deleting data, so we can be very efficient

in the way we store that and retain it. Okay, well, this has been great. Jason, is there any last questions for Jonathan before we close? No, it was fantastic talking with you. And it's always interesting to see what the latest in cyber protection is. Always great spending time with you guys. Yeah. Jonathan, is there anything you'd like to say or listen to the audience before we close? I think if you're going to be at Gartner IOCS in December, we'll be there. We'd love to see you.

Okay. Good news. Well, this has been great, Jonathan. Thanks for being on our show today. And thanks to Racktop Systems for sponsoring this podcast. Thank you. And that's it for now. Bye, Jason.

Bye, Jonathan. Until next time. Next time, we will talk to the most system storage technology person. Any questions you want us to ask, please let us know. And if you enjoy our podcast, tell your friends about it. Please review us on Apple Podcasts, Google Play, and Spotify, as this will help get the word out. Thank you.