The a16z Show - a16z Podcast: Getting Security Right Isn’t as Hard as You Think (But the Effort Never Ends)

Episode Date: April 29, 2015

The paradox of security is we pretty much know what we are supposed to do most of the time -- but we don’t do it. If you examine all the recent high profile attacks, somebody in the organization knew something was wrong before it happened. They just didn’t have the ability to escalate the problem, or the ability to raise a flag that people took seriously. The lack of foundational security hygiene is what makes companies vulnerable to relatively mundane attacks, which are far more likely to hit your company than some sophisticated nation-state mounted attack. “There’s this misconception that we can’t defend against these attacks because we can’t deal with the sophistication of the attackers,” says Tanium CTO Orion Hindawi. “In turns out, we should just be doing the good hygiene we’ve all been trying to do for the last 20 years.” In this segment of the a16z Podcast, Hindawi shares how to get your security hygiene right -- not just from a technical perspective, but from a cultural one as well. The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments and certain publicly traded cryptocurrencies/ digital assets for which the issuer has not provided permission for a16z to disclose publicly) is available at https://a16z.com/investments/. Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information. Stay Updated:Find a16z on YouTube: YouTubeFind a16z on XFind a16z on LinkedInListen to the a16z Show on SpotifyListen to the a16z Show on Apple PodcastsFollow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

The content here is for informational purposes only, should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. For more details, please see A16Z.com slash disclosures. Welcome to the A16Z podcast. I'm Michael Copeland. And I am here at the headquarters of Tainium with Orion Hindawi, CTO. Orienne, thanks for coming. Or actually, I'm visiting you. So thanks for having me. It's a pleasure either way. I just saw literally, I was coming up in the elevator, I just saw that WikiLeaks had posted hundreds of thousands of emails and more data from the Sony hack. It seems to have been a pretty bad year. I mean, it's a tough year if you've been a security person.

And 2013 was certainly a tough year, and a couple of years before that. Is it getting worse? So there are a couple factors there. So the first factor really is that we're getting better at detecting that we've been attacked. And so I think a lot of customers have invested in detections, so that they can see that bad things are happening and I think we're actually

surfacing a lot of stuff that used to happen and we just didn't even know it was happening and we're detecting it faster now and we've got better telemetry on what's happening and I think that's factoring into this I think another thing that we're seeing that definitely is getting worse is that companies are keeping more and more of their data online they've got more and more of this data accessible to the internet because they're using it for customer-facing activity and that opens up surface area vulnerability and I think the attackers are actually getting a lot better. We're definitely seeing the sophistication of the attacks that we're looking at increasing.

And I think the volume of data that they can go after and the accessibility of that data driven by business use, driven by the business that our customers are in and how having that customer data accessible to the internet enables that business is giving them more to attack. So I think of both vectors. So there's more out there. it's more valuable, so I'm a hacker, I'm going to go after it. But there's this tension then between all these systems that we want online, all this data that we want to put on online that, as you say, is part of doing business.

So what are the gaps then, if we're going to live in this world of, you know, everything's connected, I can work from anywhere, I can bring in 30-party vendors, and they can access my system to, what are the gaps then that are need to be filled and, you know, that are making us in some ways more vulnerable? So, you know, the irony of security is we all pretty much know what we're supposed to be doing most of the time. If you're a security expert and you've been doing this for a while, we all know that they're just good hygiene things we've supposed to have done this whole time. So patching your devices, having disc encryption locally on devices that have data at rest, having things like dual factor authentication and things like agents that are on endpoints like antivirus that are working. the fundamental problem that I think we're seeing is that people aren't doing a lot of those things. And I think the more that you integrate third-party vendors, the more that you have data that's present that you can access from internet-facing devices,

the more important it is that this basic hygiene get followed. If you look at the attacks that we've been seeing, you know, there's kind of this thought that these nation states with, you know, thousands of people are attacking every customer. and that may be true in some specific cases, but in many cases when you actually look at the actual tangible attacks that people are seeing, they're exploiting known vulnerabilities, they're exploding customers not putting dual factor

where they thought they would or disencryption where they should have. And those are just block and tackle hygiene issues. They're not actually these super sophisticated, you know, James Bond's style, somebody's parachuting through a skylight minging into your data centers. It is this misconception that we can't defend against these attacks because we can't deal with the sophistication of the attackers. It turns out we should just be doing the good hygiene.

We've all been trying to do for the last, you know, whatever it is, 20 years. And in many cases, our customers are just realizing that they've been failing for 20 years. And now they're actually realizing the frequency that they're being attacked by relatively mundane attackers because they haven't been doing all the things that they thought they should have done this whole time. And they just didn't notice it. So are you saying that the psychology, is today among some folks that, look, we can't win anyway, so why bother? Or is it that, well, it wasn't a problem in the past, so I don't need to sort of check all the boxes and do what I should?

I think it's more of people are realizing that they haven't been doing all the things they've been told to do for so long that they don't believe it's possible to do them. So, I mean, I'll make an analogy, right? I mean, if I told you that every day you had to go an exercise, three hours a day and eat perfectly and you know live an extremely healthy life every day I clearly I do that yeah most people would fail right yeah if you knew that you were going to die this year because you weren't doing that stuff you'd probably make a really good effort most of our customers have gotten to the point where they don't believe it's possible to do all the things they've been told to do so they're resigned

basically to dying every year they're resigned to getting attacked constantly because they don't think it's possible to patch all their devices because they don't think that it's possible for them to get all of the antivirus and hips and disc encryption working the way that they were supposed to password policies kicking off machines off their network that weren't supposed to be there in the first place. I mean these are all the problems our industry has been basically tackling for the last 20 years. And now people have been trying because they don't think it's possible to do those things to find a silver bullet. So, you know, I'm not going to name the names of vendors, but when you start looking at them, you'll start seeing some of these

guys touting that if you install my agent on this end point, everything automatically gets fixed. Right. To extend your exercise analogy, it's like just take this pill, or five minutes a day and like, boom, you're done. Even not that, right? I mean, the idea is just exercise once in your life and then it'll all carry over for the rest of your life. And unfortunately, insecurity, that's never been true. I mean, if you look back at the last 30 years of security, there's been a vendor every year that's come up with a new theory on how if you just do one thing, everything will be fine. And truth of security is it's never been that way and it'll never be that way. You have to do eating healthy and exercising every day if you actually want to keep

secure. And there's no way to be 100% secure. But the truth of the matter is if you look across the 10 biggest attacks this year, all of them tied back to pretty mundane things that the organization knew they were supposed to do that they didn't do. And really our emphasis from a security posture standpoint is, it's great that we're looking for sophisticated insider threat from geniuses. We should be doing that too. But before you get there, or nation state attack prevention,

which is almost impossible, let's just do the basic stuff. If I'm in charge of security or if I'm running a company, period, and those that have kind of made that shift where they're not looking for a magic bullet, but they're doing the sort of good hygiene blocking and tackling. if you can describe that mindset and that sort of environment that allows for that, what does that look like and feel like? Okay. So one of the biggest things that has to happen is the security and operations teams need to actually become friends.

So if you think about what I've been talking about here, a lot of it is detected by security, so flaws in the environment that aren't really up to the compliance standard that the organization is setting. And the operations team is often responsible for fixing it. So we've been talking about patches or antivirus updates or being able to do things like disc encryption. Those have to involve operations. And one of the biggest problems that we see in enterprises that we work in is that those two teams are not 100% sync. The operations team is really worried about some problems. The security team is worried about a completely different set of problems. And until those two teams really get on the same page, it's not going to work.

Because there's going to be a huge gap between what security wants to happen and what operations is actually doing. And so the most successful organizations that we're seeing, and we would encourage all of our customers to move in this direction, you've got security and operations really joined at the hip, both understanding this is an existential threat to their organization if they don't do it well, and really coordinating on finding and then fixing very quickly any gaps that exist in the org. Is that relationship, you know, is operations worried that their ability to function gets hampered by security or is it more that security sort of doesn't know the ins and outs of and vice versa ins and outs of what operations does and you know and and doesn't therefore know how to attend to it?

So I mean there are a few things. So one of them is operations is really responsible for keeping the organization working and the more change you make and the faster you make it, the more likely it is that you're going to break something. And so security always is super urgent when it comes to, you know, we've got a flaw, we think it might be exploitable, we, we absolutely need to fix it. And operations typically is going to look at it as, you know, how do we make sure that we're implementing the change at a rate where we're not dooming the org to having a huge business average because we changed something and it broke something. And so there's a natural tension there. What's really important is that security

actually understand why operations wants to be deliberate. And conversely, operations needs to understand why security is so urgent. And, you know, the reality is you really can do something in an hour across the largest environments in the world, if everybody gets together, you've got the right tools, and you're pushing as hard as possible. And I know that sounds hyperbolic to a lot of people because many people are going to listen to this and say, you know, if I'm running the largest or enterprises in the world, I've never done anything in less than weeks. Right.

And the reality is you can do it in minutes if the tool set is upgraded to allow you to do it, and if everybody understands the urgency and the requirements to make sure that the operational focus of the environment is maintained. Right. And it's urgency not in the sense of like, okay, let's all freak out now. It's urgency like, okay, we have a plan. You know, it's DefCon 5, push the button, let's go.

It's urgency in the sense that if you look at every one of the attacks that we saw, somebody in that org knew something was wrong before it happened. They just didn't have the latitude to escalate it. They didn't have the ability in the organization that affect change. They weren't

actually screaming from the parapets, we need to fix us and having anybody listen. And what you see in the best run environments is that security has a seat at the highest table and they're able to really raise a flag. And as soon as they raise it, people take it very seriously. And they understand the requirements in the organization, not to blow the organization up because we're moving too quickly. So urgency doesn't mean let's run with our hair on fire around and try and fix every issue without thinking about it. Urgency means that we can, can't afford to just forget about these things and bring them up three weeks later and then probably forget about them then and bring them up three weeks later, which in all honesty,

a lot of security organizations have vulnerabilities they detected years ago that are still not being fixed. Right. If that's the level of urgency in the organization to respond to security need, there's a very high likelihood that they're being attacked successfully. Right. And it's, you know, shame on them. And it gets back to this notion of like you need an environment where, again, people understand both sides. Like, I can imagine that you don't want to raise the alarm if that's going to, you

are worried that it's going to slow down the business and or there's been this kind of, you know, message from the top that, look, what we do is build the business and we grow, grow, grow, and we go fast, fast, fast. It's hard to put on the brakes if you see something in that sort of environment. So let me just say, I mean, you were asking about the biggest change in the last year. The biggest change we're seeing is that there's board level acknowledgement that this is an existential threat to the business. So it used to be that security was annoying, and often it was kind of, we'll accept this risk.

The likelihood that it's going to actually cause massive damage is pretty low. If it is, we can probably contain it. We probably don't have to disclose it. There were a lot of these kind of rationalizations around security, and I think the watershed moment was the target breach where the CEO got fired, the board got sued, the whole stack in IT got replaced, and potentially billions of dollars of data.

damage were caused. And when you take a step back and think about that, now I was talking to a CEO recently and he told me, and this is now a quote I've repeated a number of times, but that, you know, he's got three existential threats to his business, nuclear weapons, meteors, and cybersecurity, right? Right. And he never would have said that five years ago. And he admits that. He says, you know, five years ago I was worried about regulation and my China strategy and my competition. And now I'm worried about three things, only one of which I actually. have any control over, right? And so that change drives behavior across the organization. You look at a lot of these big companies, they're spending literally 10 times more on security than they

were five years ago. And the reason is there's a realization at the top level of the organization that we can't kick the can down the road anymore. And that having operations come back and say, well, this is annoying, is not a good enough reason not to do it, which five years ago wasn't true. You talk to a lot of large companies who are grappling with this. How is the conversation, you talk about how it happens at the board level now and at the highest levels of the company. If I'm a company that hasn't been hacked, is the conversation somewhat different than a company that just has gone through a breach? Yeah. So there's this concept in our industry that it's good for security companies when their customers get breached and it's actually not true. And the reason

it's not true is that often... what you see in companies that have been attacked is a very neurotic behavior pattern for three or four months after the attack, where they will pay anything for somebody to walk in and tell them that everything's fine, which is actually not our business, right? I mean, we don't really want to come in and tell you everything's fine, or that we'll handle it. I mean, it's really systemic change that needs to happen in the org for them to be fine, and we can't affect that change they have to. But you end up with people who are getting fired, people who are constantly in meetings

trying to defend themselves instead of actually make change. And I'm just saying, you know, generalizing across the hundreds of customers that we've seen, but it's actually not a very fertile environment for good decision-making. And so, you know, we will often get business out of those situations, but it's not the kind of business that I actually prefer. My preference is a deliberate decision by the board or the CEO or the management chain in IT that they have to really reprioritize around security, typically because they saw their peer get attacked.

Right. And then they want to actually build a strategy. So there's no real strategic thinking that we typically see in the two months after an attack. Typically we see hair-on-fire behavior, right? People are getting fired. You want to cover your job. Yeah. And those are not the kind of scenarios where we typically see thoughtful work.

Now, I will say this. We have some customers. I think Target is a great example of one of them that are extremely, thoughtful and were thoughtful in the aftermath of the breach. They spent a lot of time building a real lasting structure, and I think they've done one of the best jobs we've seen in building a security organization. They should be extremely proud, but unfortunately, they're the exception, not the rule in post-breach situations. And how has the culture sort of shifted at target? And clearly, you go through something like this. Everybody in the organization knows what happened

and the consequences, but then there's probably a tendency to sort of try and get past it and get on with business as usual. So not at Target. What we're seeing there is actually a continual realization that security is a permanent thing they need to be really careful with. So, I mean, that org suffered tremendously during that breach. And I think, you know, there's more public on this than I can repeat here that, you know, gives context. but they hired a great CISO. He hired a great set of lieutenants, all new into the org. And what he did that I thought was really nice

was he looked at the premier security executives from across the community. He hired a bunch of people from the Mandiant Fire Eye Crowd. He hired a bunch of people from other places like General Electric that were super competent people. And he built an org from the ground up, and he had the latitude to do that because the organization at the top level of Target, the CEO on the board, mandated that they do a world-class job.

And, you know, when you look at some of the people he hired, especially some of the Mandian people, they're exceptional people. And I think he's built a kernel in that organization that's going to insist on an excellent org. And that's a sea change from where they were two years ago. Let's say I'm not Target. I don't have thousands of employees and, you know, thousands of stores for that matter. how then on the spectrum do I want to view security as a smaller company but then also take us up to a big company and I also want to circle back on your view of this personally like how it seems so sort of forbidding but maybe it shouldn't be so I'll say kind of a general thing first and then I'll go through

the spectrum. security is scary because it can cause massive damage. The same way that a lot of things in our lives are scary. Cars are scary because people die in them every day. Most people aren't scared of cars. They just realize they have to drive carefully, right? Security should be treated the same way. You should just be prudently cautious about the fact that if you have vulnerabilities,

you should be fixing them. If you have users who are being added, you should make sure that there is multifactor. enabled on them. They're just kind of these good habits that everybody knows they're supposed to follow. And a lot of organizations look like they're driving 120 miles an hour drunk, right? They're not doing any of the things that they should be doing. And as a result of the fact that they're not doing those things, they are really prone to accidents, right? There are rules in security and in operations in general. You should be going and monitoring your

network traffic in specific ways. You should be implementing firewall policies. You should be implementing firewall policies that make sense. You should be patching your assets. You should be figuring out what data is being exfiltrated from the end point so that you can actually see it. You should see where your critical data is and data leak protected. There are things you should be doing. And that's exactly analogous to driving 65 miles an hour on the freeway, sober, and paying attention to the people around you. Right. So when I hear people who are kind of terrified generally about security and feel like it's an out-of-control situation, those tend to be the people where, from the analogy, they're not driving anywhere near the speed limit and they don't seem to care

and they just want to get wherever they're trying to get as quickly as possible and they're getting into accidents every day. And there's a direct correlation between their behavior and the result. So my assertion would be there's good hygiene that you should be practicing in security and operations. Everyone knows what it is. Let's just do it. It turns out that if you do it, you feel a lot better and the results are a lot better. It's exactly like exercise or like driving safely. It's, you know, Let's take it to really basic things that everybody knows they're supposed to be doing. So that's the first thing I'd say. The second thing I'd say is Tainium is focused primarily on global 2,000 companies for a reason,

which is there is not the capacity in small companies to do the same work that our biggest customers are doing. It's not that they shouldn't be doing it. It's that they don't have security personnel on staff who've been through years of training and have years of experience in ferreting out advanced threat. They may be attacked. In some cases, we're seeing stores where they've got 1,000 employees in 10 stores and they're being attacked. And the reason is they've got credit card data and credit card data is valuable. I don't know that they have the wherewithal or that they should be trying to build the expertise to deal with the same attacks that a target or a Walmart are trying to deal with.

Now that said, you know, again, there are some good hygiene things they can do. There are endpoint solutions that are designed to be heuristically kind of preventative. So you think about antivirus is kind of the most simple. one. And you look at things like host IPS or some of the other solutions that are being released on the endpoint that are really heuristic. You set them and forget them if you want to think about it that way. You should probably deploy some of those, but I'll be honest, that's not our area of expertise. Where we start playing is when we've got a 5 or 10,000 seat org. They've got enough data at this point where it potentially could be a huge disclosure issue if they actually

get attacked. And they typically have a security set of personnel in the environment. because they can't afford not to it, right? I mean, it's, again, risk-reward, risk-benefit, if you want to think about it that way. In the end of the day, if they don't have these people, then they stand to have huge risk, and so they'll expend the cost to actually build a practice within the org that allows them to kind of understand their security posture.

When you get to that point, there are a few hundred things you should just be doing. And, you know, this is kind of the theme of the discussion, right? is that we should start making sure that all those 200 things are done. So password policies, domain presence, being able to have good ideas of what's connected to the network and being able to see whether devices are unmanaged and bring them under management, making sure that managed devices are being patched correctly

and that the applications that are on them are actually the intended applications, that they're being upgraded appropriately. You know, just kind of block and tackle IT. And assuming that that's done, then you start getting to the next level. So we have many of our customers who are starting to do outlier analysis, heuristic analysis to determine whether behavior patterns are changing,

looking at things like insider threat. I'll say, though, I mean, when we walk into companies, we've now deployed this thing in hundreds of companies, and we've seen a cross-section of the Global 2000 that, you know, it's a pretty interesting cross-section. I think maybe one or two percent of the companies that we've walked into really should have started talking about insider threat when we got there. Right.

The other 98-99% They weren't through the Just block and tackle stuff And it's so fun to talk about insider Threatenation States and cloak and dagger It's just a waste of company resources Unless you've got the framework built correctly To even approach that kind of attack

If you haven't dealt with your patches You should be worried about kids that have access to Google Not nation states that want to attack you Right that's kind of the point I'm making is that you know There are thousands of people that are professional attackers that are nation-state level or criminal attackers who can get into most companies. There are millions of kids with Google who can figure out how to exploit known vulnerabilities that aren't patched. Right. In some ways, the nation-state is the meteor that hits you, not the sort of security breach that happens to a lot of folks.

I mean, I think serious people in security have realized a long time ago that give an infinite time and infinite money a nation-state, will come at you and will succeed. The reality of the situation is very few companies, very few are equipped to actually deal with that threat in any way. I don't even want to use the word prevent because I don't think it's possible, but even deal with it. I think you look at our intelligence community, they're fighting a war with other intelligence communities and nation state actors outside. They are probably more equipped, but the truth of the matter is this is a bloody conflict. It's not a clean. We keep everybody out. Everything's perfect. We go to sleep at night and everyone feels good, even for them. Well, and so as a company, I go through the 200 things

that I need to do. I might even look at insider threat sort of risk. And then what? Then I just need to keep it up. I just need to keep the sort of regime going and stay fit and stay sober. So here's what I would say. There's an almost infinite amount of operating. that you can do in security. When you've got hundreds of thousands of assets, everything that could be going wrong is going wrong somewhere right now, right? You'll never get perfect. And the goal is to reduce the surface area as much as possible

by tamping down the obvious stuff, the most obvious, and then moving up to the slightly more obvious, or less obvious, and then moving up to slightly less obvious and so far, until you get to really esoteric kind of vulnerability. Most of our customers are at the first level of that when we walk in, in our goal is to ratchet them up a couple levels of less obvious vulnerability and give them the tools to keep going but the reality is given the flux of environments given the virtualization and cloud computing that's happening

given the mobility and BYOD and all the other things that are happening the you know perimeter being dissolved in many companies in reality even if they don't want to admit it it's a never-ending process and unfortunately it's two steps forward one step back in many companies because as soon as you you've stepped forward two steps as a security org, somebody from, you know, one of your business units comes back in and has an awful idea that they want to do something. And as soon as you hear it, you choke a little bit because you realize that this is going to obviate a lot of what you just did and you're going to have to figure out how to deal with it. And so the other point that I

would make is we can't build the security house for our customers. What we can do is give them really effective tools that they can use to build the house. And when somebody wants another bedroom added or a wall knock down to make that as easy as possible, and to confirm that you did it right. Right? I mean, you know, to take the house analogy a little further, you know, many of our customers are constantly knocking down walls, and they don't even know which walls are load-bearing,

and then the house crumbles, right? You need to actually have a good view of what you have. You need to understand how it works. And again, you know, I've said this many times before, but many of our customers don't even know how many computers they have. So when you start with that level of lack of knowledge, you can't knock down walls in the house. You can't make any change and have any confidence it's going to work because you don't even know what you have.

You don't know what it's supposed to be doing. Once you know that, then you can start planning. Well, what's the deficiency between what I have and where I want to be? Somebody comes in and asks me for a change, how's that going to affect what I have today? How do I want to pivot so that I can minimize the security impact of that change or actually maybe allow that change to drive more security posture for the org? But the first step is just figuring out how many bedrooms are there in the house. Where does the house even sit?

What does the foundation look like? And many of our customers, before we walk in there, don't have any idea. They don't know how many subnets they have. They don't know how many computers they have. They don't know what's running on those computers. They don't know where their data is. Security is impossible if you don't know those things. Right.

It's not hard. It's impossible. So we would assert that you have to solve those problems first. Get the hygiene in place. Then let's go worry about everything else. Orion, thanks so much for the conversation. You haven't scared me. You've actually made it seem like this is something that's doable.

It's absolutely doable. We're seeing our customers make progress on this constantly. You just need good tools and you need to have the discipline to use them. It's that simple. And I do think people are getting better at this. I don't think that this is hopeless in any way. I think, you know, kind of the fear-mongering aspect that people are so exhausted by insecurity is an admission that if you don't do this stuff first, you don't know how to do it.

Right. That doesn't mean that it's hopeless. That means you just need to do this stuff first, and then you actually have some hope. So I think this is actually a very hopeful message, and I think people should see it that way. Well, it's work, and so I guess we have to get to it. Right on. Thank you. Yep.