a16z Podcast - Security When the Workforce Goes Remote

Episode Date: March 27, 2020

We are in the midst of a rapid and unprecedented shift to remote work. What does it mean for security when the airgap between work and life is gone? How prepared are organizations? And what should security professionals as well as individual workers be doing to protect themselves and their companies?In this podcast, a16z security expert Joel de la Garza breaks down the current risks and how to defend against them. But beyond just immediate security needs, he explains what bigger transformations may be happening, most notably a shift from the traditional hub-and-spoke, point to point, security architectures to a more distributed approach to workloads as well as trust.

Hi, and welcome to the A16Z podcast Goes Remote. I'm Doss, and in this podcast, I call A16Z security expert Joel Delagarza to chat about what the rapid, widespread move to remote work means for security. With so many people going remote the same way that we are, what's top of mind for you as a security expert? There is a concept in information security, which is the belief in defense in depth. And that means that you don't rely on any one thing to protect you. You have a series of things. that you use and you stack them on top of each other. And you use those series of things to offer multiple layers of protection. You don't just put a moat around the castle. You also put walls and you have archers and you have hot oil ready to pour on people that try to storm it. And so in security, we have those same sorts of controls. The challenge for security teams is that a lot of those controls for a lot of companies only live in their office and only live in

their corporate network. And so when users take their machines home with them, or they're remotely accessing, they don't necessarily have the same controls in the office as they do at home. And if you look at some of the large breaches over the last, let's say, five years, you'd see that there are a number of instances where remote employees using a home computer that's perhaps shared with someone in the house that doesn't have protections on it is used to access internal corporate information by an attacker that's hacked it. Are the things that we're dealing with new things or just things that were underway happening a lot faster. We've had multiple scenarios in the corporate and enterprise world where we've had to

make employees work from home and work remotely. You know, the first real encounter in at least my adult life with this sort of a scenario was obviously 9-11 when we had fundamentally a city that became unavailable in the workforce there being mostly unavailable or having to move to disaster recovery sites. And I think 9-11 really taught a lot of large corporations about the importance of building really resilient business continuity programs. The actual new thing about this is just the scale, is just the entirety of a workforce for a company being forced to work remote, as well as their suppliers, as well as their customers. We had the advent of things like SaaS and Salesforce and box and all these tools that were basically derived so that people could

access their work materials anywhere. And it sort of became expected that some percentage usually sales people because they're in the field, but some percentage of your workforce would be remote. And so we've been building infrastructure to support that workforce for some time now. This is less of like, oh, it's a new way to work and we have to change everything. This is more like we have to re-engineer everything to handle the capacity and just the sheer number. How are the best security teams you know properly preparing their organizations with this really rapid shift to remote work?

You know, I think that the right way to think about it is to just build a matrix in your mind that sort of enumerates all the different security controls you have available to you in the workplace, in the office, and have some understanding of how they translate to the different scenarios all of your employees will find themselves in now. And so I think there's two things that the really good security teams are fundamentally doing. The first is getting their people stood up online outside of the office because security teams don't necessarily always have. great disaster recovery in business continuity plans. And then second, making sure that what they're doing is actually safe and secure. If you were in an organization right now and say you were going from 20% to now 90% of your workforce is remote, break down for me very specifically how you would do a risk assessment. Over the last couple years, most things have left the building. And so most services are provided by third parties. Most of the infrastructure that you run isn't running on your premise. And so for the last three or four years, most CSOs, or chief information security

officer, has spent a tremendous amount of time thinking about their third party risk. Who are their vendors? Who are their counterparties? Who are the people that they transact with? And you have to think about them, not just from a security perspective, because that's a little bit narrow in terms of impact of the business, but you need to be more comprehensive. And think of terms of like confidentiality. So is shifting all of your voice traffic to this third party, does that provide you with the confidentiality you need to run your business? While it may be okay to have a sales call with a customer where you don't discuss anything confidential over a video conferencing system, now you're having your board meetings over this video conferencing system, does it meet the requirements that you have? And then you have to think in terms of integrity, the systems that you're relying on, now that you've moved everybody onto them, have the controls in place. to ensure the integrity of the operations of your business.

Are they going to lose your data? Is there going to be some sort of a disruption to the quality of the output? Are the systems of record truly capable of being systems of record? And then finally, you have to think in terms of availability. Not just you as a company are moving your entire workforce to this service provider.

The entire planet is. Will the service provider be up and running in the face of this kind of demand? Or will they just fall over because of the excess utilization. I like the way that you broke that down. So it sounded like the first bucket there was really around confidentiality and what transactions were happening in person,

providing a measure of security, now happening virtually. So let's focus in on that for a second. How would you go about assessing that? It really depends on the vertical. And it depends on the industry. There's a very, very rich tapestry of requirements and regulations that you have to really understand. and it's very specific to the business that you're in, specifically if you're regulated,

and you have to make sure that the tools that you're using can support those industry-specific regulations. If you are, for example, in the healthcare industry, and let's say you're a hospital network, and hospitals right now are rushing to provide telemedicine and to remotely treat potentially sick people. The issue with that is that there are these regulations called HIPAA and high-tech that mean that you actually have to work to maintain the confidentiality of your patient's information. So then I guess looking at the second bucket that you talked about, which was really selecting these new tools and introducing these new third-party vendors that you maybe weren't using before. So for instance, you and I are using a totally new tool for A16Z that

we rolled out as soon as we went remote so that we could keep running our podcast. How are you or security professionals thinking about these third-party tools and how do you go about assessing them? Well, it's always about the data. For example, we're recording a podcast, this is public information, eventually it's going to be released. And so the sensitivity of our discussion that we're recording right now is slow. It's fundamentally public data. Whereas if we were talking about a portfolio company, this might not be an appropriate tool because it might not adequately protect those discussions.

And so we really have to understand first the sensitivity of the data and then match that data sensitivity to the security features and capabilities of the tool. Generally, marketing teams are kind of free to experiment with tools that are maybe not industrial grade security. But the moment that you start talking about transferring customer records or transferring personal information of your customers or any intellectual property, then you really need to understand the tools. And a very quick adoption and migration path could potentially get you into a not so great place. It's interesting you mentioned quick adoption because that is absolutely what we're seeing right when you suddenly have, in our case, all of A16Z going remote, we suddenly needed all these new

communication tools that we didn't use before. So we are rolling them out relatively quickly. How are IT and security teams keeping up with the fact that people are rapidly adapting to this? Things are changing daily. How do they balance that with security? At A16Z, we've been fortunate in that we've probably spent the last two years really focusing on eliminating any kind of custom solutions, not having servers under people's desks, not having servers at all, focusing on using cloud infrastructure and SaaS. And so when this event happened and we had to pivot, credit to our IT team, they did some wonderful work, but we were really well positioned. There wasn't a whole lot of stuff other than adding a few new services that were disruptive.

I think the way that the modern enterprise has built their data stores is somewhat similar, so that a lot of the data that a company has that could very easily, flow out of the organization are generally pretty well controlled. Often we're used to these large enterprise rollouts of new tools. They take a long time. But now you have a workforce going remote and you maybe need to roll tools out faster. What steps are you seeing people cut or needing to add to get the tools out and into the hands of workers in order to do virtual work? Usually one of the longest pulls on any of these kinds of tool deployments is the legal and contract negotiations. It's the kind of thing where

the length of your proof of concept is probably half the length of the debate you're going to have with the vendor about limits of liability. People complain about IT, but if you really want to prolong something, bring a couple lawyers. And especially when you have to have IT people, technical people work with lawyers, it compounds it. So I think where I've seen things getting quicker is just on the procurement side, on the contracting side. We've gone through a three-year process of large enterprises telling employees don't use your credit card to buy a SaaS service. That window seems to have opened up a little bit.

And so you're seeing people paying for things with personal or corporate cards to get services deployed and unrolled. And I think IT and legal, they're going to be flexible. They're going to keep the business moving. There's probably going to be a lot of contract review and a lot of teeth gnashing over the next couple months as they figure out what they've allowed into the enterprise. What in your mind works and doesn't work to be communicating out to the organization at this time and what would you be reiterating to individual workers?

The user tends to be the weakest link in any security system. And so there is this desire to blame. And then the products that grow out of the desire to blame users tend to be of the variety that look to shame users into behaving better. So building tests that try to get users to fail and then highlight. highlighting their failure. And we've seen tools that take that approach. They're really good at getting the level of

compliance up, but only to a certain point. I think the real key is going to be figuring out how to decatize employees and users, how to make them feel part of this, instead of smacking them on the hand for making mistakes. And then that's really hard for security people because we

do tend to be a bit pessimistic. But building systems that reward good behavior, I think will go a lot further than the desire to name and shame. From a most important tips perspective, I think for me it's always two-factor authentication. At its most simplest level, two-factor authentication is the way you log into a system using two factors or two things. And from a security perspective, you want one of those things to be something you know, like a password. I've got a password, and that's the thing I know.

And then the other one of those things to be something you have, like a hardware security key. And so it becomes very difficult for an attacker to get access to your system because not only do they have to have your password, they also have to have access to your key. And so it really frustrates what is ultimately the single largest source of hacking in the world, which is stealing someone's username and password. In general, while using a phone is better than just using a username and password, from our perspective, it's not as strong as using a dedicated piece of hardware to protect your login accounts. So if that text message is that gives you the code on your phone, probably not as good as some sort of hardware key

you're plugging into your computer when it comes to two-factor. Correct. And for systems that you care about, I mean, you should really use a hardware security key. And if you're at home and you're not using strong two-factor on your corporate resources or even on your personal laptop, then certainly make sure that you enable that. I also think at home, if you're not using a corporate issued laptop or workstation and you're using your own equipment to access your workplace, double click on the security of your own machines. You make sure that they've got usernames and passwords, that you're running some kind of antivirus software, that you're patching your systems. Ideally, you're not sharing

computers. So you've mentioned business continuity planning a couple of times. Explain to me kind of what that concept means to a security professional. It's kind of the job of a security professional and more broadly risk professionals in an organization to sit around thinking about what's the worst possible thing that could happen to the business? And so you come out with this list of things that could potentially disrupt your business. Now, they may be hurricanes, they could be earthquakes, it could be a hacker attack, it could be a breach, it could be ransomware, it could be a nation state attack, could be war, whatever the case may be. You estimate their risk to the business, like if this happened, how big of an impact would it have? What's the

probability of a global pandemic happening, for example? or an event that forces all of your employees to work from home. And a business continuity plan is developed to help manage those risks so that you can continue to run your business through any sort of adverse changes. It's not dissimilar from what a CFO or a financial planner would do where they try to figure out their risks from a credit perspective, like our credit market's going to shut. Do we have market risk?

Is our stock price going to fall? Which industries and orgs are having a harder time with that business continuity and maintaining going remote, and why? What are the unique challenges if you start to break it down by industry? I think if you break it down by industry, you'd see that the businesses that are having the biggest challenges are the ones that have never had a significant disruption. Whereas if you look at banks,

who were primarily the ones impacted by 9-11, they've been able to fairly seamlessly transition to remote work. They've been able to take up different locations and implement their pandemic response plan. There haven't been any disruption. to the financial system. We've seen people doing panic ATM withdrawals and the ATM and banking infrastructure doing just fine with it. And if you look at Deutsche Bank on 9-11, Deutsche Bank invested a bunch of money in business continuity. They could seat their employees on the other side of the river.

They had backups. They were running off-site. In response to that catastrophe happening, they were able to quickly resume business, settle their trades, not suffer material financial impacts. I'm sure in every meeting leading up to the event, there was probably some someone saying we should cut that budget. But lo and behold, you hold fast and it turns out to be an investment that's worthwhile. I also feel like there's certain industries where either regulation or the nature of the critical infrastructure, say a power utility. They have some unique challenges. I'm curious if that's something that you're seeing or hearing about. I think that power utilities and a lot of these critical infrastructure components, they sort of

have their zombie apocalypse plan. They plan for that. And I generally have faith that they're doing it Well, I think the one industry, the one segment that's going to be really impacted, and we're seeing that, is actually pharma and health care. I think that there are just major capacity constraints in a lot of countries that just won't be able to handle a major flood of inbound requests for care. And ultimately, the reason why we are all working from home is to protect our health care system. Whether we're conscious of it or not, we are all engaged in a business continuity plan for the public health system right now.

I mean, that is what working from home is doing so that we can keep capacity available to treat and care for people. All right, so I want to shift now and talk a little bit, not just about the security practices, but what this means for the architectures that organizations have,

because as the workforce goes more distributed, it does seem like there might be a need to re-architect the way that we do things. What are your thoughts on how this might impact organizational architectures? I think the cleanest example of where there needs to be a massive re-architecture is when it comes to, like, traditional VPN or virtual private networking technology. VPNs are mostly based on IPSEC, which is an internet security protocol that was developed many years ago. And these protocols and these infrastructures were designed to be point to point.

You would have many, many points around a central hub that would aggregate all of that information and then send it to other central hub. And so in that architecture, if one point on the hub wants to talk to another point, it has to go through a central point. When you move your entire workforce onto that kind of hub-and-spoke point-to-point infrastructure, you get traffic jams. And security systems tend to fail closed. So if a VPN or a firewall starts falling over, they tend to shut down and stop all traffic. And so it's really clear that we have to get away from the centralized, the ring of trust model. And we've got to go more towards a web of trust. You're seeing this with a lot of the new security technology that's coming out,

where they're creating these more distributed trust environments. Cryptocurrencies and blockchain are very much about that distributed trust model. Is it too broad of a generalization to say that the ability for us to scale and to not be real-time stress testing our systems is really directly related to how fast we can re-architect to distributed trust? The point-to-point architecture scales fairly linearly. And so for every increase in capacity or increase in utilization, you have to add a fixed amount of capacity. And it's just not a great way to scale from an infrastructure perspective.

And so we have to get to a way where we can use capacity that's more at the edge and get away from the centralized infrastructure. You talked about this process of re-architecting. And I've also heard about this concept of shifting to zero trust. Is that the same shift, or are those things different? They kind of cohabitate the same space, and I'd say there's a lot of overlap. But zero trust is, it's an idea that was kicked off, I think, by a forester researcher in the late 90s, and the idea was that you have to eliminate transitive trust. Transitive trust is basically the principle that if I trust you and you trust Bob, then I trust Bob.

And as you could imagine, that is what attackers would use to exfiltrate data, to get access to intellectual property to do generally bad things. Transitive trust is a very dangerous thing. And I guess the layman's way to say that is, in the old world, if you went to the office and you plugged into the corporate network, on your corporate network, you had access to a bunch of systems. And a lot of that data didn't have passwords or logins or encryption because it was on the corporate network and the corporate network was considered safe. The moment that you got access to the corporate network, if you were an attacker, you had access to all the data. And so zero trust is about creating a distributed trust environment.

We're taking away the castle and moat and every person's home is becoming a castle to reuse that phrase. With the changes that you see underway, with this shift away from a hub and spoke, how would you advise startups to start thinking about security in their products? I think that you're going to see a lot of companies that historically wouldn't use bleeding-edge technology, actually moving towards adopting a lot of bleeding-edge technology just because of the disruption. And I think it's a really wonderful opportunity for entrepreneurs that are making enterprise tech right now. I think this is their time to really get significant adoption from customers that in the old days would have wanted to see something

on-prem, but now you can't get access to your premises. So you've got to try something new. Generally, we tell our startups, obviously, security is important, but as you get bigger and larger and later in your fundraising, it becomes more and more important. And then finally, when you IPO, there are specific public company security requirements that you have to meet before you even get to go public. So it is a blocker at that level. I think the focus on security is kind of shifting. I think it's going to come a lot earlier now. Typically, you'd see series B companies, sometimes series A companies focusing on security. I think it's going to be like a seed stage thing.

So as we wrap up, what here is a passing challenge security teams have to meet? And what is just a longer term shift in how we think about security? What's our new world order? I think the growing pains are a passing challenge. I think a lot of the large cloud providers and service providers are going to add capacity. And to be quite honest, a lot of the services I'm using right now are working fine. So I'm not super concerned about the capacity. I think the longer-term change is just going to be more about keeping the security mentality.

I think a lot of this ultimately comes down to users. And in a workplace where we see each other every day, you still had people falling for scams where a co-workers send you a request for money from a sketchy Gmail account and you send the money. So I think that when you put more of a social isolation in there, I think the risk of targeting users going for social engineering, to defraud people will potentially become more successful. And so I think the real focus for these organizations is finding ways to keep employees who are at home in their pajamas still thinking like

foot soldiers in the battle to protect their company and their data. That's going to be a real challenge. And I think training is always proven to be one of the best returns on investment. That is a terrific note to end on. Joel, thank you so much for joining. Thank you. My pleasure.