Risky Business - Special Edition: Chris Krebs, Alex Stamos and Patrick Gray

Episode Date: April 24, 2024

In this special edition of the Risky Business podcast Patrick Gray chats with former Facebook CSO Alex Stamos and founding CISA director Chris Krebs about sovereignty an...d technology. China and Russia are doing their level best to yeet American tech from their supply chains – hardware, software and cloud services. They’ll be rebuilding these supply chains – for government systems, at least – from components that they have complete visibility into, and control over. Meanwhile, America’s government faces different supply chain challenges. It has a supply chain that won’t be weaponised against it by its adversaries, but it lacks the same sort of visibility and control that its adversaries will eventually achieve over their supply chains. So where does this leave the west? Where does it leave China and Russia?

Transcript
Discussion (0)
Starting point is 00:00:00 Hey everyone and welcome to this special podcast here at Risky Business. It's a new series we're going to be doing monthly for a while. It's a regular podcast with me, Patrick Gray and Sentinel One's Chief Trust Officer, Alex Stamos and Chris Krebs, who heads intelligence and public policy at SentinelOne. As a lot of you would know, Chris was the founding director of CISA and Alex was the CISO for Yahoo and Facebook, among others, during his career. And the two of them wound up forming a business called the Krebs Stamos Group, which was acquired by SentinelOne last year and so SentinelOne and us we got together and decided to partner on this podcast series and it doesn't have a name yet but who cares with a
Starting point is 00:00:52 lineup like this so yeah a huge thanks to SentinelOne for partnering with us on this we are absolutely stoked to be doing it Chris and Alex have both made regular appearances on the show over the years so it's great for us to actually finally be doing a podcast together. So this is episode one of a yet-to-be-named podcast series, and in it, we're tackling a big security theme, supply chain trust and how it manifests as an issue of sovereignty. China and Russia are doing their level best to yeet American tech from their supply chains, hardware, software, cloud services, you name it, and they'll be rebuilding these supply chains, at least for government and military systems, from components that they have complete
Starting point is 00:01:36 visibility into and control over. Meanwhile, America's government faces different supply chain challenges. It has a supply chain that won't be weaponized against it by adversaries because it is mostly American and the other bits come from allied countries. But it lacks the same sort of visibility and control that its adversaries will eventually achieve over their supply chain. So where does that leave the West? Where does it leave China and Russia? Alex Stamos and Chris Krebs joined me to chew it over, and we'll start with Chris Krebs here responding to my first question. Does he think it's true to say that governments on both sides of the liberal and illiberal divide are starting to have a bit
Starting point is 00:02:15 of a freak out about supply chain security and assurance? Here's what he had to say. I think so, but it probably also requires a little bit of further delineation within supply chain attack of what we're talking about, because XZ and the Microsoft compromises both by the Chinese and the Russians are, I think, a different style of attack. Maybe more the SVR specifically, where it's, look, they got into Microsoft and they stole secrets. But see, this is exactly what I mean, right? Because that wasn't really what I would call a supply chain attack.
Starting point is 00:02:54 Correct. But this is definitely a supply chain security concern, right? And we kind of need to bring those things together, don't we? Yeah, no, I think that's right. And this kind of gets to something that that going back to when I was in government and talking to the intelligence community and kind of based on what their reporting was, you know, they, the bad guys have been watching this shift in the digital transformation over the last several years.
Starting point is 00:03:18 They've, they've, the Intel community has been watching them, particularly with COVID in this massive rapid shift to the cloud. And they know that the Chinese, the Russians and others are watching everyone go to the cloud and they're kind of playing this Willie Sutton moment of, you know, Willie, why do you rob banks? Because that's where the money is. Same thing here. Why are they going after Microsoft, GCP, AWS? Because that's where the data is and that's where the access is. So it's almost like a jump point to something much, much greater with a smaller set of targets. Plus with the legal structures that are around cloud vice on-prem, the NSA can't go grab an instance of Azure, put it on the bench and beat the hell out of it
Starting point is 00:04:05 and provide red teaming support to Microsoft. It just doesn't work that way. Exchange, all day long. It's just a different set. So the advantages absolutely tilt towards the adversary in this new modern landscape. Alex. Yeah, I agree.
Starting point is 00:04:24 I mean, it is where the data is. And especially when you talk about Microsoft, these are two totally different hacks, right? Microsoft is, if you can get a foothold in Azure AD, then pretty much the entire Fortune 500 is your playground, right? Like, as I think you discussed on the pod already, they have effectively a monopoly in a number of cloud services, especially enterprise identity management, partially because they are the only cloud identity provider that can really work side by side with on-prem AD. The vast majority of companies we work with and that we do response at are running in an Azure AD hybrid mode where they have both on-prem and in the cloud.
Starting point is 00:05:03 And the downside of that is instead of that being more secure, you end up with the, not the intersection, but the union of all of the vulnerabilities in on-prem and cloud products. Through their badness combined. Yes, exactly. Right. You see all the fists coming together of like, or this should be the predator handshake of vulnerabilities.
Starting point is 00:05:24 And that you could do Mimikatz and password sprays against unsecurable endpoints in the cloud. And the bad guys have figured that out for a while, right? Like when we look at ransomware activity, a huge chunk of it is attacking the specifics of how a hybrid Microsoft architecture works. But your standard Russian ransomware actor does not have the capability of warming their way into Microsoft corporate,
Starting point is 00:05:49 and obviously the SVR does. And the other thing this demonstrates is just the incredible complexity of Microsoft's cloud products, especially Entra slash Azure AD, in that from what we know, which we can talk about this, we don't know that much, unfortunately, because Microsoft's not being very forthcoming. But from what we know, it looks like this was based upon misconfigurations that a number of people can make. And that, you know, Entra is so incredibly complicated that even Microsoft themselves don't really understand how to secure it. And so if you're a bad guy and you understand that,
Starting point is 00:06:18 and that you spend your entire time thinking about these products from an offensive perspective, that puts you at a huge advantage, even over who should be the best prepared defenders. Now, I mean, I accept everything that you both are saying about cloud. I think one little wrinkle in that is that on-prem on its own, you know, presents its own supply chain issues. And we've been talking a bit on the show about how China is moving away from Intel and AMD you know they're they're moving away for government systems at least right and they're moving away from American-made technology like Windows and whatever and they're going to spin up their own Linux distros presumably
Starting point is 00:06:54 which is incredible because like as soon as they announce that we see this this XZ thing so so I I guess what I'm saying is that I think there are broader issues at play here when it comes to platform trust, whether that's on-prem, whether that's cloud. And I feel like there is this, I mean, we've been talking about balkanization, right? This concept of balkanization in technology for 20 years, right? But it feels like we've hit a tipping point with that where people are saying, oh, okay, this is actually something we need to take seriously. So on the Chinese and Russian side, it's like, well, we can't use any of this Western tech anymore because we can't trust it. And on the American side, it's like,
Starting point is 00:07:36 well, we've got supply chain issues here because we don't have any control over companies like Microsoft and how they choose to allocate their funds. Do you see what I'm saying? Like, it feels like we're hitting some sort of sword. Yeah. Well, you would hope, I mean, I am hopeful with the CSRB report that first off, this is what the CSRB should exist for, right? Is exactly for this kind of report. It finally gave us the details we needed for the rest of us in the industry who are being targeted by the exact same actors to try to defend ourselves. But it also is as aggressive of a bully pulpit
Starting point is 00:08:11 that currently exists in the government. Since there are no real good alternatives, I think, if you're the Department of the Interior or Commerce and you want your mail hosted, on-premise is still a disaster. Like you said, I'm not telling people to get out of the cloud. I think this is just unfortunate that, especially in the very, very high end, it is no longer an automatic decision. It used to be just super simple to tell people,
Starting point is 00:08:35 go move up to O365. Securing your own exchange is insane. Sure, sure, but I guess, Alex, my point is the Chinese don't trust it, should we? Well, the Chinese don't trust it because of- That's the crazy part of this? Well, the Chinese don't trust it because of FAA 702, right? And despite, we have the Cloud Act, which was supposed to secure overseas data from certain kinds of US requests. But that is really something that's only being applied to our allies, the UK, the EU,
Starting point is 00:09:01 the other Five Eyes countries. For the Chinese, the Cloud Act does not get you anything, right? And so I do think they're reasonable in their risk, but that's not, you know, for us as Americans, Chris and I, we are within the jurisdiction of the United States no matter what. So like whether or not, you know, we have Fourth and Fifth Amendment rights and they have to get a warrant to get to our data. But for like if you're a foreigner,
Starting point is 00:09:26 and your country is non-ally of the US, I think it's totally reasonable to be afraid of US cloud providers. I think the EU situation, there are legitimate complaints. And it's also a little bit silly, because the EU folks who are complaining about American intelligences, one, the EU has way worse intelligence laws than the United States does. And two, they don't talk enough about the Chinese risk. But see, the Chinese aren't just intelligence laws than the United States does. And two, they don't talk enough about the Chinese risk. But see, the Chinese aren't just worried about 702. If that were the case, they wouldn't be pulling out Intel and AMD chips, and they wouldn't be pulling out Windows, right, for on-prem.
Starting point is 00:09:54 So I guess what I'm saying is they definitely have some security concerns about the use of Western tech. And what I'm getting at is maybe we need to share those concerns, right? But for different reasons, right? For different reasons, yes. This is, I mean, you're getting at more of a bug door issue, right? That maybe the security controls in place in Redmond and elsewhere aren't up to what our level of security should be.
Starting point is 00:10:22 Yes. Which is not, I think, the Chinese question or their concern is that it's actively being exploited in cahoots with the NSA, which is not, to my knowledge, the case. Yeah, I see no evidence. I mean, China wants to be independent on semiconductors, not because American semiconductors are all backdoored, but because they see it as a critical, competitive issue for the 21st century. And the Biden administration's AIEO has demonstrated for the Chinese of how easy it is for the United States to cut them off from what they consider to be something that's completely critical for
Starting point is 00:10:55 their national competitiveness. I don't see any evidence of them legitimately thinking that AMD chips have a bug door in them. And I think that would be an absolutely insane thing for the US government to try to do. Of one, the number of people at these companies that would have to know about it would be quite large, right? Like it's a hard thing to insert something like that into a CPU that has thousands of people working on it. But also creating, the nice thing about the XZ backdoor and that you've seen in the US backdoors in the past,
Starting point is 00:11:22 like the dual ECD RBG, is that they are nobody but us back doors, right? Like they're specifically cryptographically designed. And I have not seen any evidence of something of that complexity in the chips. There's bugs all the time, right? Like the errata on an Intel processor is incredibly long. I mean, I'd accept what you're saying on the hardware side. And Chris, though, I do just want to challenge something you said there, which is the idea that the NSA would be in cahoots, you know, with Microsoft and pushing out, you know, means of access to customers based
Starting point is 00:11:56 overseas. Like, you know, that isn't kind of unthinkable right now. But I think if in 2028, you know, China and the United States wind up fighting each other in the Taiwan Strait, I think those sort of conventions tend to go out the window, you know, China and the United States wind up fighting each other in the Taiwan Strait. I think those sort of conventions tend to go out the window, you know? You've been reading somebody's book pre-release, haven't you? And that is, so whether that would happen, I honestly don't think so, at least based on some of my prior conversations. But, you know, these kind of norms tend to be broken. But, you know, every large software company in the U.S. has has had that internal discussion with their general counsels, with their presidents and chief legal officers. So this is not an uncomfortable or it is an uncomfortable conversation. It's not an unfamiliar conversation. But again, to the bigger piece is the new battle space is technology. And it doesn't matter where you are in the world, whatever conflict you're in the middle of, whether it's cyber or info ops or whatever, it's now a critical part of military doctrine.
Starting point is 00:13:00 And that is never going to go away. That is life as we know it, as our kids know it. We are now in the Neuromancer space, right? I mean- So the question becomes though, Chris, and that's why I was so keen to have this conversation is what unfurls from this realization? Because this is stuff that we've known, right?
Starting point is 00:13:22 All the people on this call have known this for quite a long time. As we've kind of established, governments are having this wake-up moment, right? And not just in the West. Globally, governments are realizing that their sovereignty is threatened by their reliance on foreign technology for a multitude of reasons. It could just be because it's a critical capability that they need to have assured access to in the case of hardware. You know, it could be security concerns around either cloud services or on-prem software.
Starting point is 00:13:50 Open source proprietary doesn't matter. But when you start trying to chart a path out of this to a safe place, I mean, I get stuck trying to think of what that looks like, right? So that's why I wonder how this can unfurl. I think it's realigning what our expectations and baselines are. I mean, I get stuck trying to think of what that looks like, right? So that's why I wonder how this can unfurl. I think it's realigning what our expectations and baselines are. I mean, even with ransomware or cybercrime in general, we're going to have to accept some kind of environmental level of activity and crime, just like in the real world, right? There is crime. It happens. That's why you have locks on the door and security systems. There's going to have to be some kind of baby that we're just not going to be able to stop.
Starting point is 00:14:26 There are not enough law enforcement resources combined in the world to deal with where this is going to go. We're just going to stop the big stuff. We're going to tamp it out. Espionage. Real world meatspace spies still happen. It's happened for forever. So some of that, some of this stuff that whether it's the
Starting point is 00:14:47 recent svr breaking in and stealing secrets it's going to be you know to use your line you know you don't have to hand it to them but it's like ah you know some of this stuff and it's more about detection and managing blast radius. And that's what, when you go back to the Chinese compromise of Microsoft, I know it's very confusing. There's just been so many lately. I myself get a little lost. I mean, I do find myself thinking now, which one of it was the one that stole the key material?
Starting point is 00:15:21 Was that the SVR or was that the MSS? So it actually gets confusing, right? I wonder if the Chinese and the Russians themselves get confused about this, right? That's like, we just finish each other's sentences. It gets so, they're so tight inside of Microsoft's network. They just went like that. I think that happened to SolarWinds too.
Starting point is 00:15:39 So with the signing key, what was so interesting about that one was how it was initially uncovered and actually quite quickly, right? So the state department had baselined the network, really crammed down on admin privileges and accounts, and they had a good alert management system and they were running analysis over the top of there like, that's interesting. That's never happened before with that account.
Starting point is 00:16:06 They flagged it for Microsoft. Microsoft was like, we don't know what this is. It was actually an internal tool, wasn't it? It was an internally engineered detection that actually picked this up. What was it? Yellow bus or taxi? I can't remember the name they gave it, but well done to the State Department people who are probably listening to this. Oh, so well done on multiple fronts.
Starting point is 00:16:25 But if you know anything about federal civilian government, you know that State Department has had a hell of a time over the last decade plus with keeping the bad guys out and not being able to keep them out. This shows to me significant improvement in the security posture at the State Department. So it was just like round of applause all around for State Department. This was really impressive work. I got to see this when we were at Facebook. And I mean, in that era, State Department employees were just friending people on Facebook who said, oh, I'm also an intern. And I saw you in the cafe. And it turns out it was like an Islamic Revolutionary Guard, you know, social engineer who's then planting malware on their State Department
Starting point is 00:17:14 computer because they're checking Instagram from work. So, yes, this was a huge improvement in State Department. But to echo what Chris said, the core here is on these platforms, both in the cloud and then for the things you bring in like open source is you have to make sure that you have at least a chance of catching the exploitation right and that that that is the difference between these two hacks is at least with a if open ssh gets popped there's at least steps after that that attacker has to take to be successful right they're going to pop a shell on the machine they're going to look at files they're going to touch things that they might want they might need east west going to pop a shell on the machine. They're going to look at files. They're going to touch things. They might need east-west movement
Starting point is 00:17:47 if they're not on the correct machine. The concerning thing about the Microsoft hack is if it is completely invisible of all the activity that's happening, as a customer, you can't control it. And I think that's both what you need to build is everybody should be building systems like the State Department
Starting point is 00:18:02 of utilizing the APIs that are available. But then we also have demand of our cloud providers to provide us a level of transparency into the internal operations of their system that they find uncomfortable. And that is a- I think to your point, State Department had the G5 logging, right?
Starting point is 00:18:17 The really expensive premium logging. And if they didn't have that, they wouldn't have been able to find this. And perhaps this is something that should be offered to civilian agencies. And I believe Microsoft has finally conceded some ground on that. I think they fixed that just for the government,
Starting point is 00:18:30 not for enterprises. But like, this comes back to like, and I do appreciate you and Adam talking about my LinkedIn post, which is, you know, the most middle-aged thing I've ever said, right? Thank you for reading my LinkedIn post. It's just instantly aged myself.
Starting point is 00:18:46 But my biggest complaint is Microsoft cannot upsell. You cannot announce your breach and upsell your products. If you're going to provide a security-critical cloud product, then the things people need to protect themselves and to detect intrusion have to be into the base product. And I think it's just completely unethical to turn this into a packaging thing of you cannot get access to the API that lets you check whether the Chinese are reading your
Starting point is 00:19:11 email unless you pay for the G5 license. Well, I think too, just you were talking about the XZ stuff and how even if there were a backdoor in OpenSSH, eventually you'd catch it if you were doing a good job. And that's true. And I think everybody needs to be operating under a model that assumes Oday exists, right? So whether it's an intentionally placed backdoor or whatever, you just have to operate your security program to understand that at some point an attacker
Starting point is 00:19:34 may mysteriously appear on a box for reasons that aren't clear to you and what do you do then, right? Which is why you've got to do your, you know, decent robust detection of lateral movement, strange things happening in accounts and whatever. So I do feel that the xz stuff i mean it was fascinating it was a great story i'm a journalist i lapped it up you know we led with it it's fantastic but to a degree that sort of thing is overhyped in that we always have to assume this sort of stuff exists the way it's
Starting point is 00:20:01 not under overhyped is if it had made it into if it had made it into like base ubuntu and debian uh and fedora and red hat linux is it could have it could have been the i think it would have been a hard bug to use super quietly because you are going to be spawning shells and such yeah but i mean shell shock shell shock made it into all of the mainline distros heart bleed made it into all of the mainline distros do you know what i mean and of course the mainline distros. Do you know what I mean? And of course they were big news as well. But my point is intentional or unintentional kind of by the by, right? Right, right. But there hasn't been a like RCE, an easily exploited RCE in OpenSH for a while.
Starting point is 00:20:36 This would have been an incredible 12 hours, right? Like it's possible they would have used it super stealthily of like you're already in a company and you're only using it for East-West movement movement you're only using a situation where you know they don't have edr they don't have logs or you have the most incredible 12 hours of your life because every single open port 22 on the internet becomes yours um and i think either of those are possible depending on you know who it comes down on the attribution it could have even been a possible thing of like we're going to tuck this way for for World War III or some equivalent of like to be the opening moves of us to be able to disrupt massive amounts of infrastructure.
Starting point is 00:21:10 You just said something interesting there, which was World War III, right? So Chris, earlier you were saying, well, you know, we have to accept that there's no such thing as perfect security. Espionage will continue. Cybercrime will continue. It's about managing those numbers, right?
Starting point is 00:21:23 So where I get the sense that things have changed is that people are a little bit more worried about the sort of existential world war iii scenario and that's what's driving a lot of these concerns around sovereignty and technology okay and that's not a place we've been before we saw this what like a decade ago where australia was i believe the first to ban Huawei, right? We were like, we don't want them controlling our telcos because – and the concern – people have reported it as like people are worried about backdoors and surveillance and things like that. just brick our entire LTE, you know, or 5G network. If hostilities, you know, and in the time since, there's been a bit of drama between Australia and China, not to that level, but, you know, certainly it seems in retrospect, like it was a good decision. So I guess what I'm getting at here is that some of these sovereignty concerns now are not, it's not like, oh, wow, you know,
Starting point is 00:22:24 we're vulnerable to espionage because of these supply chain concerns. It's if a hot war breaks out, because it really does feel like the world is becoming a much more dangerous place, what then? Yeah, I mean, look, this was my argument in 2018 and 19 in government for Huawei and 5G, right? When the prevailing narrative, particularly with Five Eyes allies, was, oh, you know, people are going to be snooping in on our conversations. And that's a lot of the messaging that you were hearing between governments.
Starting point is 00:22:56 And I was like, no, no, no, no. Look, I mean, that's what you got encryption for. You know, you should. It's whether the network's up or down when you need it. And if they are on the control plane and they bring it down, you're toast.S. and China right now. All of the Volt Typhoon stuff that came out last year, the concern is about preparation of the environment. It's that they're getting a foothold into not just military targets between bases and telecommunications facilities and ports, but also mundane everyday critical infrastructure in Atlanta, in Dallas, in Chicago, like wherever. Because they, and this is what I was talking about earlier with the InfoOp side, their
Starting point is 00:23:57 military doctrine hinges on technical attacks that result in societal panic. They want to get up here. They want to get up here. They want to get in our heads. And they want to undermine our ability to subsequently prosecute that war. So this is not so much about cyber Pearl Harbor or cyber 9-11. It's about the thousand cuts.
Starting point is 00:24:21 It's about bang, bang, bang, bang, bang. And it all builds up. It accretes. And my concern about the thousand cuts. It's about bang, bang, bang, bang, bang. And it all builds up. It accretes. And my concern about the China stuff is, one, they are planting back doors that they know have to have a limited shelf life. Right. And so it does concern me that there's people in the PLA, assumably, who believe that it's their job to be ready if the word comes down within 72 hours. They need to use this stuff within the next six months or it's gone. Right. That's somebody who's going to say, I want the power out in San Diego, and that's
Starting point is 00:24:49 going to be something that they've done some work on. And what concerns me here is, for people who are interested in this, I strongly recommend CSIS did a war game scenario of a Chinese invasion of Taiwan and a US and allied response in January of last year. And it's absolutely fantastic document. It's obviously unclassified and they don't have the classified inputs, but a lot of people who are doing the war game come from the high side. And so I expect it is as close as possible. You'll see in the civilian world to the kinds of war gaming that done both in Beijing and Washington DC on this. And I'll just read real quickly from the conclusion. The overall finding is that China is unlikely to succeed in the invasion of Taiwan in 2026
Starting point is 00:25:29 if four conditions hold. One, Taiwan has to vigorously resist. Number two, the United States must join hostilities within days and with the full range of its capabilities. Delays and half measures make defense harder. So effectively, their simulation showed that if the U.S. was slowed down in responding in the Pacific by just a couple of days, that completely changes the outcome of the war. The Chinese clearly know that, right? So they don't need to knock out power in the United States to turn us back to the Stone Ages for six months or a year. If the power's out in San Diego for two days, if the power's out in Guam and Pearl Harbor, if the railroads aren't working to get the Marines to the bases in Seattle and San Diego,
Starting point is 00:26:05 then maybe that's it. Right. Cause all they need is an extra 72 hours. And that's, that's what concerns me is that like people always talk, like Chris said, that cyber Pearl Harbor, where it's the total end of the world. It's like, you know, diehard, uh, you know, style fire sale. Right. And you don't need that. Like if, if America was distracted because the stock market dropped 2000 points and powers out in a bunch of cities and the water's not working and it's very hard for DOD to do their job, then perhaps that's enough that by the time the president of the United States gets out of the immediate mess,
Starting point is 00:26:34 the Chinese already have boots on the ground and, and throwing a million American lives into that war doesn't make sense. I mean, just going back to what Chris said, then that I'm sort of swinging back more towards your position, which is, you know, I, even though under the Entra ID model, everyone in the world is essentially on the same directory,
Starting point is 00:26:55 like the chances of China being able to vape that directory are pretty slim, right? So, you know, then you are talking about these paper cuts and you were talking about, you know, we're back to that old problem statement of just trying to control it as best we can. I don't know. I mean, if they were able to knock out a significant amount of Microsoft's infrastructure, that would, I mean, the economic impact would be pretty significant. Yeah, absolutely. I find it highly unlikely, even with all the crap Microsoft's gone deservably over the last couple of weeks, that you could just nuke all of Azure AD, but I'm
Starting point is 00:27:23 sure you could cause a decent amount of disruption. And if 90% of the employees of large companies in the United States can't log in, we've never tested that from an economics perspective, but it would probably not be that great, right? If every morning nobody can get their email. It's an unscheduled public holiday for everyone except the security team.
Starting point is 00:27:43 Yeah, exactly. And I've had these conversations over the last probably 10 years with the major cloud players. And I think you would expect them to say this, but when you really, really push them, and I don't mean just to ask them, but you really spend a lot of time pushing them
Starting point is 00:28:00 on developing a scenario that would result in a catastrophic outage, they're really unable, particularly for a long term, they're unable to generate that scenario, one that they find believable. And so I think that goes to Alex's point, it's like, you know, the survivability and the resilience of the major cloud providers is actually, it's not bad. The question is, do our, you know, like,
Starting point is 00:28:30 when I say SLAs, I mean it more like from a casual perspective, like what's our, as society, expectation of the SLA with the cloud providers and what can we deal with? You know, from a business sense, like, yeah, you know, it's a couple of days we'll be out.
Starting point is 00:28:46 Like a couple of days in the real world? Uh-uh, that's not going to cut it. Things start dropping out and that information space is going to get filled with a lot of nonsense and the bad guys will take advantage of it. I also think if somebody's intentionally, if they know what they're doing and they're intentionally taking out a cloud provider,
Starting point is 00:29:04 I think it'd be more than a couple of days. We simulated this at Facebook of, we tested outages all the time. The infrastructure team there is incredible. And they would do things like pull the plug on an entire data center to see what happens. Right? So like entire global data center would be shut down to make sure that everything else could adapt. And we simulated, well, what if it's not an accidental thing
Starting point is 00:29:25 like a data center go down, but somebody had the ability to push instructions to millions of production hosts and RM-RF. And the problem is for hyperscalers, like the cloud providers, the Googles, the Facebooks, the Microsofts and such, is that the way you bring up a data center is you copy it out of another data center, right?
Starting point is 00:29:42 Yeah, I mean, I was, the whole time you're talking about this, I'm going back. I can't remember who it was who first pointed this out. It might've been Dan Geer actually years ago, which is saying if one of these major cloud providers fell over, the amount of data they hold, we don't have the networks that would be capable
Starting point is 00:29:58 of transferring it to one of the other providers and they can't scale up to meet it anyway. I'm sure we all remember when COVID first kicked off and lockdowns happened, teams started falling over because Azure didn't have the capacity to spin up more. And I think that's the issue. Like getting an entire hyperscale cloud provider to fall over is a stretch.
Starting point is 00:30:19 Yeah, but getting a couple regions and then the cascade over into the remaining regions, you're going to see significant performance degradation to the point where things will start, you know, to your point, timing out and dropping off. Right. Yeah. But is that risk? Hang on, hang on. This is right at the core of what we're talking about. Is that risk, do you think, Chris, well understood in government?
Starting point is 00:30:41 Ah, that's a good question. Um, I w I would suggest in certain areas more like in, um, you know, well in NSA, yes. Well, I think it's probably limited to a couple of places. One would be in, uh, some areas of the Pentagon, some areas of the intelligence community, some areas of the federal communications commission, of the intelligence community, some areas of the Federal Communications Commission, and then our good friends at CISA. But it's beyond that. I'm telling you right now, man, the complexity of the United States economy is far beyond anything i think the u.s government has the capability to truly understand and we saw that in covid so many little pieces broke we saw that would change health care right didn't see that one coming did we like it's there are there are systemically important pieces of infrastructure
Starting point is 00:31:44 out there that we don't have a full understanding of how they fit into the bigger equation. And that, that's what I, by the way, so that's what I said, we set up the National Risk Management Center for was to get know the things that are important. Do we know who's providing them? That's the big information gap, knowledge gap. I guess, you know, this comes back to the whole premise of this conversation, right? Which is, okay, you've got China and Russia turning their back on Western technology. The United States is in a fortunate position because it owns most of the supply chain. You know, most of its own supply chain is either American or comes from allied countries, right? I guess the point is, though, owning the supply chain isn't enough to make it secure.
Starting point is 00:32:37 Just because you own it doesn't mean you can trust it. Is it a state-owned enterprise? I'm talking about your Microsofts. I'mowned enterprise? I know. I'm talking about your Microsofts. I'm talking about, you know. But that's the point. They're not state-owned enterprises. If we were living in the upside down and Microsoft was Chinese,
Starting point is 00:32:56 they would be controlling the absolute crap out of it. And that's what leads us into this next part of the conversation. A lot of that American supply chain is in Taiwan. Certainly in hardware, yes. That's absolutely true. When you talk about those Microsoft machines, those Amazon machines, most of them are manufactured by Taiwanese OEMs
Starting point is 00:33:12 like Quanta and such, right? Like they're not buying Dell. And so I... But I guess this leads into a part of the conversation I know you're keen to have, right? Which is when SolarWinds experienced an incident at the hands of Russia's SVR, there was hell to pay. They will never be able to separate their brand from the incident.
Starting point is 00:33:33 They are being pelted with rocks as they walk down the main street wearing a sandwich board that says shame. Microsoft has the same sort of stuff happen to it and nothing it's like they're skating i mean with the exception of the csrb report which seems to be a you know a decent step in the same direction but this comes back to this whole conversation about sovereignty and supply chains you know the united states has its own supply chain that it seems unable to really strongly influence. China doesn't, but it's trying to spin one up that it can. So it just seems to me that there's a bunch of interesting stuff happening around these concepts. Yeah, I think we are having also a very
Starting point is 00:34:19 naturally, a very kind of security focused conversation here, obviously. I mean, when you step back and look at it, with the top 10 most valuable companies in the world, eight of them are U S companies. And then like six of those are tech. So, you know, we're doing something right. And to your point on, you know, the ability to influence, I, I think the government's still still struggling with what are the right market interventions? Regulation is new legislation. I mean, we don't even have a federal privacy law. I've talked about this. Alex talks about this all the time. Well, there's one being floated at the moment, so that's progress. There's at minimum one being has... One has been floated every Congress for the last 10 Congresses. So it's like, I've seen this movie before.
Starting point is 00:35:08 And at some point, the retailers and the banks will come at it and they'll fight over it and we'll see what happens. Right, because everybody forgets that privacy laws don't only apply to tech companies, right? There's a huge industry in the United States of traditional companies like the retailers, the banks, especially financial services that collect a huge amount of user data and have been able to hide so far. That's yes. So that's an entirely different podcast. And then you've got at the same time you've got AI and, you know, half a dozen AI bills out there that I honestly don't I'm'm not sure we'll go much of anywhere, but to your point, there's got to be, I think there's got to be a lot more on the power of the purse side for the federal government. And at least this administration, well, the last one too, with section 880 and saying, hey, we're not going to buy Huawei. We're not going to buy all these other Chinese
Starting point is 00:36:02 backed products, using things like the binding operational directive to uh to block uh and rip out kaspersky um that was by the way with commerce's recent announcement that they're finally going to block it in the domestic i just saw that this morning yeah like a wide a country-wide band on ban on kaspersky it's's like, what year is it? What's happening right now? I think influencing the domestic manufacturing base for tech and software is more than likely going to happen through the purchasing power of the Pentagon, for instance. And look, you know- But we've already established that microsoft has kind of a monopoly on the type of tech that is used by organizations like that there's no alternative to
Starting point is 00:36:51 excel there's no tightly integrated business suite that combines you know identity that you can sync with on-prem and with a cloud suite and teams and this and that it's all beautiful right so i'm a real skeptic when it comes to the purchasing power argument. Yeah. So, well, I mean, you would hope that the government through the federal acquisition regulations, whether it's FAR or DFAR, can keep leveling up on the requirements. Now, we all know that those requirements are heavily influenced by special interests and lobbyists and consultants and stuff like that. And they tend to be, you know, those with the biggest bucks that can do it. But we will see
Starting point is 00:37:30 improvements. And my hope is that if that happens, if those, you know, let me be a little, you know, rose-colored glasses here. If we can get those standards up, if the procurement does require better performance. I don't think that these software companies are going to bifurcate. They're not going to have, whatever they do to improve the federal government sale or SKU will likely be the same code base for the commercial.
Starting point is 00:38:00 That's my hope. Well, you say that, but I mean, I recently had a conversation with someone from the US defense industrial base where there's a whole bunch of requirements for Dib companies to exchange unclass information, which are at a much higher level of security. And there's a full Microsoft suite of products that check off those compliances, and it costs like 10 times as much as regular Azure because it's a specialty product and it's not available to everybody else. So I don't know that I'm as rose-colored on that as you. That's a different situation,
Starting point is 00:38:31 but you do make a good... One point though, it's going to cost more. Whatever we do here, when we raise this level, cost is going to get baked in. And that's going to be how the companies come along. Because they say like, okay, fine, we're going to spend more, we're going to spend more to secure the product. But you know, cybersecurity is it has to be an allowable cost in the contract. So so there will be, there will be a price tag associated with whatever the outcome is. Alex, you got some thoughts on this? I want to go back to your question about, which I think is back to, does anybody in the government understand the fragility here? And I'm not sure because I think the fragility of the cloud providers in particular is based upon the incredible operating leverage
Starting point is 00:39:15 they have of the number of employees they have versus the number of machines. And that operating leverage is 10 to 100 times better than any government agency, right? You couldn't take the Pentagon down even if you had the best SVR team because everything's heterogeneous, right? Like you got your AS400s next to your Windows 2008 machines, next to your Linux boxes, whereas the cloud providers are completely homogenous. And that is why you can have a 100,000 to one ratio of DevOps engineers or system administrators to systems. And so I think that's part of our challenge here is that one of the reasons that people love the cloud is it is cost effective, but the cost effectiveness also is why you have things like outages of all of
Starting point is 00:39:56 us East one, or why you can have a global Google outage when you don't hear about that from other companies, like generally big fortune 500 is the entire company doesn't go down. It's because everything's disconnected and everything at Google is connected up to the same systems in the same code. And so I do, I am concerned of the operating leverage we get as a society from cloud is wonderful and great in lots of ways. And it's also something the bad guys have figured out exists. I want to add, this just triggered something, by the way, when you asked, does anybody in government understand this? I left out NIST and NTIA. I'm sorry. I'm sorry, Adam and Kevin.
Starting point is 00:40:34 But look, NIST gets it, NTIA I think gets it as well. But again, the question goes to what can they do about it? Yes. And a lot of these agencies are just in a place where they influence policy. But that was my point about, well, China's trying to take control of its stack. And it's a long way behind, and it's not going to be anything as nice as what America has. But they will have a lot more levers. And that's something where I think America's in a tough spot, because it doesn't have the
Starting point is 00:41:01 same levers available to it, where it can just send some party flunky into the boardroom and say, you're doing this. Yeah. It's a big difference. I mean this, you know, when we have the discussion of Chinese companies, um, or, you know, the having multinationals that have folks who are either in China or, or higher Chinese nationals over and over again, we see the evidence that the PRC is willing to use every bit of leverage that exists for them, including people's families. And, um,
Starting point is 00:41:31 so I guess what we're saying is the United States should embrace communism. No, uh, guys, that's what, right. That's a totally reasonable, uh,
Starting point is 00:41:38 read of what I just said. Yeah. Absolutely. Like, where does that, where does that leave them? Right. They're off.
Starting point is 00:41:43 They're on red, red star Linux. And? Do you think the quality is going to be that great anytime soon, to your point? Like 30 years from now, maybe? But that's the point. Red Star Linux would have had this backdoor
Starting point is 00:41:57 if they had waited on that. But I guess what I wonder is, who has the edge here? Does control beat the free market supply chain that America's built? And no, clearly it doesn't, but there are advantages in both, I guess, is my whole point. And again, like, what are you trying to achieve? You're not going to hear Chris and I talk about the downside of American capitalism. But, like, I think if we want to really face the threat, especially from the PRC, but from rising authoritarianism everywhere, we do have to look at our own weaknesses, honestly.
Starting point is 00:42:30 And I do start to see, I'm starting to see that, right? Like you do see analysts who talk about the weaknesses in the American system. I also think like over the last couple of years, for a long time, America, we completely gave up the idea of industrial policy, right? Of the free market is all, lowering trade barriers, we're not going to do anything to protect industries, was the appropriate way. And that has reversed aggressively, and it's working, right? Like the CHIPS Act is working.
Starting point is 00:42:56 The AIEO, while seen as an act of war by the PRC, it's partially because it's working. And so I do think we're hitting a good ground where we look at the decisions that are made by the PRC, it's partially because it's working. And so I do think we're hitting a good ground where we look at the decisions that are made by the free market. And every once in a while, we're going to have either incentives through money or disincentives through the ban of certain exports and such to kind of reconfigure that and make sure that the West, that the free world is self-sufficient in these technologies. So I think we're getting into a reasonable... We're in a damn sure better spot, right?
Starting point is 00:43:28 I mean, that's the thing. We'll still control the heights of the market or whatever you want to say. And that's what's been... When I read the CSRB report, I kind of sit in this unholy trinity spot as a former Microsoft trustee, where the computing guy, former government guy,
Starting point is 00:43:46 and now competitor, it's like all these, there are three wolves inside me with this one. And there's, there, there's a, there was a bit of me that was just like, I'm not mad.
Starting point is 00:43:56 I'm just, I'm just disappointed. Like the decisions you made, come on with the right decisions being made. I think we can be, we'll be okay here. All right. Chris Krebs, Alex Stamos. Thank you so much for joining me. I've been looking forward to doing, you know, to having this conversation, to doing this podcast. It's been years really in the making to make this happen, right? It's the great crossover event of the year right here. Yeah, and huge thanks too to SentinelOne
Starting point is 00:44:28 for backing this and partnering with Risky Biz to actually do this. It's terrific. Good vibes and high fives all around. Great to see you both, and I look forward to doing this again in a month. See you, Patrick. Thank you so much for having us.
Starting point is 00:44:40 Thanks, Patrick, for having us on. That was Chris Krebs and Alex Stamos there with a chat about supply chain trust. I found that fascinating and I hope you did too. And that is it for today's podcast. I do hope you enjoyed it. These will be happening about once a month for the foreseeable future. Let me know what you think. I'm Risky Business on Twitter, one word, and on Mastodon. There, I'm on the InfoSec Exchange instance and feedback is welcome. I'll be back next week with more risky business for you all. But until then, I've been Patrick Gray. Thanks for listening.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.