Big Compute - The SolarWinds Hack: Worst Case Scenarios

Episode Date: March 23, 2021

Never before has a hack of this sophistication and scale been seen.  But now that 18,000 organizations are considered breached, what can the hacked information be used for?  We walk through the worst case scenario possibilities of what the cyberattackers could do with the SolarWinds hack data -- from espionage to overwhelming electric grids -- and what that could mean for all of us, including those in the high performance computing industry.  We also explore the Senate and congressional hearing testimonies given by Kevin Mandia, CEO of FireEye, and Brad Smith, President of Microsoft, about what the hackers went after once they were in the system, and whether the future of cloud poses a greater risk or a stronger solution.  

We've mentioned Firefly who had hacking tools stolen in this breach. Firefly or FireEye? Oh man, why does it say Firefly? I don't know. I wonder if I wrote that. I think I did. I was like, that's my fault. I'm sabotaging you. Hi everyone. I'm Jolie Hales. And I'm Ernest DeLeon. And welcome to the Big Compute Podcast.

Here we celebrate innovation in a world of virtually unlimited compute, and we do it one important story at a time. We talk about the stories behind scientists and engineers who are embracing the power of high-performance computing to better the lives of all of us. From the products we use every day to the technology of tomorrow, high-performance computing plays a direct role in making it all happen, whether people know it or not. So, Jolie. Yes. We left the last episode on a little bit of a cliffhanger, didn't we? Uh, yes, I should say so.

I mean, in fact, if any of our listeners missed the last episode, I want to take this moment to highly recommend that you go back and listen to that one before continuing with this episode so that you have all the background information that you need to understand what's coming next. It's basically this is part two of that episode. Right. Or you might be a little bit confused about what's happening in this episode because we will definitely reference some things that we talked about in the last episode. And for everyone else who has already listened to that one, just a quick review on where we are. So SolarWinds, who has been in the news a lot lately.

SolarWinds. SolarWinds. SolarWinds, who has been in the news a lot lately. SolarWinds. SolarWinds. SolarWinds. Is an IT management software company that provides products to tens of thousands of organizations, including a cybersecurity company called FireEye. And just a couple months ago, FireEye was the first to discover that they had been hacked. And eventually, the source of that security breach was traced back to a commonly implemented SolarWinds software update on their Orion product. Yes, the malware was actually attached to an update that rolled out to countless organizations, ultimately breaching the systems of at least 18,000 organizations. Right, and FireEye was just the first to catch it. And the other organizations that were breached include major tech behemoths like Intel and Microsoft, as well as United States government agencies like the Department of Defense.

And actually, I'll note something here. There was a congressional committee hearing that happened not too long ago where they questioned some of this. I'm so glad you brought that up because after we recorded our last episode about this hack, I got really curious and I ended up watching and taking scrupulous notes on all eight hours of both the U.S. Senate intelligence hearing and the U.S. congressional hearing that took place at the end of February just a few weeks ago, where leaders of SolarWinds, FireEye and Microsoft all spoke about what happened. One of the hallmarks of this operation was the great care that was taken by this adversary

to use bespoke infrastructure and tradecraft for each victim. That's Senator Marco Rubio, who serves as vice chair of the Senate Intelligence Committee. For those who haven't seen that news clip, you should really watch it because there's an interesting exchange between one of the congresswomen and I believe it's the CEO.

Oh, I know exactly what clip you're talking about. The congresswoman was Katie Porter, who actually represents my district here in California. And let's just say that she didn't go easy on the new SolarWinds CEO. Is it true that some servers at your company were secured with this Cracker Jack password, SolarWinds123? Congressman, I believe that was a password that an intern used on one of his GitHub servers back in 2017, which was reported to our security team, and it was immediately removed. I've got a stronger password than SolarWinds123 to stop my kids from watching too much YouTube on their iPad. You and your company were supposed to be preventing the Russians from reading Defense Department emails. Oh, man. Oh, man.

It was pretty brutal. That would be a hot seat to be in, I think. Even if it's true that it was an intern, let's just hypothetically say it was. This is a failure at the organizational level in terms of security. And that's where it lies. The individual who did it, yes, they are obviously culpable for doing it. But there are many layers that had to have failed beyond just the one person. I agree with you. And to be frank, I actually have a lot of sympathy for SolarWinds because,

I mean, being hacked these days is so easy. It often just takes one person being duped, you know, through social engineering or whatnot. and then a whole organization can end up paying the price from what I understand. But at the same time, general security measures concerning high stakes passwords should be common sense implementations and an intern really shouldn't be given that level of control. That's right. As a matter of fact, I often say it's not a matter of if you're going to be hacked or breached or compromised. It's a matter of when and how. Yeah. Right. And in the security world, that's kind of the assumption we always work on.

Our goal is obviously to try to stop that from happening. But our secondary goal is to notice when it does happen and rapidly respond to it. Well said. And when we left our last episode, I still had a lot of questions about this hack and what it means. So specifically, I want to know what kind of information was actually taken, if we can know that, and what can it be used for in like a worst case scenario? There are a lot of things to unpack here. So let's start with

what kind of information the hackers made off with. Let's. This matters for many reasons, not the least of which is what further damage can be done with the stolen data? Yeah. And that's probably my number one question at this point. Understandably, we've mentioned FireEye who had hacking tools stolen in this breach. Yes, FireEye. And I've got to say, when I watched hours of those hearings, in my perspective, their CEO, Kevin Mandia, I mean, he might as well have been wearing a cape because he came across as not only very credible, but I would even go so far as to say heroic

for leading this effort that found the hack first and then for sounding the alarm immediately. I know that probably sounds kind of cheesy, but he clearly knew his stuff during the hearings and he seemed to authentically have the nation's best interests in mind from the very beginning. So he definitely won me over. And he explained the hack in a way that a lay person could understand. Whoever this threat actor is, and we all pretty much know who it is, this has been a multi-decade campaign for them.

That's the FireEye CEO slash undercover superhero himself, Kevin Mandia. I want to explain how we found this implant because there's no magic wand to say, where's the next implant? When we were compromised compromised we were set up to do that investigation it's what we do we put almost 100 people on this investigation almost all of them had 10 000 hours there's so to speak 10 000 hours of doing investigations and we unearthed every clue we could possibly find and we still didn't know so how did the attacker break in so we had to do extra work. And at some point in time, after exhausting every investigative lead,

the only thing left was the earliest evidence of compromise was a solo inserver. And we had to tear it apart. And what I mean by that is we had to decompile it. Specifically, there was 18,000 files in the update, 3,500 executable files. We had over a million lines of assembly code. For those of you that haven't looked at assembly, you don't want to. It's something that you have to have specialized expertise to review, understand, piece apart. And we found the proverbial needle in the haystack, an implant. But how did we get there? Thousands of hours of humans investigating everything else.

And that's one of the reasons I share that is you wonder why people missed it. This was not the first place you'd look. This was the last place you'd look for an intrusion. And honestly, I don't know about you, but I kind of feel like FireEye deserves a lot more praise than they might be getting just for making this discovery and then immediately alerting the public. If I had to sum it up in like one sentence, FireEye did the right thing. They noticed that they had been hacked. They noticed the types of things that right? Legal and PR. And they try to keep it to themselves until they're sure of what happened. And then they go public with, you know, some kind of spun statement.

In this case, FireEye just came out with the truth right away, put the list of tools that had been compromised and also listed a bunch of mitigations to help stop potential attacks with those tools. So they should be praised in this situation because they absolutely did the right thing. We did not have a legal requirement, at least based on the legal advice that I got to disclose at the time that we did.

So we did so based on we're a security company, we work to a higher order. Yeah, it's all built on trust and you gotta report. Now, when you say hacking tools were stolen, what are you referring to exactly? So if you want more detailed information into what was actually stolen from FireEye, you can go to their website. They have a blog post that actually covers the entire incident and it lists the specific tools that were potentially stolen and mitigations they put in place. Take some of our red teaming tools that we use to assess people's security programs. Let's just say there were many tools that are used in a toolkit for doing all kinds of hacking

to find and probe for known vulnerabilities in software. And these were all taken. OK, so tools that the ethical hackers typically use were taken by the not so ethical hackers. That's right. So Orion is what got them into the FireEye environment and then they stole these tools. And that brings up a really interesting point. We've been told that the Orion software compromised 17 or 18,000 organizations, but it's my guess that once the hackers had access to that many they probably had to pick and choose who to focus on from there. I'm thinking that maybe not all 18,000 organizations are going to be of value to some nation state but a cyber security company like

FireEye or a U.S. government agency I can see why those would be targets. That's right because there's only so much bandwidth, right? They probably had a list of 18,000 companies they could have gone into and looked for stuff. But when they looked at the list of companies, they targeted specific ones. Like you said, U.S. government agencies, FireEye, a very prolific security research company, Microsoft, which handles a lot of enterprise email, enterprise directory services, which include authentication and access. So yes, unless they had an unlimited amount of people to put eyes on all this, they had to focus on the highest value

targets first. Right. So if like a florist in Wyoming uses SolarWinds to manage their IT, I don't. That is a terrible example because I don't think they have a lot of digital flowers. Or a high school in Ohio, right? That's a better example. A high school has enough assets that they would use a product from SolarWinds, but they're not really a high value target like a government agency or a fortune 500 company. After stage one, the attackers had a menu to over 17,000 companies that had downloaded the implant, but that doesn't mean the attacker stole anything from 17,000 companies. The stage two victims are

where the attacker decided I want something. And the attackers manually engaged with about 100 different organizations. In stage two, the attackers did three things. First, steal your keys. They came in through the trap door in the basement that you didn't know about. They took your keys and with those keys, they accessed your information

the same way people and employees do. Second thing they did is they did very specific and focused targeting of documents and emails. And the third thing these attackers did, I put in the other category based on the victim. They stole source code or software. And in the case of FireEye, they stole assessment tools that we use to assess the security of organizations. So when these hacking tools were taken, the immediate concern was, what will the hackers use these tools for in the future? And the truth is, we don't know the answer.

One is espionage, obviously, to obtain information, especially, say, from U.S. government and other agencies. That's the voice of Brad Smith, president of Microsoft, talking about three suspected purposes of this hack. Second, to learn more about technology, because obviously technology is the plane on which this organization's activities take place. That's why 50% of the victims that we identified

are communications and technology companies. Third, I think there's an aspect of this that you'd almost put in the context of counterintelligence. They focus on red team tools so that they know how to withstand attacks. They look for what a company like Microsoft may be knowing about them so that they're able to try to circumvent what we're doing in the future. That's true for other tech companies as well. If I had to speculate, I would guess that they will use these tools to exploit known vulnerabilities in software and systems that other more lucrative targets have. However, you want to define the word lucrative.

So then what do you mean by known vulnerabilities? This is going to sound completely simplistic, but if a company has a digital security vulnerability, especially if they're a lucrative organization, shouldn't they just fix it if it's known? Or are we talking about companies like the small public water facility we mentioned in the previous episode in Florida that was hacked because maybe they don't have the budget to upgrade their digital systems. Yeah. So there's kind of two definitions we look at here for known vulnerabilities, right? Typically, the one that most people are familiar with is the one that you are referencing here, which is it is known publicly. It is perhaps on a public bug tracker of a software project, something like that. It has what we call a CVE attached to it. There's one thing I consistently find today. It's that many of the public sector computers and information systems, software, especially at the state and local level, are not as modern as they should be. Just to give you one example, one Department of Health at the state level that we're working with on the distribution of vaccines, we went to help them strengthen their work.

And when our consultants looked at the manual for the software program they were using, it was for a company that Microsoft acquired more than 20 years ago. So the software was more than two decades old. So part of what I think we need to do is strengthen CISA. But I think part of what we need to do is really across the country at the state and local level, embrace the modernization of our IT infrastructure, and in so doing, embrace the modernization of our cybersecurity protection. But there's also known vulnerabilities that might be known within the ethical hacking community or even more specifically known within an organization like FireEye that they've not released to the public because they are using it for their own penetration testing. So then the hackers, once they got into FireEye, had access to the tools that expose these vulnerabilities that they would not have otherwise known about?

Right. All different kinds of known vulnerabilities, right? Because there could also be the case where customers were running certain types of software that had known vulnerabilities that were public, but the customers themselves are not running aggressive scanning and vulnerability assessments against their own infrastructure. And therefore, while the vulnerability is known to the public, the customer may not know that they are affected by that vulnerability. Okay. Okay. So these black hat hackers who stole hacking tools from FireEye could now use these tools to further breach other targets.

Now, what other targets are we talking about? These are the government agencies or the high value targets that we were talking about earlier? That's right. It could be government agencies. It could be Fortune 500. And depending on the type of tools that were stolen, the type of vulnerabilities that they want to exploit, it could be anybody. So that's really the issue here is that FireEye had a very large wealth of tools that were used for this. So the tools could be used against just about any target. But as you mentioned earlier, they're going to target very specific high value targets

first. And that is primarily because of bandwidth. But also it is not uncommon for an organization to notice when a larger attempt is being done at a breach like extrication of data from the environment. And at that point, you know, they muster all of their resources and kind of shut it down. So if you're going to be found out, you want to get the highest valuable data out first. Before you're discovered. Before you're discovered.

Make the most of your time. Interesting. Okay. Ironically, had they not tipped their hand with the FireEye breach, you know, where FireEye noticed it and then put out that PSA, the larger SolarWinds breach could still be going on today, compromising many more enterprises and agencies over time. It's already suspected that the SolarWinds hack was in place for months. So it would have been a very long time before it was noticed outside of the context of FireEye. For us being a stage two, we had firsthand account of what they do.

The attackers came in through the SolarWinds implant. And the very first thing they did is went for your keys, your tokens. Basically, they stole your identity architecture so they could access your networks the same way your people did. And that's why this attack was hard to find, is these attackers from day one, they had a back door, imagine almost a secret door into your house. And the first thing that happens when they come through that secret door is all your keys are right there. They just grab them and now they can get into any locks you have in your house, the same way your people do. And I think during a pandemic where everybody's working from home, it's way harder to detect an attack like this where the only indicator of compromise was just somebody logging in as one of your employees.

And there was nothing else far-fetched about that. And more than likely, the SolarWinds hack was intended to slowly gain access to as much critical infrastructure as possible before finally launching a larger attack that would have been nearly impossible to defend against at that point. I mean, that's pretty frightening to think that it was all in preparation potentially for a massive attack. I'm the kind of person who likes to know what the worst case scenario is. I'm one of those people who likes to plan for a worst case scenario. But had this breach gone unnoticed, practically speaking, what are some of the worst things that could have been done down the line with everything being as digitally interconnected as it is?

That's actually an excellent question to ask because the answer to that changes every couple of years, every decade maybe that goes by, because more and more systems are attached to the internet. And a lot of times the command and control systems of critical infrastructure are not updated or not patched correctly, just like we talked about the water utility down in Florida. So the worst case scenario, there's several different things that could happen, right? So one of them is essentially stealing state secrets that have to do with defense, right? So if this was a nation state

actor, one of the things they're going to be after is secrets to our military capability, aircraft, sea craft, satellite surveillance, intelligence that we gather. They want to have all of this so that if there's ever a confrontation between that nation state and ours, they have the upper hand in terms of the data. They may not have the technology to overcome what we have, but they at least know what our capability is. Interesting. And they could even make defensive preparations in advance, if not offensive. Correct. And in some cases with other nation states, they're so far behind us, this could catch them up on decades worth of research in either weapons design or whatever the case is. As we look at the world, we have espionage threats, we have disinformation threats, and then ultimately we always have the threat we were talking about before of actually damaging a society or a country. The more likely scenario is something like what happened in Florida,

which is an attack against the actual people of the country as opposed to the government or the military. You may not have heard about this attack because it hasn't affected your daily life. You still go home to a warm house every night. You can still flip all the television at night and watch TV. You can still FaceTime with your friends and family. But that's only because the attackers chose not to disrupt those activities. That's Congressman James Comer of Kentucky during the congressional hearing.

Now imagine if an adversary had the ability to take our electric grid offline in the dead of winter or the peak of summer. Now imagine if this took place during a national crisis. Imagine if an adversary wanted to toy with our financial markets. Imagine if an adversary had the ability to control supply chains and manipulate whatever they wanted. Think of a scenario where a hacker figures out that the electric grid in a certain portion of the country doesn't have the right security in place for the various systems that manage the power delivery throughout the grid and they figure out a way to overload the grid and essentially shut it down. That is something that they would be looking to do because what it does is it essentially creates chaos, right?

If you look into the world of disaster preparedness, right? There's an entire, you know, FEMA deals with this a lot, but there are entire think tanks and organizations that deal with things like, how do I prepare for X? How do I prepare for a hurricane? How do I prepare for an earthquake? Some of the scenarios, though, are how do I prepare for the electric grid failing? It would create chaos. And that's really what they're after in those type of attacks.

So they're looking for vulnerabilities in critical systems like power, water, and also things like affecting supply chains for food. That's another area that disaster preparedness think tanks and whatnot work on. So if there's a disruption to the supply chain with respect to food to where you halt it, kind of like what happened in Texas where the trucks couldn't move because of the ice on the freeways. You have about 72 hours to recover from that before you get into a situation where there's no food on the shelves anymore. And then people get to the point where they start resorting to violence because they don't have access to food or water or whatever the case is. Right. So, again, it's not an attack for the sake of, you know, just attacking. It's

an attack for the sake of creating chaos. What's the goal? I mean, this is all very apocalyptic, but what's the goal in creating chaos? Like, what do they gain? What's the hacker gain from that? It destabilizes governments. OK. That's what chaos does.

At the end of the day, it destabilizes a government. I always have this vision in my mind of someone hacking. This is ridiculous, but someone hacking all of the traffic lights and making them turn green at the same time. Well, you must have watched the critically acclaimed Hackers movie from the 1990s. Hackers. Wait, that happens in the movie? That happens in the movie. I thought I was the first person who came up with this idea it would be insane but i don't even think that's possible because the traffic light systems are probably not connected you would think but you as i said this story evolves over time where let's say 20 30 40 years ago these systems were all independent and

controlled in small little grids a lot of that stuff is connected to networks nowadays so oh my gosh this is all so crazy yeah i'm gonna go live in a cave right so so jumping back to the the tools that were stolen from fire eye that's just what was stolen from FireEye. As you know, the hackers gained access into many other companies and networks as a result of the SolarWinds breach. So what else did they steal? I'll let you guess. I really have no idea here because I don't know exactly what information is accessible in this situation. I mean, are we talking personal citizen information like social security numbers and tax info and all that that we are afraid of being taken or bank fraud? This breach, to me, from what I can observe, and I was a firsthand victim of it, wasn't about stealing the information of consumers' PII.

This was about stealing documents that were relevant to the collection requirements of another nation. That's Kevin Mendia of FireEye again answering my question. So if it isn't about stealing consumer information, is it more along the lines of proprietary code like owned by a high profile company? Or, I mean, I don't have the slightest clue what information the U.S. government connects to the Internet regarding nuclear weapons we were talking about or top secret national security, you probably know more about that than I do. I do. And while I can't go into a lot of detail on that for obvious reasons, a lot of that information is compartmentalized and not accessible to the generic Internet. That's a relief. You know, we have some people who know better. But ultimately, the most accurate answer to your question is, and I know this is going

to sound like a cop out, but it's that we don't know. Emails and documents are taken. And quite frankly, the people targeted, all that information that was taken, I believe the threat actor is still learning how they can use that information. It's going to emerge over years and it's going to take months and months for organizations to get their arms around all the possible uses of the stolen documents. Man, we don't know a lot of things. And that's the problem here, right? So we know of the specific names of the companies and agencies they hacked, but most of those entities did not disclose what exactly was

accessed. In fact, from watching the hearings, Microsoft was one of the few big tech companies that agreed to appear and answer questions. Well, it appears that other large companies declined. And Brad Smith of Microsoft, who came across to me as being pretty credible and authentic, was asked about the potential of other companies being hacked and maybe not disclosing it. Some of the largest companies in our industry that are well known to have been involved in this that still have not spoken publicly about what they know, there's no indication that they even inform customers. And I'm worried that to some degree, some other companies, some of our competitors even, just didn't look very hard. If you don't look, you won't find and you'll go to bed every night being blissfully ignorant, thinking you don't have a problem when in fact you do.

We know they got access to some of Microsoft's source code. Microsoft says they got access to a small number of repositories. One had to do with Azure, Microsoft's premier cloud offering. Our build systems were secure and they were not penetrated in any way that we had no uh customer data that was uh touched in any way and that we found no evidence that any of our services or products were used as a vector of attack to launch an attack against anyone else what we did find in certain instances was once this intruder was inside a network of say a customer you know say a federal agency one of the things it was able to do was get access to an account that had what we call elevated privileges like an i.t administrator it was able to find the password or get the key

to get into that account when When it was in that account, they found that that individual had access, say, to the Office 365 email of a portion or all of, say, a customer. And so once they were there, then they went into the Office 365 cloud service, and that's when we identified their presence. If the hackers had access to code that specifically dictates who is able to access what resources, what data they can see, whether or not they can download, save that data or extricate it in some other format, that can be a problem. So then that means if somebody had those kind of

tools, they could then control who had access to what. They could actually say, OK, well, John over here is now going to access this level of security and give themselves basically access. Is that what that means? Right. So what's going to happen in this case is they may not have access to those systems directly right now. But if they have access to the code base that handles identity and authorization management

for those systems, they can look for ways to exploit that code and then come up with a vulnerability, which they use to gain access to the system. So this doesn't directly give them access, but what it does is it gives them essentially the code base to look for ways to break it. At this stage, we've seen substantial evidence that points to the Russian Foreign Intelligence Agency,

and we have found no evidence that leads us anywhere else. At Microsoft, as we worked with customers that had been impacted by this, we stepped back and just analyzed all of the engineering steps that we had seen. And we asked ourselves, how many engineers did we believe had worked on this collective effort? And the answer we came to was at least a thousand. I should say at least a thousand very skilled, capable engineers. So we haven't seen this kind of sophistication matched with this kind of scale.

From supersonic jets to personalized medicine, industry leaders are turning to Rescale to power science and engineering breakthroughs. Rescale is a full-stack automation solution for hybrid cloud that helps IT and HPC leaders deliver intelligent computing as a service and enables the enterprise transformation to digital R&D. As a proud sponsor of the enterprise transformation to digital R&D. As a proud sponsor of the Big Compute podcast, Rescale would especially like to say thank you to all the scientists and engineers out there who are working to make a difference for all of us.

Rescale, intelligent computing for digital R&D. Learn more at rescale.com slash BC podcast. Now, keep in mind, there were other enterprises also compromised and various data was stolen from them as well. Considering the 18,000 customers were exploited, we could spend months just combing through the data to see what all was stolen. But we don't have all that time on this podcast episode, so let's focus on the more serious fallout here. So as we mentioned in the last episode, several federal agencies were hacked as a result of the SolarWinds vulnerability. And to me, that's probably the scariest part of all of this. Right. It would have been very concerning if just one minor agency had been hacked.

But that was not the case. Several agencies were hacked, some of those dealing directly with national security. In fact, after our last episode, I decided to do a little more digging into exactly which United States government agencies were compromised. And this is based on articles that I will link to in the episode notes. And it is, again, important to mention that this situation is very fluid. It's still evolving. And this is just what the public has been told so far in this collection of news articles and publications. So first, the Department of Commerce, they had their high ranking officials emails breached. And it seems that the hackers were

probably spying on their email communication. Also, the Department of Defense, as we've mentioned, which includes parts of the Pentagon. The good news here is that apparently the SolarWinds Orion software wasn't particularly popular. So while we don't publicly know the extent of the breach, there's optimism from what I've read that it wasn't incredibly extensive. The Department of Energy was also breached, which includes the National Nuclear Security Administration, which houses the country's nuclear weapons stockpile, which sounds really scary. Now, the good news here in an ongoing investigation is that it seems that the Department of Energy attack was isolated to business networks only and didn't reach their mission essential type national security functions. So that goes back to kind of what you were talking about with your knowledge of government agencies too, Ernest.

And it's hopefully that's true. We hope. Yeah, we hope. Exactly. Also, the Department of Homeland Security. And all I could find was that they had been a part of the breach, but details are not publicly available, which I guess if any agency is going to be tight lipped about it, it makes sense that it would be the Department of Homeland

Security. So yeah, there's there's two right DHS and and DOD are going to be two that would not publicly disclose. Yeah, and that makes sense. And the Department of Justice includes the FBI, the ATF, the Drug Enforcement Administration, among others. And they reported that around 3% of their Microsoft Office email accounts were potentially compromised. But again, some good news here is that they don't believe any classified systems were impacted. So hopefully, again, hopefully that's the case. And then the State Department, they apparently had their email servers hacked by what they're saying looks like the same Russian state hackers that hit them back in 2014. So this could be the same people.

And next, the Department of Treasury, which manages national finances, which includes the IRS. They actually, from what I read, may have been hit relatively hard. And while it appears that taxpayer data weren't breached, the hackers likely stole encryption keys from government servers. Which, correct me if I'm wrong, but I would think that having access to an encryption key would make it possible for a hacker to read and access information that was originally encrypted for the sake of security, right? That's right. But in this case, jokes on them, the country's actually broke. So actually more than broke. It's in the hole by like 20 trillion dollars. They're not getting much out of that. Yep. And then the final U.S. government agency I looked into was the National Institutes of

Health. And this one was really interesting to me, where it's thought that Russia's foreign intelligence service was actually after the coronavirus vaccine research. And that's fascinating because if you remember, Russia started vaccinating people in their country surprisingly early. I'm just saying. That's right. As a matter of fact, when they announced it, I think they called the vaccine Sputnik, if I'm not mistaken. Oh, did they? Yeah, which is a jab at us from a little bit of a Cold War perspective. But the Twitterverse was afire when that happened. And I remember seeing one prominent, let's just say, scientist saying, good luck with that. Interesting. And then in addition to these major agencies, apparently there were local and state governments that were also part of the breach, as well as

critical infrastructure entities and private sector organizations. We've talked about a few of those. So it's definitely wide reaching. It's even being called the biggest U.S. hack in history. Not that we've had the Internet for that long, but still, it's pretty menacing. It's a little bit like a burglar who wants to break into a single apartment, but manages to turn off the alarm system for every home and every building in the entire city. Everybody's safety is put at risk. And that is what we're grappling with here. And as someone who worked directly within the FedGov space in the past, I can tell you that when something like this happens, the assumption is that all networks and all systems are potentially breached. Right. And I want to I want to emphasize that word potentially.

One of the things that concerns me the most in my role as a security person is when a breach happens and the company immediately comes out the same day saying no user data was stolen or nothing was stolen because there's no way you can know that in that amount of time. Right. So in this case, CISA issued an emergency directive to mitigate the SolarWinds hack. Interesting. And for our listeners, CISA is. CISA is the cybersecurity and infrastructure security Agency dealing with all things U.S. cybersecurity. So what that means is anybody who's doing business with the federal government, including the federal government agencies themselves, are bound by this agency. Anything that they put out in terms of cybersecurity, both awareness and preparations and mitigation should a breach or something of that type happen. And the directive issued by CISA is pretty lengthy.

But the TLDR is that pretty extensive measures now need to be taken to secure affected systems and networks. When I read the directive, it was quite shocking because it essentially said that any system or network where the SolarWinds Orion product was used is considered to be compromised. So even if there wasn't direct proof that a customer using Orion was hacked directly, we have to err on the side of caution and assume that they were because sometimes these fingerprints aren't left behind. So yeah, in addition to considering all Orion customers compromised, all of the command and control infrastructure for SolarWinds is to be immediately disconnected or powered down.

Furthermore, we are to treat all hosts monitored by SolarWinds Orion as compromised by threat actors and assume that further persistence mechanisms have been deployed. What this means is that the hackers would have deployed measures to persist the compromise past attempts to remove or contain the hack. When the implant in the SolarWinds software ran, one of the first things it did 11 days after it installed, mind you, it slept for the first 11 days, is it looked at the system it was running on and it looked for common safeguards like Windows Defender, like CrowdStrike, like FireEyes Endpoint, and it shut them off. And again, the implant ran at system level. It had the permissions

to do whatever it needed to do. So it just said, what security is running? Kill it. And that's why we couldn't detect it in the first stage of the attack. So in other words, we need to assume the worst case scenario and act accordingly because these hackers probably put something in place that would protect them even if their hack was discovered. Exactly. So imagine this, right? You get a virus on your computer. Your antivirus sees it and it goes and removes it.

The next time your machine reboots, the virus is back, right? That's a persistence mechanism. So what it does is it conceals itself somewhere. And in the event that it's removed, it's able to reinsert itself. Oh, man, we do have to assume the worst case scenario. Absolutely. And in addition, in the CISA directive, all hosts monitored by the SolarWinds Orion product are to be fully rebuilt using trusted sources. Oh, wow.

Now, this is insane because any enterprise or business using SolarWinds Orion in their infrastructure likely had most, if not all systems connected to it, which meant system administrators would essentially have to rebuild their entire server infrastructure from the ground up. There's no doubt in my mind, this was planned. It was an operation. There was a lot of people involved. And the question really is, where's the next one? And when are we going to find it? Additionally, all credentials used by or stored in SolarWinds software should be considered compromised. Now, you can imagine how much work this caused already for short-staffed and overloaded system administration staff. I imagine a lot of late nights and long hours. Oh, yeah. I've been there. I've done that many

times. While this might seem like an extreme response, when you are dealing with matters of national security, you have to assume the worst in this kind of scenario. Now, I'm curious, obviously, with Intel and Microsoft being affected among probably a number of others also dealing in high-performance computing, how does this Orion hack affect the HPC industry in general? Yeah, I want to call back to a former podcast episode with one of our undercover superheroes, Dan Stanzione, executive director of the Texas Advanced Computing Center, or the TAC, who we interviewed a couple of episodes ago. I really love how he referred to the on-prem supercomputers at the TAC. You know, in many ways, I think our overarching

stance on that is just simply that we are the cloud, but we are a very special kind of cloud. In the HPC world, we still use many of the same types of technology that traditional on-premise enterprise computing or cloud use, namely compute, network, and storage, the big three. The fundamental principles of cybersecurity are the same across technology stacks. At the end of the day, we're still working tirelessly to secure compute resources, network resources, and storage resources. Whether it's HPC in your own data center or HPC access via commercial cloud services, the cybersecurity approach is virtually the same? Right, mostly the same. There are some key areas where we have security opportunities in HPC that

are not as prevalent in the non-HPC world. Okay. One of those is that HPC demands certain performance levels across many different servers simultaneously, and typical security measures we might take in enterprise computing, for example, end-to-end encryption everywhere, may not make sense in the HPC context. Sure, there are ways to offload certain things to hardware when you're talking about standard Ethernet-based networking, but when you're working with InfiniBand,

you're in a whole different ballgame. You're often trading a more hardened security posture enforced in the kernel for a faster connection that is out of band from traditional Ethernet networking. Many of the existing tools that we use in the cybersecurity space were not designed to engage with this type of traffic, and the few tools that do have a significant performance penalty. Does that make HPC harder to hit by a black hat hacker or harder to protect? I would say it's a wash. In some ways, it is harder to protect. But also some of these systems, because they're out of band, they're out of reach of somebody

who doesn't have either hands on or direct access access to the out-of-band method that we're doing it. So this is just one example. My larger point is that while we have some significant differences between HPC and traditional enterprise computing, the majority of the technology is similar. And at the end of the day, any malicious actor seeking to extract data will have to do so via traditional networking channels. That makes it easier to monitor, but at the same time, some attacks do not have the explicit end goal of data extraction. Some attacks just want to lock up resources and cause a denial of service. Some attacks seek to encrypt the critical data and hold it for ransom. I know a little bit about that one, as we heard on our last episode.

Yes, you do. Some attacks seek to exploit hardware firmware vulnerabilities and cause hardware failure like Stuxnet. In the HPC space, we have had several malware waves that attempt to hijack HPC clusters and mine cryptocurrency. Oh, that's so interesting and it makes so much sense. You see people hoarding GPUs to mine cryptocurrency these days on their own personal computers, but if they could just tap into a supercomputer that's maybe full of GPUs, dang, I imagine you could make a lot more money much faster

and, of course, be a much bigger jerk. That's right. And securing these HPC systems is very similar to how we secure traditional enterprise computing systems, with a caveat that we have to do so with little to no performance impact to the infrastructure. You mentioned the cloud, which I wanted to point out was also mentioned quite a few times in the Senate and congressional hearings. I especially found it interesting because I've heard how some organizations have been

hesitant to move to the cloud in the past, and even in the present, in part because maybe they're not convinced that it's secure enough. When by listening to these hearings, both Kevin Mandia and Brad Smith advocated for the opposite and did so stronger than I expected. All 60 of the Microsoft customers who were attacked had their networks penetrated on premise, meaning in their server room, in their building. It was not in our cloud services. It's like if someone broke into your house but not my house, I would not know until you told me. Or this case what they did was they went into your house they found the keys the passwords so that they could go into the service in the cloud once they got that once they stole your keys once they entered our

cloud service we saw them and then we called you and we said did you know that they're in your house did you know that they're in your house? Did you know that they've stolen your keys? He really makes the case that Microsoft's cloud, Azure, wasn't the problem here, but more of a potential solution. And if we're hesitant to take the word of a president of a company that offers cloud services, Kevin Mandia of FireEye actually also didn't disagree. After 30 years in IT security, I believe it'll be easier to secure the cloud than the last 30 years of us trying to secure everybody's home offices and secure inside four different walls all over the place.

And as Congress continued to ask about cloud, Kevin Mandia got more specific. Migration of cloud is going to happen whether we want it to or not. It's rare in history where something costs less and is better. Cloud is actually costing less and is better. For example, if I wanted a server set up at FireEye, I could ask an IT staff to do it, or I can go to an infrastructure as a service provider and get it in five seconds. So the cloud's coming and then you add the pandemic to it and the work from home, all the major enterprises, all the major organizations are going to the cloud's coming and then you add the pandemic to it and the work from home, all the major enterprises, all the major organizations are going to the cloud. The upside is it guts both ways, but you should get better visibility and better controls in the cloud. And the reason why is you're putting all your decentralized IP and value into one place. It's easier to monitor it,

easier to safeguard it. You don't have distributed security controls at that point. I think we're in the middle of the cloud migration, but over time, what we will see is organizations recognizing at least the infrastructure portion of the cloud will be more secure because these companies have to secure it, meaning the providers have to secure it. You know, as we've been talking about this, I can't help but find it a little bit kind of painfully ironic that this digital virus type malware infected so many computers over the exact same time frame that a human virus, the coronavirus that you may have heard of, was spreading over the globe.

It seems like 2020 just wasn't a good year for health, for people or machine. That's right. And one of the things I will remind our listeners here, and I say this all the time, I may sound like a broken record, but hackers often use times or situations where humans are under duress to do what they need to do. For example, at the very beginning of the pandemic, when panic was very high and curves had spiked up really high, there was a surge in social engineering type hacking attempts across the board because they knew they could take advantage of humans already being in a state of duress globally this time, not just in a certain region. That's interesting because everybody's guards down because they're so focused on stress in another area. That's right. And people, you know, if they're working from home, they're increasingly isolated from the rest of their coworkers. Whereas if they had been in an office and something suspicious happened, they could just walk over to the security department and be like, hey, this suspicious thing happened. Is this normal? And they could have quashed it right there. When someone's working from a home, they now have to either

draft an email, go on Slack or something, wait for an asynchronous response. It just turns into a longer process that can be exploited. We may never know the full range and extent of damage, and we may never know the full range and extent as to how this stolen information is benefiting an adversary. At the end of the day, it's up to us to do our due diligence here and perform the correct forensic analysis on all of these systems and ensure that any potentially compromised systems are rebuilt from the ground up. Now, that's both time consuming and expensive, but we can't just hope our systems are ineffective, right? We have to proactively go after this and prove to ourselves and our customers or whoever the other stakeholders are that those systems are in fact unaffected, uncompromised and functioning as they should. That makes sense when the stakes are this high.

That's right. And that's where we're going to leave it for this episode. Well, wait, I still have some questions. So if you don't mind. So we have guidance on what to do to eradicate the effects of this particular hack. But who's to say that it couldn't happen again? Or if something isn't eradicated in a certain place that it spreads out again, right? Like these companies that have been hacked, if they don't get the SolarWinds Orion vaccine, right? What if it could still spread? I guarantee that there's probably some so-called intern out there just waiting to accidentally launch another cyber crisis.

So is there a way to really prevent this sort of thing from happening? Have we learned of maybe a new technology or something that can be put into place? I know that's pretty vague, but... That's a great question and one we will address in the next episode. Oh, we're making this a three-part series. It looks like it. Well, then, in the meantime, I'll avoid clicking email links to doordash.lol. That was a phishing email that came in just a few days ago, and I did not click. Yes, please don't. And for our listeners, if you'd like to help spread the word of the Big Compute podcast like a carefully crafted virus,

you could leave us a five-star review wherever you get your podcasts. A good virus. Is there a good virus out there? Oh, and apparently at the end of our last episode, the please sir may have some more was from Oliver Twist, not Christmas Carol. My husband fact checked me on that one. Well, I'm glad somebody knew because I certainly didn't. Well, thanks to everyone for supporting the Big Compute podcast and stay safe out there. Right. Make sure you use multi-factor authentication and we'll catch you in the next episode.