Big Compute - The SolarWinds Hack: What Happened?

Episode Date: March 2, 2021

It was a dark day in cybersecurity when the world realized that the largest and widest reaching data breach in history had hit over 18,000 companies and organizations, including the U.S. Department of Defense, Microsoft, and just about everything in-between.  In this episode, we take a look at what in the world happened in the SolarWinds hack.  How did it puncture cybersecurity barricades guarding information for some of the world’s most secure organizations?  From SolarWinds to Florida’s recent public water facility hack to a thwarted ransomware attack on one of our own computers, we talk about what appears to be our day’s modern wargrounds -- the internet.

FireEye, contrary to popular belief, is not the same as pink eye. FireEye. That was a good one. Do people think that? No, but your eye is red and it's burning, so I figured it was. Anyway. Hi, everyone. I'm Jolie Hales.

And I'm Ernest DeLeon. And welcome to the Big Compute Podcast. Here we celebrate innovation in a world of virtually unlimited compute, and we do it one important story at a time. We talk about the stories behind scientists and engineers who are embracing the power of high-performance computing to better the lives of all of us. From the products we use every day to the technology of tomorrow, high-performance computing plays a direct role in making it all happen, whether people know it or not. So Jolie. Yeah, what's up? Have you ever been digitally ripped off? Like hacked? Hacked. I have. Can I tell you a story? Yes. Okay. Okay. So a few years ago, I was doing video production for a company in Southern California and I was working on a

computer in their office building. Okay. And at one point I wanted to, I think it was, I wanted to add an image to a video that I was editing. So I did what I always did. Right. You can imagine I navigated to a specific video production network drive, went into the assets folder, and then I double clicked an image so that I could look at it before importing it. But unlike every other time I've done this, instead of the image opening, I got this error message saying that the file format was unsupported and could not be read. Oh, this sounds great. Yeah, okay, it gets even better.

So the file could not be read, which was super weird, right? Since I had literally used this image in the past and I couldn't imagine what would have corrupted it and it was a network drive so it wasn't like it was this you know physical hard drive corruption thing and so I was curious and a bit suspicious and so I tried to open a couple other files in that same folder and again I was given the same error message and the files just wouldn't open. And that was really weird because I had just opened a few of those image files like literally an hour ago and they had worked just fine and now all of a sudden it was almost as though they had become corrupted but I did not know why. That sounds terrible especially considering it was almost as though they had become corrupted, but I did not know why. That sounds terrible, especially considering it was on a network drive.

Yeah, exactly. And since I'm a filmmaker, I've unfortunately lost a lot of work over the years because hard drives have been faulty or the computer has crashed and they weren't backed up properly and all of that. And so when this happens, where something like this happens, I get kind of concerned. And so in this case, or something like this happens, I get kind of concerned. And so in this case, I was determined to get to the bottom of what was going on with these files so that I could then make a plan of what to do about it. Because any filmmaker out there knows losing creative work is the worst. So I backed out of the assets folder and then I took a look at the main drive itself in Windows Explorer

And I scrolled and down at the bottom sitting underneath all the folders Were these three files that I did not recognize And they all had the same like gobbledygook kind of file name But they had different file extensions And as the hyper detail-oriented, organized control freak that I am, I knew that I hadn't put any of those files there, and I was pretty sure no one on my team had done it either. And I noticed that one of these three files was a plain text file. And with the understanding that plain text files can't execute viruses, I ran it through a

quick virus check to be safe anyway, because I'm obviously not a cybersecurity expert, but you know, try to get my ducks in a row. And when it came back clean, I then opened that text file and there staring me in the face were the words, oops, your files have been encrypted and then there were these instructions on how to pay a bunch of bitcoin to get the files restored a beautiful ransomware attack so let me ask you this do you remember what the number of bitcoin was to get them restored i think it was 500 or something 500 american dollars is what it translated to um i i don't remember exactly it wasn't an insane amount it was almost like this was targeted at an individual computer like a not a corporate network drive yes and let and let me tell our users right now never ever pay for

this stuff like it doesn't matter what the consequences are of it. Do not pay for these things. All it does is encourage them to continue doing this. Exactly. So in this text file, I see the words written in there for what I recognized at the time to be a ransomware attack. But I wasn't supposed to notice that text file in order for the attack to work. That text file was actually supposed to help generate the graphical pop-up message that would appear after all the files on the drive had finished being encrypted by the virus. And encrypting every file on a 2TB network drive takes a lot of time. I imagine it probably takes hours. So it hit me at that moment that the ransomware encryption must have

been happening right then. So obviously, yikes, that put my butt into gear and in order to verify how much of the drive was affected thus far I quickly clicked around and checked other familiar files on the drive, and it soon became clear to me that the encryption must have been happening in alphabetical order by folder and file name, which is why the assets folder, which I just happened to be working in

at the time this encryption started, was the first to contain corrupted files, right? Because, you know, going back to grade school, assets obviously starts with A. And then I clicked around a little bit more and I realized that the files inside the assets folder toward the end of the alphabet were still openable at the time, which meant that the encryption must have just barely begun

and had so far apparently only affected a few dozen files out of thousands upon thousands on this drive. And at the time, ransomware attacks were somewhat new. They were just becoming popularized. But I had happened to have heard about them on one of my favorite podcasts, which some of our listeners may know, Radiolab, just a few days before this happened. And I'll include a link to that awesome episode on bigcompute.org, by the way. But because of listening to that podcast, the subject of ransomware was fresh in my mind. And that's why I recognized what was happening so quickly. So at that moment, did the song come in your head, who are you going to call?

No, but it probably should have. And it's funny because all of this happened so fast, right? Like I was able to draw the conclusion within probably 60 seconds just by doing some quick clicks. And obviously, I am not a cybersecurity expert. And so yes, I needed to call for help at that moment. I had no idea, let's be honest, how to stop this encryption process at all. So, and I don't know if this helped at all, but I disconnected my computer from the network just with the physical cable, although I'm pretty convinced the infection came from some other machine in the building. And then I called up our in-house IT team and then I was literally like frantically yelling into the phone. I can just imagine their faces as they're hearing me say,

Get over here now! Russian hackers are encrypting all the files on my drive in hopes of getting a Bitcoin ransom! And they thought I was joking and they're all making fun of me because I guess I'm known for joking around for some weird reason. But eventually I finally convinced them I wasn't kidding. They came over to my office. They worked their magic. They isolated the virus and they stopped the encryption. So what happened to all the files that had already been encrypted? Well, see, that's the thing. So because I was lucky enough to catch the virus early, it had only encrypted the files in folders that started with A. And since that network drive had been backed up 36 hours prior,

all of the files except for three Photoshop files were able to be restored. And I was bummed because I had done a lot of creative work on those particular files. But then, and this is the kicker, the next time I opened up Photoshop, recovered versions of those three exact files popped up because apparently I had had them open on my machine when we abruptly shut it down. So in the end, I legitimately lost zero files and zero Bitcoin, which I consider to be a complete miracle. Let's be honest here, since it never made it to the C folder, none of the cat pictures were taken. You know, I'm more of a dog person,

so the D folder would be more at risk on my personal computer, but it didn't reach the D folder either, so we're lucky. Yeah, and this is the thing with viruses or hacks in general, right? The person perpetrating them obviously has a goal in mind. Who knows what it is?

Sometimes it's something corny like money, in this case, Bitcoin. Other times it's much more nefarious than that. But at the end of the day, these hacks can cause a lot of trouble for a lot of people. Oh, it's so true. I mean, even if the ransomware attack that hit my drive had worked, the consequences would have been minimal when compared to what can happen if like larger or more important systems were hacked. And when I start to think about what hacks really have the potential to do, it's kind of scary.

I mean, it feels like cyberspace is kind of the new war grounds because everything is so digitally interconnected. There's a lot of opportunity to really kind of mess things up. That's right. I mean, just a few days ago, do you remember that public water utility hack in Florida that was caught? I do. An investigation underway after a hacker tried to poison a Tampa Bay area water system. The bad actor increased the amount of sodium hydroxide or lye in the water supply from 100 parts per million to more than 11,000. Sodium hydroxide also known as lye is the main ingredient in liquid drain cleaners. Thankfully, a plant operator noticed

someone on the system making the potentially harmful change in chemicals and reverted the levels before any water was contaminated. Which I imagine could have been incredibly dangerous had the supervisor not been paying attention and reversed it, thank goodness. And people may not know this, but you, Ernest, are a cybersecurity expert. I mean, that's basically your job when you're not recording podcasts. Yeah, I'd hesitate to use the word expert. I mean, that's basically your job when you're not recording podcasts. Yeah, I'd hesitate to use the word expert. I think anybody in my field would, but I'm a practitioner of cybersecurity on a regular basis. Yes. So yeah, you know your stuff when it comes to cybersecurity. And I thought it was crazy because

you had even told me a few days before the water facility hack that you suspected that something exactly like that would happen someday at a public water facility? That's right. And that's mainly because very early in my career, I worked for a public water utility as well. And one of the issues we had was that many of the pumps that are being used that pump water essentially into the water towers were managed and or run with what we call SCADA systems. And those systems at this point, even back then, were very archaic, often ran on very old versions of Windows. Think Windows XP, Windows 3.1. Things that are long since out of support by Microsoft have no security patches being

made for them, and they should never be connected to the internet. However, a lot of companies are connecting these things to networks for essentially convenience reasons. And that is a massive, massive security vulnerability just waiting to be exploited. That's so interesting. And you would think that anything that has to do with our public, that that would be at the forefront of making sure that it wasn't hackable like a public water facility. At least that's what a naive person like me would think. So when you told me that you thought that somebody would try hacking into a public water facility system and then literally a few days later it actually happened, that was enough for me to go out and immediately drop like 200 bucks on a 55 gallon water storage tank. Though it is still sitting in my garage and it doesn't have any

water in it because it's a whole to do to like prepare the water to put in the tank. So it's not really going to do me any good. And yeah, I agree. It's these aging infrastructures, right? You know, you often hear people talk about our aging infrastructure with reference to, you know, bridges that are very old and starting to fall apart or things like that. But we also have very old computing infrastructure or management infrastructure for some of these critical systems like power and water. Why? Is it just because they're government run or government assisted and government doesn't update very quickly? Well, that's actually an interesting question to ask because in some cases they are and in some cases they are not. I think what it comes down to at

the end of the day is public utilities in general are not working on high margins. They have very tight budgets. I gotcha. So updating the software is an expense that doesn't necessarily make it to the top of the list. That's right. Especially when you're you're dealing with priorities. Right. So like as an example at the water utility I worked at, we often had just call them minor disasters where, you know, a water main would break under a street or something like that. And these things routinely happened because the infrastructure was old. So a bunch of those add up in a year and you've essentially used up all of your budget. And it's not like you can just make money up, right? You either have to raise rates to the rate payers, which they're not going to be happy with, or you try to use some

other financing tool to get it done. So yeah, at the end of the day, you're right. It's that the cybersecurity aspect of a lot of these things are lower on the totem pole than they probably should be. That's very interesting. Very eye-opening too. I mean, it's crazy to see how much cybersecurity threats really are real, even to our public life more and more, right? We saw the giant Twitter hack that was done by some kid in Florida, right? Yep. Several high profile Twitter accounts appear to have been hacked.

The accounts include Jeff Bezos, Bill Gates, Elon Musk, Uber, and Apple. Quote, due to COVID-19, we are giving back over $10 million in Bitcoin. All payments sent to our address below will be sent back doubled. They then post a Bitcoin address and say that this

is only going to be going on for the next 30 minutes. NBC News reporting that over 250 transactions have already been sent. If that had been a very organized effort with some really sinister intentions, that could have been something really bad. And that's just Twitter. Yes. And one of the things that we'll point out is that there are many areas that are prone to attack or vulnerable in terms of cybersecurity. But the weakest link in any of these is always the human being. And there's a specific, if you want to call it, area of study within cybersecurity that has to do with what we call social engineering. And that is the art of conning or... Tricking somebody. Right. Tricking somebody into giving you elevated access to a system that you should not have access to. Like even with the Twitter hacks, I remember it was incredibly interesting to learn that they literally made phone calls in order to get verbal information transferred to them

that they then used for the hack. I had never considered the fact that it was like person to person verbal communication in some instances when it comes to hacking. I thought everything had to do with, you know, behind the scenes code. But you're right. It's that human element. And that's true. And, you know, you bring up the Twitter hack and there have been quite a few high profile

ones lately and some of them have actually hit the news. So Jolie, tell me, what do you know about the SolarWinds hack? Oh, SolarWinds. So I don't know very much about it. I know that there was a hack and I know that it was apparently humongous, like absolutely massive. And it was relatively recently discovered, I want to say a couple months ago, but I really don't know much more than that. And I've always been curious. Yeah, it's a pretty big hack and there's a lot to cover. So it's going to take a few episodes. There's no way to cram it into one. Totally fine. Worth it. It is. So many moons ago, I mentioned this earlier, I worked at a public water utility

company. Just like the one in Florida that was hacked. That's right. And when you work in security in these type of areas, you know where your vulnerable points are. So this is kind of why when we talk about the one in Florida, when I see certain patterns happening, it immediately brings to mind the SCADA systems. Well, you definitely predicted it to the point where I was like, man, if I didn't know Ernest really well, I would have thought he did it. Yeah. So at this public utility company, we used SolarWinds to manage our static IP addresses. I don't remember the name of the product. This was a very long time ago, right? Before all the acquisitions and new products that comprise the portfolio of products that SolarWinds offers today. So when you say SolarWinds, just want to back up for anybody who isn't really familiar.

It sounds like the SolarWinds company is a software company. Sure. So SolarWinds basically creates IT infrastructure management software. They're based out of Austin, Texas, and they have a few thousand employees. OK, so so not like a solar company, not a wind energy company and not a company that has anything to do with streams of charged particles coming from the sun. SolarWinds is an IT software company. Got it. Correct. Which, by the way, you and our listeners know I'm dying to talk about an actual solar wind and why that probably wasn't the best name for a company.

But I'm not going to do that here. We'll save that for one of our future space episodes. Ooh. Anyway, when I was working for the public water utility company, we used this basic product from SolarWinds to manage IP addresses. It had the ability to scan the network and tell us which static IP addresses were in use at any given time and what the host names were for the machines at those IPs. That's about all it did. It was a very simple product that really didn't do much that Microsoft domain services wouldn't

eventually usurp entirely in the enterprise. Okay, interesting. So you've had some hands-on experience with SolarWinds in its earlier days. Right. Again, a very simple product when they were probably a very small company. Fast forward to December 2020 when I was working in high-performance computing cybersecurity, right? We in the cybersecurity industry suddenly had a bombshell dropped on us. We had long expected something of this scale to happen, but we had no idea it would be so soon. And it all started with FireEye. Wait, what's FireEye? So FireEye is actually a pretty large and pretty relevant cybersecurity company. They basically detect and prevent major cyber attacks for companies. For instance, they've been called in to investigate a lot of the high-profile attacks we've heard of, like with Target and Sony Pictures.

You mean like when North Korea retaliated after Sony made a movie they didn't like? The North Korean regime has called the movie terrorism. So on December 8th, just a few months ago, FireEye announced that they had been hacked. Uh-oh. Yeah. And many of the tools they use for white hat hacking had been stolen. And in case you don't know what white hat hacking is, it's the term used for hackers who basically choose to use their hacking powers for good rather than evil.

A lot of times they're security specialists who attempt to find security holes by hacking a system. So white hat hackers are like ethical hackers? Exactly. In the industry, we kind of have three main groups, if you will. We have white hat hackers, black hat hackers, and gray hat hackers. And it's exactly what it sounds like the white hats are the ethical hackers doing it for good trying to secure things the black hat hackers are doing it for nefarious purposes um trying to steal things sometimes just to create chaos a lot of times they don't even want money they just want to sow discord they just want to be jerks yes and the gray hats are in the middle. They waver from one side to the other, depending on, you know, however they're feeling that day.

Huh. I like the idea of the white hat hackers. I feel like they're definitely undercover superheroes. Now, white hat hackers use the same, typically the same hacking methods and tools as those who hack for sinister reasons, which we talked about a second ago. We call them black hat hackers. But white hat hackers typically have permission from the system owner first, which makes the process legal. Oh, okay. So these ethical hackers test the possibilities of hacking company systems and basically perform vulnerability assessments that they then turn over to those companies to help the companies secure their systems better. And in other cases, big companies like Apple, Google, Microsoft have what they call bug bounties.

Oh, I've heard of this. And so these hackers will often find vulnerabilities and then submit them into the bug bounty program. And if it gets accepted as a valid vulnerability, they get paid a certain amount of money based on the type of vulnerability it is. That's brilliant. It really encourages white hat hacking. Yes, it tries to take what are traditionally black hat hackers or gray hat and on the type of vulnerability it is. That's brilliant. It really encourages white hat hacking. Yes. It tries to take what are traditionally black hat hackers or gray hat and turn them into white hat by giving them money directly as opposed to- Through some sinister method. Right. Because typically the black hat hackers are not doing it for themselves. Really? Not all of them, right? But typically they're doing it and then trying to sell whatever

hacks they come up with to nation states or whoever's paying the highest dollar, to be honest, for that vulnerability. Wow, that's interesting. So then FireEye is a cybersecurity company that basically employs a fleet of white hat hackers. Yes, that's correct. And just to make sure I understand, the black hat hackers are what we normally think of when we think of that typical computer hacker. And on the Internet, they're always depicted in pictures as being these faceless dudes in black hoodies hunched over a computer in a dark room. Yes, that's absolutely right. As a matter of fact, I'm going to I'm going to interject something here. If our listeners have never watched the movie Hackers from the 1990s, the one that had Angelina Jolie in it, you have to watched the movie hackers from the 1990s the one that had angelina jolie in it you have to see this movie it is another one of those that is so bad it's great well it's great a hacking movie made in the 1990s i mean how well could that hold up hidden beneath the world

we know is the world they inhabit day yeah mom Yeah, Mom? What are you doing? I'm taking over a TV network. Finish up, honey, and get to sleep. They're hackers. But this time... Come here, look at this. It's some kind of virus. Unless $5 million is transferred to the following account,

I will capsize five oil tankers. They just hacked the wrong guy. Game's over. It's amazing. If you're a fan of you know bad movies that are so bad they're good hackers so back to this story so it's it's december 2020 and this cyber security company fire eye announces that they've just discovered that they've been hacked right so they essentially

sound the alarm and they start digging into the details of the hack. How someone could have breached their systems being that they're super vigilant and always looking at this kind of stuff. Yeah, I would imagine. Yeah. And what kind of data they had access and for how long. Big news in the tech world this week as FireEye, a top cybersecurity firm based out of Silicon Valley, announced that hackers made off with tools that the company says can be used to mount future attacks. At that time, those of us in the cybersecurity industry thought FireEye was the sole target of the attack. But we also know that anytime the tools of the trade are stolen, meaning the tools that are used to hack and or information or data about known vulnerabilities that are used for hack, anytime that stuff is stolen, we have to assume that those will be used

to hack other entities in the immediate future. So in other words, black hat hackers usually hack one system in order to access another system and so forth. So you have to remain consistently suspicious. Yes, that is one of the things they do, right? Like in a large environment, they'll have to hack one system to get access and elevate access. And then from there, they look for other systems. However, in some cases, they go after whatever system they want to start with. And in that case, they only have to get one. But that's not usually how it happens. It's usually a progressive thing where they find a backdoor somewhere, get in, and then figure out how they're going to spread out from there.

Okay. In the sales lingo, we often call that land and expand. That's exactly what they're doing, but from a nefarious perspective. Interesting. Okay. Yeah. So it turns out that while FireEye was definitely targeted for the wealth of security tools that could be used for nefarious purposes, they were not the only target.

Enter our friends at SolarWinds. So as the FireEye team investigated, they discovered that the breach had originated from their use of SolarWinds. But the question was how? SolarWinds has a product called Orion. I know, the space puns are all over the place today. Anyway, Orion is a platform used by tens of thousands of companies to manage their information technology resources. It's not only common, but fairly ubiquitous for

software companies to push updates to their systems, often to enable new features or patch known security vulnerabilities. Ironically, in this case, it was this update mechanism that enabled the proliferation of the hack. Okay, so let me get this straight. So FireEye is a cybersecurity company that uses SolarWinds IT management software in their business. Right. And SolarWinds pushed out a patch or some kind of update for their Orion software that was used by FireEye, and they were completely unaware that it had malware attached to it until FireEye noticed something suspicious and made the discovery.

That's right. Someone hacked into SolarWinds themselves and crafted a malicious software update package that was then injected into the update server from which SolarWinds used to distribute updates to the customers. And then that malware ended up breaching the systems of, I mean, I would imagine just about any company that used the Orion product. Exactly. Dang, that's pretty scary. And it's even scarier how the hack first happened. Uh-oh.

Okay, now you have to tell us. I will after the break. Oh, now it's your turn to leave us hanging. Gotta get paid. From supersonic jets to personalized medicine, industry leaders are turning to Rescale to power science and engineering breakthroughs. Rescale is a full-stack automation solution for hybrid cloud that helps IT and HPC leaders deliver intelligent computing as a service and enables the enterprise transformation to digital R&D. As a proud sponsor of the Big Compute podcast, Rescale would especially like to say thank you to all the scientists and engineers out there who are working to make a difference for all of us.

Rescale. Intelligent computing for digital R&D. Learn more at rescale.com slash bcpodcast. Okay, back to the hack. So every SolarWinds customer who used the Orion product was now compromised by doing something they should have been doing, which is updating software. The beauty of the hack, if you want to call it that, was in the distribution method chosen. They didn't try to hack a web server and upload a compromised installer file or anything so mundane. They found an opportunity to attack the update

supply chain and thus remain completely under the radar. That is so crazy. So we always think, I know that I do at least, I'm always thinking that software updates are a way to protect ourselves from security threats, right? Whenever there's an update that comes out for my phone or my computer or, you know, my operating system, I'm always getting that update installed immediately. So whoever these attackers were, it's like they used our own psychology or something against us. Right. And I'd love to give them the credit to say they did that. But I think as we'll find out later, it was more of a crime of opportunity. Okay. So because Orion was used to manage and in many cases update the many IT systems of SolarWinds customers, the attackers

now had a vector to install even more malware on these customer systems without their knowledge. This was truly an inception level hack. So how many customers were actually affected by this? So the truth there is that we don't know. Wait, really? Not even a little bit? Well, sort of, but not really. So SolarWinds claims that as many as 18,000 customers install the compromised updates, many of those being Fortune 500 companies and several agencies of our own U.S. federal government. 18,000 customers. So we're talking 18,000 different companies, not 18,000 individuals, right? That's correct.

Oh, my gosh. that is so bad. It is. Now, from what I've gathered from various media reports and through my own sources in the cybersecurity community, we know that FireEye, Microsoft, Cisco, Intel, and a few other tech companies were among those attacked. Oh, they're big names. Big names. We also know that the Department of Energy, the Department of the Treasury, the Department of Homeland Security, the Department of Defense, and even the National Nuclear Security Administration were all hacked as well.

Even if you don't live in the United States, the reach is global. Yes. The reach of the attack is stunning. That's crazy. And not just in the high-performance computing industry, which we're focused on, but pretty much everyone in every industry. That's right. So the worst part of this that we know of is that the hack was not discovered immediately. Oh boy.

It's thought right now, again, because the story is evolving, that the purpose of this hack was to do reconnaissance on all of these companies, agencies and systems and to prepare for a future cyber attack. But we don't know that for sure. As far as we know, this attack was in place for months before it was discovered. Dang. I mean, if it's been in place for months, who knows how much information they had access to in that time. Right. We're still operating on a lot of hypotheticals here, and it will take months more to figure out the full extent of this hack, it'll probably take years to ensure that all the possible compromised systems have been rebuilt from the ground up and all the networks purged of possible malware. So when you say rebuilding all possible compromised

systems from the ground up, what do you mean exactly? Are we talking fresh installs and updates or completely new software development or something completely beyond my scope of cybersecurity understanding? No, you're absolutely right. We're talking about fresh installs and updates, decommissioning of systems that were potentially compromised, what types of systems you should consider compromised, whether or not you know they are, and just kind of a general operating guide as to what to do. And you know SZA obviously sets policy for the entire federal government and many private industries follow it as well just as a precaution. So we will get into that in the next episode but yes absolutely that's

exactly what we're talking about. Interesting that's got to be a big job. Yeah so now we get to the two questions that our listeners have probably been asking this entire time. Who did this and how did they manage to hack the update system of SolarWinds? That's exactly what I was thinking. Yeah, because that's once you've built up a story like this. The first question in my mind is, OK, who pulled this off? Right. And then how does it affect me? And then we are. What's the fault for me? So to the first question, we do not have definite proof of who did it.

We do, however, have a lot of indicators, things we call fingerprints, that point in the direction of Russia's foreign intelligence service. Experts believe Russia was behind the hack of a company called SolarWinds, sending malware to 18,000 private and government organizations. Russia has denied any involvement in the hack. Dang it, Russia! I swear they're the same people who put that ransomware on my computer, though probably not the actual Foreign Intelligence Service, probably just some dude in an apartment.

Perhaps. Now, you may be familiar with the names cozy bear or fancy bear these are terms that are used for units believing to be operating out of russia at the behest of the russian government how dare they defy such cute and cuddly names now you might be thinking to yourself if this hack was this massive and this pervasive in nature it must have been an epic undertaking right i mean you would think so it immediately starts bringing things to mind like Mission Impossible and the kind of, you know, dropping in from the ceiling through a vent that had lasers looking for you and hacking some mainframe. Yeah, wearing sunglasses indoors. Right. Something where the perpetrators had to find the right window at the right time and exploit a known zero day vulnerability

to gain access to one system within the SolarWinds network, then work their way around to the update server to inject their compromised update software package. Yeah. Sounds reasonable given the breadth of the hack, right? I mean, I don't know much about hacking, but I would assume it would be an intricate and complicated process, yes. Well, as is often the case in life, the reality of the situation is far less sensational than it might seem. Really? You see, SolarWinds was using an FTP server to host the update files that its customers used to update their systems automatically via Orion.

And our users out there, FTP is file transfer protocol. It's an archaic protocol that people should not be using anymore for just about any. I realize that there are some very unique edge cases where you might use it, but please stop using FTP if you're using it. I was going to say I use that maybe 15 years ago. Yeah. Haven't seen it since. There's a reason why. So in 2019, a security researcher notified SolarWinds that the password to this FTP server had been leaked on

GitHub in plain text. GitHub? There's a lot of people on GitHub. This isn't like some underground dark web. That's right. And mind you, this was a public GitHub repo, which means anyone could see this password, not just SolarWinds employees. Oh my gosh. But that isn't the best part of the story. Even if it had not been leaked can you guess what the password was please tell me it wasn't password no it wasn't but it's pretty close oh no the password was actually solar winds one two three it was Yes, ladies and gentlemen, the password was the name of the company followed by the numbers 123. And that password led to a ginormous hack of major companies and agencies, including the Department of Homeland Security and the Department of Defense. That is absolutely correct.

Oh, my gosh. Some of the most secure agency networks on the planet were hacked by way of a password that was SolarWinds123. That is so incredibly insane. And of course, this makes me think up so many other questions like what kind of information did the hackers get away with? And what does that mean for Americans or people in other countries? I mean, what does that mean for Americans or people in other countries? I mean, what does that mean for us in high performance computing? You mentioned Microsoft, which is a sponsor of the Big Compute Conference. I mean, what can this information be used for? Was the plot for a larger cybersecurity attack simply thwarted? I mean, do we know the answers to these questions?

Or could some nation like Russia or China now have a bunch of information they could use to hurt other countries? I mean, do we have any idea what the hacker's intent might have been? I know this is a thousand questions, but if I have them, I imagine our listeners also have them. Ernest, please guide us down this path. We need to know. So I want to explore all of those answers with you, but we've already run out of time for this episode, so you'll have to ruminate until the next one. Oh, man. Talk about a cliffhanger. Everything in everyone's digital life has been compromised.

Tune in next time to learn more about the cyber apocalypse. That's like what it feels like. meantime you can help us spread the word of the big compute podcast by leaving us a five-star review on apple podcasts google podcasts spotify or wherever else you listen and you can check out bigcompute.org for more information about the solar winds hack i like that transition it's like we go from cyberpocalypse to please leave us a review five stars please leave us more more uh what's that that uh scrooge the uncle scrooge thing where he says please more food or what is it oh please sir may i have some more yeah no like like in the are you talking about a christmas carol yeah and like disney christmas carol yeah no wait is it that no that's God bless us, everyone. I don't know.

No, it's the one with Uncle Scrooge. Please, sir, can I have some more? That one? That one. That one. And it's like, no, you got to go leave us a review first, kid. Yeah, you got to leave us reviews and then we'll continue with our disparagement of large corporations and government agencies. Well, this is definitely a cliffhanger.

I'm excited to learn more. So we got to get that recorded ASAP. Thank you all for listening and we'll catch you in the next episode. Stay cyber safe. Thank you.