Hacked - CrowdStrike Incident

Episode Date: August 2, 2024

We all just watched one of the largest IT events in years unfold in real time with the CrowdStrike incident. We wanted to understand it better, so we called up security researcher and educator John Hammond to get to the bottom of it.  Learn more about your ad choices. Visit podcastchoices.com/adchoices

You got some folks thinking, uh-oh, was this a cyber security attack? Is this a vulnerability or an exploit and a hack? Well, no. We'll be straight up and up front with that. I think CrowdStrike has gone on to say, unfortunately, this just was an accident. It was a mistake. Before we get to CrowdStrike and the biggest IT incident in a very long time,

let's talk about a big old weird building. So there's this tower in New York called 33 Thomas. Street. It's this tall, brutalist granite facade tower. It's built in the 1960s and 70s. And the thing you notice after a second of looking at it is that other than the ground level front entrance, this almost 30-story building, has no windows. It's just this looming 500-foot plus tower made out of stone from sidewalk to sky. Today, most folks say it's the location of an NSA mass surveillance hub called Titan Point, which is a great name for an imposing building with no windows where spies go to work. But for most of its history, 33 Thomas had a different role and went by a different name.

The AT&T Long Lines Building, as it was then known, was a telephone switching hub. Long distance phone connections ran over copper wire and the Long Lines building housed these massive switching stations that connected all the traffic. And this imposing design kind of makes sense in this context. Switching machines and copper cables don't care about natural light and it's easier to manage the temperature in a building without a lot of glass. Long Lines was the switching hub in the eastern U.S. Pretty much all communications running up and down the East Coast went through this one building. It was also right next to the termination point for the primary transatlantic cable connecting the eastern seaboard to Europe. So the Long Lines building was really important. It was

the hub for telecommunications in the eastern U.S. and for transatlantic communication to Europe. Or put another way, it was a single point of failure. And on September 17th, 1991, it failed. At the same time, a technical malfunction involving power equipment, a human error, and a poorly timed request from the electrical company all lined up to disable AT&T's central switch housed inside of long lines. And in an instant, more than 5 million calls are dropped. Communication across the eastern U.S. goes dark. And because the incident also took down the Federal Aviation Administration's private lines, air traffic control to 398 airports went offline.

Planes were grounded around the world because one building lost power. Sometimes you don't realize something is a big point of failure for a lot of important stuff, until it fails, which brings us to CrowdStrike. This is an NBC News special report. Good morning, good to see you. You are coming on the air right now with breaking news, a massive global technical outage tied to CrowdStrike, which is a major cybersecurity provider,

has knocked critical computer infrastructure offline all across the country and, in fact, all around the world. I hope everyone who works in IT who listens to this is doing okay, because it's been a really bad time since this happened. We'll get into this in my conversation with John, but for anyone who doesn't know, CrowdStrike is a large American cybersecurity company. They're used by some of the largest companies in the world.

And on July 19th, I'll just read the CrowdStrike press statement. Quote, CrowdStrike released a sensor configuration update to Windows system. Sensor configuration updates are an ongoing part of the protection mechanism of the Falcon platform. This configuration update triggered a logic error resulting in a system crash and blue screen of death. on impacted systems.

Basically, CrowdStrike pushed out an update that caused a lot of very important computers to crash. CrowdStrike is a security company. Falcon is a platform of theirs for corporate clients. And it seems like what happened is a bug in the configuration and a bug in the system for catching bugs in the configuration kind of lined up with each other. And this bad update was able to shoot through to a lot of people. We've talked about this before, but it's almost like the holes in Slash,

slices of Swiss cheese lining up and something being able to drop through. As I understand it, the Falcon platform has kernel level access. The Windows blue screen, a system crash is what happens when the system isn't confident it can continue operating safely. Anything dodgy at the kernel level flips that switch. So this update goes out and a bunch of computers go, ah, something's wrong, and just crash themselves out to be safe. Computers in airports and in hospitals and businesses.

around the world. And the result, I heard someone say, this is kind of close to what we thought Y2K was going to be. Well, the result is a 41% decline in CrowdStragg's stock price in the last month.

That's a big chunk of the results. That's pretty rough. Poor guys, feel bad for them. I did read a bit about how the bug managed to escape the quality assurance testing. And it seems, like the software passed all of its tests, but the software loads essentially configs or filters

from ancillary files. And that's where the issue was, was in one of the library files, essentially, that doesn't get as rigorous testing as the actual application itself. So the application was actually working fine. It's just that when it loaded some bad content and caused a caused a bug. So at least that's the best that I could decipher through all of the encoded messages coming out from CrowdStrike. Yeah, I regularly find myself in over my head

with stories we tell and having to sort of claw my way up through it to be able to talk about it. And this was like, I was very glad to have had this conversation in the immediate aftermath and then a good week to try and wrap my brain around it because it certainly required it. In the intervening days, there's been a lot of interesting conversations.

also as people get things back up and running to do with that kernel level access. Since this incident, Microsoft has publicly hinted at starting to limit kernel access to cybersecurity vendors. This is, of course, a double-edged sword, as we discussed with John. I found that very fascinating because that wasn't really where the conversation started, but it's certainly where it's gone in the intervening days. Well, the thing for me is, like, you can't take away the security provider's access to the kernel because the the security attackers are going to be trying to take like kernel level access. So you really get into a situation where if the good guys don't have access to that much power,

but the bad people do, then how do you stop them? Like the technically the malware that has kernel level access will have more control than the prevention system. So the malware can just take it out. Like it can manipulate it. it's it's like I don't know how much you know about kernels are you a colonel guy big colonel guy build your own kernels for your unix operating systems you know not not in a while yeah so the the one thing with kernels is like the kind of as the the heart of the computer or the heart of

the operating system you know they interface all the hardware they do a lot of the like super low level stuff happens in the kernel and the entire speed of the computer relies on the efficiency of the kernel. So the code's very, you know, thin. There's not a lot of air handling. There's not a lot of, you know, debug modes. There's not a lot of stuff like that in the kernel because you want it to literally operate as fast as possible.

Right. Which is why when something happens that it shouldn't, you get a blue screen rather than like, there was an exception in your kernel, you know, talk to the vendors manufacturer about something. It just turns off the computer, essentially. So the, yeah, the kernel piece is interesting because, like, we talked about kernel access in the video game hacks. It's like everything these days seems to want kernel access, whether it's good or bad.

So it's like at what point are we going to make a multi-tier kernel? where it's like, you know, the core operating system gets level zero access, you know, start scaling up from that because the reality is, we're going to end up, we're going down the path of ending up where like almost everything on your computer will have kernel access. Yeah, I definitely get the idea that a cybersecurity vendor might need that sort of like ground level access because if a compromise happens at that level, you need a security vendor to be able to prevent it. There's a logic to that.

Then there's all the stuff in the middle of like also call of duty really doesn't want cheaters. So they like access to it. It's like, those feel different. But the problem is is that the cheat, like because kernel access is controlled by the end user largely in corporate enterprise is different. I can access. Like if a piece of software that I install asks for admin access, I can just click the yes button without thinking twice. and I've just given kernel level access to some random piece of software. So it's like, it becomes, I don't know, yeah.

It is very different and it's going to be very challenging, especially for companies like CrowdStrike. To me, one of the other interesting things was the single point of failure thing about the Long Lines building. You know, we've created an ecosystem with our computing. not just with CrowdStrike, but like, you know, let's just talk about Windows. It's like if anything, like Windows updates roll out all the time. I'm glad they have like an insider program now, which is essentially like a beta test QA program for their updates.

But the, but we're in a situation now where one large technical issue gets pushed out. Like the, I think the beauty of this one, I'm kind of jumping around here. So I apologize for that. But the beauty, the thing that made this kind of okay. is that CrowdStrike only existed on essentially large enterprise computers, which means that they have large enterprise IT infrastructures and staffing. If this had been a straight up Windows update and had broken every mother, father, kids, students, laptop, PC at home, like, could you imagine, like, people that don't have IT access

and don't have staff and bodies to come around and help. solve this problem. Could you imagine the headache this would have created for society of everybody's computers stopped working? Yeah, I was reflecting on how this is. We talk about that in the conversation that this product, thank God, is only used by corporate clients. It's not good that airlines and hospitals and businesses were having to deal with this, but they employ dedicated people who are knowledgeable and on it to be able to try and at least start responding. Exactly. I had even really considered what it would have looked like if this had gone wide. Well, just like imagine the Best Buy geek squad.

Just if you live in a town with two and a half million people and two point four million computers and if like I don't know what Windows market share is these days like let's call it 70, 80 percent. 80% of those computers go down. And the only way to fix them is a manual bypass at the file system level. There's no automated update that you can push out to a bunch of computers that aren't booting. So like every single one of them would have need to have been booted into safe mode, had the file system manipulated, et cetera, et cetera. And it would like it would have been this at some point in our life,

I foresee something like this is going to happen on the broad scale. and it's going to be truly catastrophic from like a economic sense. The fallout of this publicly has been pretty crazy so far. Yeah, and it's honestly only going to get crazier. Like just yesterday, Delta came out. They've already employed a major law firm to seek out or seek lost revenue. Oh, wow. So they're estimated, they've done a preliminary estimation,

and they say that the cost of the outage was over a half a billion. So given the crowd strike does a proximate. approximately, what is it, like $4 billion in revenue? $3 billion in revenue? I think it's like $4 billion in revenue-ish. If one client lost a half a billion, you know, if this becomes a class action, like there's the insurance world for software that's rolled out like this. Like, could you imagine what's about to happen?

these companies are about to be uninsurable. Because the lawsuits, like a couple of day outage for 8. What was it, 8.5 million computers is going to be like billions and billions and billions of dollars. Like no insurance company wants to be carrying the bag for that. No, especially if those lawsuits work. Yeah. So it's going to be really interesting to watch the fallout in the courts because if they are found to be liable and it does stick, then I don't know, it's going to be very risky for software companies going

forward. I'm sure Microsoft will have a big vested interest in these lawsuits, making sure, just given that they are such a widely adopted platform, that if at some point in the future they do this, like push out an update that breaks things, they won't want to be held financially responsible for every corporate computer in the world. And the lost revenue and efficient see, so. Interesting. Yeah, the fallout's been mad. I wanted to understand what was going on a little better. So I called up John Hammond, who's been covering this closely. John is a principal security researcher at Huntress, but he's also a public figure and educator and security. And he was talking to a lot of people directly responding to the immediate aftermath of this. So in the days

right after the incident, you know, I wanted to know what he was hearing. So I called him up. He was very generous with his time. This is my conversation with security researcher John Hammond about the CrowdStrike incident here on Hacked. John, thank you so much for sitting down and talking with me about this. Hey, thanks so much. Super happy to be here. Before we get to the incident itself, what is CrowdStrike and kind of what is its role for anyone who's unfamiliar with it? Oh, so for folks not tracking CrowdStrike, they are a very big name in the cybersecurity

provider space, like a vendor, a company that wants to offer protection for you, for your company, for your business and organization. and that is looking for malware, trying to stop in its tracks, trying to get out in front of hackers, and layer on defense and make sure that you, your computers, your devices, infrastructure, and environment is safe and secure. July 19th, 2024, what happened here? Take me through the story of this update incident, I'll call it. Goodness.

Well, if I may, I think I'll start to see the kindling flame, really, even as far back to like 10 p.m. Pacific as when I started to see this thing catch fire. on July 18th because there were some chatter over on Reddit, you know, the online forum, subreddit for CrowdStrike, and someone had posted, a user had said, hey, is anyone else seeing an outage or like a handful of blue screen of death boot loops for their computers seemingly running the CrowdStrike agent? And that just opened up the floodgates. You can see, hey, users chiming in, responses in the thread saying, yep, I've got an environment with, I don't know, 500 servers, maybe 1,000, maybe 50,000 endpoints, works.

and all of them are stuck and unable to finish booting and they get online. And then folks are saying, yep, I'm working at a bank and this is working me throughout the day. I'm on call and got to chase this thing. And folks are saying, hey, it's happening at this airport. Airlines are down. And then folks are saying I'm at a hospital. Folks are saying I'm at school. It's just suddenly folks coming out of the woodwork and you start to see this catch like wildfire.

Uh-oh, mass IT outage from unfortunately this crowd strike agent. their falcon sensor. So people start to realize, okay, stuff isn't booting up as it's supposed to be then what happens? What's the next step in us figuring out maybe just how big a deal this actually is? Well, right then and there, I think honestly about 10 or 20 minutes following when I see that post, there is a representative from CrowdStrike that chimes in and says, hey, we're aware, we see this, we know things are happening and we've posted a, I think it's a technical advisory TA. I'll admit, I don't know the acronym they call it. I'll be the first to admit, I'm not a CrowdStrike customer.

I don't tend to use CrowdStrike. But trying to chase this, trying to track it the best I can. And that had some of the information behind their login portal, behind their support resource. So tough to get our hands on that if you weren't a customer. But then folks try to chime in and start thinking, oh, there's a workaround. Because we're seeing this blue screen of death, oh, that means a computer crash. That means like a kernel panic equivalent. And, well, it at least gives you some.

debug and troubleshooting information that this came from file csagent.sys. And for folks that might be familiar, that's a kernel driver, some software that runs at the very low level, like raw core and roots of the operating system or a computer. There's a lot of back and forth on, okay, the real culprit here. We'll talk about some of the channel files soon and how that's really playing apart. And now it's not strictly the kernel driver. But I digress. Some folks are saying, look, if we just rename.

the folder, then it won't load that. And okay, we could go along our merry way. It'll boot up and start things naturally. But CrowdStrike chimes in and says, no, hang on, we should clean up the channel files. Get a little bit more accurate, a little bit more precise, and how we can recover, remediate from this. The problem is that's not something you could do kind of at scale in an automated way, especially because, well, all the computers are stuck. They basically won't turn on. So unfortunately that Friday night and now through the weekend and I don't know I'll admit if it's a week if it's a month how long this takes to fully recover but imagine a lot of technicians and engineers just running around trying to manually tweak and correct and fix each individual computer workstation

and server and that is what was made for quite a nightmare scenario here yeah it seems like it's been a pretty gnarly call it three days since this has happened at the time of recording for anyone in any kind of an IT position. As I said, it's been two, two, three days when we're having this conversation. So far, who has been impacted by this? What kind of stories have you heard in terms of the fallout of this incident?

Oh, well, I will say I uploaded a video on YouTube, and I know that's silly, but I try to chase some other YouTube activity to, I don't know, help spread the word and get some cybersecurity education and awareness out. And that has had a wild outpouring in the comments of, hey, I'm a faithful. affected, again, whether it's a school, bank, airline, et cetera, someone had said, look, I had a family

member that needed to go to the hospital, and now they've had to move him or do some patient relocation. Really tough stuff. I guess I don't have anything off the cuff more specific, but I have no doubt, and my heart goes out to the folks affected. Yeah, I was seeing a couple of snapshots of big wide shots of airport terminals just full of more people than should have been hanging out in those airport terminals, just flights not taking off. There are some crazy videos of San Francisco Airport, Los Angeles International, and you can see folks just lined up trying to zoom in with their phone camera, their phone camera to look at the TV screens and monitors that all have the blue screen

and frowny face. But it's wild, I think, to see how, you know, technology and computers and cybersecurity, even, you don't ever think of that for maybe the layman or normal folks, just how well it affects our world. You got some folks thinking about, uh-oh, was this a cybersecurity attack? Is this a vulnerability or an exploit and a hack? Well, no. We'll be straight up and up front with that.

I think Crowdstrike has gone on to say, unfortunately, this just was an accident. It was a mistake. Yeah, I think I saw you use the word whoopsie somewhere. Yeah, I tried to, well, I'm understanding that's not something to be so jovial and, I don't know, watered down. but I got to think, you know, I work at Huntress, my day job, and Huntress is just as well a cybersecurity provider and vendor for managed security, stopping hackers and malware, yada, yada, yada.

We work with a kernel driver because that is a necessity for a lot of the protection that we want to provide. We'll look into memory. We'll be able to hook those API calls. We'll get more raw insight signals and telemetry. But it's fragile there. It's super-duper sensitive, and one mistake could lead. to this. And I'm sure that could go on for other cybersecurity vendor, ABC, XYZ,

12, 3. Anyone could have been susceptible to this. So hug-ops to the crew is fighting fires for this. I do not envy crowd strike in their position, but it could just as well happen to any of us. None of us are immune. I mean, that brings us nicely too. Immediately in the aftermath of this, people are trying to figure, okay, what is going on? Is this a leak? Is it a breach? It becomes pretty quickly apparent that it is neither of those things. It is an error. Hearts go to the people involved in it. Let's dig into what actually happened here. High level.

What occurred? What was the mistake? What was the error that caused this? Yeah. CrowdStrike, again, this is part of their normal process. This is part of the typical workflow to push out changes and updates for really lack of a better word. I know we can kind of spin our wheels on that word update, but just, hey, new detection capability,

new configurations and features for their endpoint agent. And this happens all the time when a security company sees, uh-oh, there's new threats on the horizon, there are things that we should push out to help keep our customers and clients safe. This one just had a bit of a gimmick, had a blemish, had a pimple, and that is what crashed all these computers.

I know I was getting a little bit nerdy on the CSagent.Sys, but that kernel driver loads in these other things like those called channel files. that presumably have a little bit more logic and information in a small compartmented way for it to seamlessly and, I don't know, correct all those and do it in a good orchestration. But, whoops, a lot of back and forth on, oh, was it the null bites in the file? Crowdstrike his claim that's not. But there was an error, an access violation, something that crashed at that low-level kernel,

and that is what broke all this down. Yeah, I saw that the spotlight so far is sort of shining down something called Channel File 291. What is that and how did that go wrong? Absolutely. These are, and I'll admit, I think some of this is Crowdstrike lingo that I'm not by any means an expert on, but I'll do my best. They're small little units. Hey, some small modules that can better explain the logic or the functionality and kind of like plugable modules for the kernel driver doing its work. And I believe if you dug through some of their technical details, CrowdStrike has done a great job trying to get some more of that messaging out, share some insight.

Well, this one is specific for named pipes. And again, I don't mean to get too nerdy and geeky, but malware and a lot of, I don't know, ransomware, info stealers, whatever, it's crypto miners. That laundry list can go on and on. Sometimes they'll communicate with some inner process capability in Windows that's called named pipes. So rather than leaving a file on your file system or, I don't know, trying to leave some other artifacts that it could resume and check back into later, they'll use that capability. channel file 291 was purpose built for named pipes but again hey there was a mistake and unfortunately I think they're still trying to dig down and find that root cause analysis there's speculation online there are folks chiming in with their hot take but I got to admit it's sometimes best to

just wait for the concrete proof definitive answers from the source think about the last time you heard a breach story on this show it always starts the same way someone somewhere saw something too late, an alert buried, a signal missed, an SOC that just couldn't keep up. Arctic Wolf set out to solve that problem by rebuilding security operations from the ground up for a world where attackers are already using AI. They created the Aurora superintelligence platform, a fully agentic system powered by the swarm of experts. Instead of single-purpose bots or lucky-guess LLMs, this swarm is full of deterministic agents

that handle whole entire workflows. Humans stay in the loop and on the loop to validate the critical decisions and keep everything trustworthy, and all of this is just off running on their secure operations graph. A constantly updating intelligence engine fueled by more than 9 trillion telemetry events every week and over a decade of real-world incident response. The system reasons on real signals and real context not synthetic training data. And the result is the new Aurora agent SOC. It's the first SOC that is agent led by design.

You get agents that coordinate, agents that investigate, agents that respond at machine speed, and hundreds more that automate the repetitive work that normally buries human analysts. Arctic Wolf didn't try and bolt AI onto an old model. They rebuilt the model entirely. What makes it even more effective is how it works with Arctic Wolf's concierge experience. The team brings customer-specific context directly into the platform so every AI-driven decision reflects your environment instead of generic assumptions. The automation frees your concierge security team to focus on higher value strategy and proactive risk reductions while the agents handle the grind. If you want to see what trustworthy,

production-ready AI and security operations actually looks like, go to arcticwolf.com slash hacked. Ever feel like cyber threats are evolving faster than anyone can keep up? Last year, 2025 was nothing short of a record-breaking year for major breaches, from sophisticated ransomware operators to AI-enabled attacks that turned defenses on their head. Organizations around the world saw headlines they never expected and cybersecurity teams were tested like never before, but here's the thing. These incidents aren't just news headlines.

They're learning opportunities. And that's why Arctic Wolf is hosting a live webinar on February 5th, diving to the most impactful breaches of 2025. Their field CTO and security leaders are going to unpack not just what happened, but why these attacks succeeded. And most importantly, what businesses can do to fortify their defenses for it's too late. You're going to walk away with real insights into how threat actors are evolving, how defenders are responding,

and what strategies can help you stay ahead of the next big breach. It's not fear mongering. It's practical, actionable, intelligence from experts in the trenches. Register now at arcticwolf.com slash hacked. What does a potential fix for something like this look like? And where does that come from? Oh, super good question. And there's been some talk on this,

because folks are wanting and rightfully so a bit of an easier solution than running around with the USB drive or, I don't know, manually slamming the F8 key to boot into safe mode one by one by one. So some folks were thinking, oh, can we automate or deploy or push out? Like group policy as an example. Some are Microsoft Intune. Any of those capabilities to at scale make changes to computers.

But again, for one thing, you have to hope that maybe your domain controller or your managing group policy instance is not offline and stuck in the boot loop. And then the downstream endpoints, the workstations that it would try to fix and remediate, well, those need to be in a state to be able to even receive. those updates and follow through with those operations. So while we keep trying to think up and brainstorm and think, okay, how could we best clean this up? It's going to take some manual effort and it's going to take some time. I don't mean to keep falling down the rabbit hole, but a lot of folks might be thinking, oh, you mentioned safe mode and, oh, you got a couple commands to delete

or clean up these channel files. But wait a second, what if you have like hard drive encryption? Some folks might be familiar with Bitlocker, and you're absolutely right. You, you need a recovery key for a little bit more access booting up into safe mode and doing that. And forgive me, I'm sorry, Jordan. Hey, please pull me back if I'm getting too far in the weeds. But this is a thing that, again, just adds another variable because the system administrators, the network owners and architects, well, you need to have the recovery key in the case of BitLocker for every single endpoint.

And, hey, do you have that documented? Is that written down somewhere? Do you have that printed out and not at a digital form? Because remember, all your servers are kind of kaput's right now. Goodness, I hope I'm not understating, yeah, how much of a nightmare it could be. No, I don't think you are. I think there's a sense that this is, especially at a time of recording, like, very, it's not great. Largest IT incident in recent history is I've seen that headline being floated around by people,

and I have no reason to know whether or not that's true, but I get it. It sure looks like it. I did see a number from, Microsoft, I believe, that said this incident has affected eight and a half million computers, which is bonkers and crazy to think of. Well, and given that those are computers running CrowdStrike, we can assume they're not, you know, a person's laptop sitting on their desk. Computers probably were tasked with some pretty important stuff.

Yes. And thank you. I'm sorry. I should have colored that picture a little bit better. Again, maybe folks just on your own, I don't know, home device or your personal laptop playing games and all, well, you're right. Maybe you're not running, oh, oh, the crowd strike falcon sensor because it's not a corporate or business environment. But just as you mentioned, whether it's infrastructure, whether it's stuff that can help power and protect airlines, banks, schools, hospitals, et cetera, et cetera.

That is where we're seeing quite the disaster across, dare I say, the economy, the society. I've been fascinated to see. So there's the incident itself and then there's the way it ripples out into the larger tech and business ecosystem. I saw you posting about people springing up domains to sell. I survive. I went to the CrowdStrike, 2024 incident, and all I got is this T-shirt, kind of merch celebrating this. Can you tell me about sort of like peripheral to the incident itself,

what you've seen happening in the ecosystem? Oh, thank you for pulling on this thread. Because I think this is another thing that's absolutely worth noting. Hey, whenever some crazy shenanigans unfolds, whether there's something that shocks the world, well, other threat actors, adversaries, actual ill-intended people, will try to capitalize on that.

They'll take advantage of it and they'll keep spreading that chaos and uncertainty, fear, uncertainty, and doubt. So you'll see fishing emails, of course, hey, scam phone calls. I think I believe I've heard some folks that were receiving calls from people impersonating Crowdstrike, claiming, oh, they'll offer support, hey, they'll help,

but no, it's a scam, it's a lie. and you're right the thread that I had shared over on Twitter or X right was hey maybe some new domains new websites that are out and about and some could be a funny joke some could be a gag

selling t-shirts and merchandise but some could very well be used for hey we're gonna offer you a crowd strike hotfix dot zip file download that run the contents and well that's malware now you've just made this situation even worse so it's unfortunate

but I think absolute reality whenever things go down like this. Wow. My next question was going to have to do with people fishing attacks and stuff, with people pretending to be either from Microsoft or CrowdStrike. But it's interesting to know that there's sort of this middle ground of people being like, yeah, I'll sell you a solution to this thing that happened three days ago and no one has a solution to yet.

We harken back to, I think, what was it, CDK, a lot of automobile and the car vehicle industry that had quite a similar conundi. still kind of in the aftermath and the embers of that. But I know that was quite a story is, hey, we just keep getting these phone calls from folks, again, impersonating, masquerading, trying to lie and act as CDK. But goodness, you got to be on your guard. And I hate to sound so silly. I know it's a basic boilerplate stuff that everyone says, but stay vigilant.

Stay, keep your ear to the ground. Just stay in the know and really be on the pulse with this so that you can be aware and conscious of what's coming up on your computer screen. Or not. Blue screen of death instead. What is CDK? Oh, I'll admit, I don't know if that's an acronym or I'm not a car guy, so I'll fall on my sword here. But I believe that's one of the providers for, is it tech software, tech supply? I had not followed that story to the best of my knowledge.

But it's just another in the, what is it, cyber bad weather folks might tend to say, just another incident after another breach, after another news and headlines. unfortunately when you're in the midst of it, you tend to see it all too often. I mean, it brings up a good thing as, you know, I was reading about other large historical single point of failure type events. Someone was posting about the AT&T Long Lines Building. I'm not sure if you know about that story from, I think it was the early 2000s, or maybe late 90s. It was a single tower on the eastern seaboard that was a, we figured out after the fact it

was just a choke point for all telecoms going into the eastern part of the U.S. and then it was near the termination point for the big Atlantic cable. Something bad happened in that building one day and it just shut down communications on the entire eastern side of the country and across to Europe. Now there isn't one single point of failure. Was this a freak accident or are these sort of larger potential single point failures kind of just inevitable?

Ooh. Very good philosophy question. Philosoph. Early in the day, a little kind of lob at you. I'm teasing. I think, you know, there's a lot of conversations of, look, I miss the old, hey, software

that would just run on my computer, a local desktop app, not, oh, something in the web browser that's using the cloud or someone else's computer to serve that data back and forth. It's like have we decentralized or now centralized into just someone else's provided infrastructure?

Again, whether that's Amazon, AWS, Microsoft, Azure, Google Cloud, blah, blah, blah. I can't say. I don't know. I really wish I had the right answer. I wish I had the solution. I just know that that's where the world is going. And that's something that we can stay cognizant of.

But it's funny, you have conversations with a lot of system administrators and network engineers that say, hey, do you have automatic updates turned on? Especially, again, hearkening this back to our CrowdStrike conversation. because you want to get those patches, you want to get those hot fixes, when there's a new vulnerability out and about, you want to make sure that that's plugged up and clean the best that you can

right in the moment. But you are running the risk of, wait a second, what if something goes wrong, crashed as a computer, unstable state, memory error, blah, blah, blah. When you have that conversation with the system administrator, that's very hard to find the right answer as well.

It's a balancing act. But in the case of this crowd strike conundrum, it really isn't even up to the administrators of themselves. This was an unfortunate push from that single point of failure that, in this case, is crowdstrike, which is a little bit mind-blind. Is there anything else people need to know about this story, anything else we didn't cover? Hey, I'd love to get your hot take just as well. But I think if I could try with some of those parting shots,

I know that this one took us all by surprise, myself included. And it's very, very tough when folks might. ask, how do we prevent this? How can we stop this from happening in the future? What could this do, et cetera? But again, you are trusting this provider. And unfortunately, that is the chokehold here. So I think the very best that we can do is try to have some more of that strategic planning. And I know this is fluffy. I know that's very vague and broad. But look, can we tabletop exercise this scenario? Can we think about, oh, who are we going to call? Who do we have the numbers for? Who do we know in disaster recovery?

Do we have backups in place? Do we have a checklist? Do we have documented like standard operating procedures when something like this goes down? And again, I know it's fluffy. I know it's vague, but it is I think the best that we can do when we try to prepare for the stuff

we feel like we can't even prepare for. We're fighting an unknown enemy. No, not fluffy and vague at all. What happens next? Just to keep it on, you know, in the ether, where do you think this goes next? I am hopeful that this will maybe be cleaned up this week. I don't know if that's going to happen.

I don't know if it's going to take two, three more than that. I think some folks are getting back into action. We're in a frenzy still, and I don't know if I'm shell-shocked from just, hey, trying to chase this thing, all that all fell away on Friday. But for now, it's still taking those lessons learned. It's still having these conversations and trying to share and spread that messaging. so more folks are aware and can get back into action the best they can. John, I really appreciate you taking the time to sit down and talk with me about this.

Hey, thank you so much. I hope I wasn't rambling, yapping for too long, but it's a real treat. Thank you again and again. Cheers. Appreciate it.