Darknet Diaries - 57: MS08-067

Episode Date: January 21, 2020

Hear what goes on internally when Microsoft discovers a major vulnerability within Windows.GuestThanks to John Lambert for sharing this story with us.SponsorsSupport for this episode comes from ProCircular. Use the team at ProCircular to conduct security assessments, penetration testing, SIEM monitoring, help with patches, or do incident response. Visit www.procircular.com/ to learn more.This episode was sponsored by IT Pro TV. Get 65 hours of free training by visiting ITPro.tv/darknet. And use promo code DARKNET25.Support for this episode comes from Blinkist. They offer thousands of condensed non-fiction books, so you can get through books in about 15 minutes. Check out Blinkist.com/DARKNET to start your 7 day free trial and get 25% off when you sign up.Sources https://blogs.technet.microsoft.com/johnla/2015/09/26/the-inside-story-behind-ms08-067/ https://www.justice.gov/opa/pr/payment-processor-scareware-cybercrime-ring-sentenced-48-months-prison https://www.nytimes.com/2019/06/29/opinion/sunday/conficker-worm-ukraine.html https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0601 https://www.wired.com/story/nsa-windows-10-vulnerability-disclosure/ Book: Worm AttributionDarknet Diaries is created by Jack Rhysider.Episode artwork by odibagas.Theme music created by Breakmaster Cylinder. Theme song available for listen and download at bandcamp. Or listen to it on Spotify.

Hey, it's Jack, host of the show. Let me ask you a question. Whose job is it to keep the roads you drive on safe? Is it the driver's sole responsibility? What about the car makers? Are they responsible for keeping the roads safe for other drivers? What about the cops? Maybe they need to come by and watch everyone to make sure they're obeying the law and keeping everyone safe. Or wait, maybe it's the job of the civil engineers, the people who design the roads. I mean, a crazy curvy bumpy road with a speed limit of 100 miles per hour is obviously not safe. So it must be their job to design it to be as safe as possible, right? So whose job is it to keep the roads safe? All these people. We need drivers to drive safe. We

need cars to be built with safety in mind. We need cops to catch people who aren't being safe. And we need civil engineers to design us safe roads. And I think this analogy applies to keeping our networks and computers safe too. We need users to be smart at what they click on and do. And we need software makers to design the software to be secure. And we need the cops to arrest people when they break the law. And we need groups who set up industry standards that guide us to safety. We cannot rely on one person to keep our networks safe. It takes all these people to always be vigilant to keep our computers safe. And this is a story about what happens when a software maker finds a bug in their own software and what those effects were.

Specifically, this is a story about when Microsoft found a massive bug in Windows, which paved the way for the largest worm in history. These are true stories from the dark side of the internet. I'm Jack Recider. This is Darknet Diaries. This episode is sponsored by Delete Me. I know a bit too much about how scam callers work. They'll use anything they can find about you online to try to get at your money.

And our personal information is all over the place online. Phone numbers, addresses, family members, where you work, what kind of car you drive. It's endless. And it's not a fair fight. But I realize I don't need to be fighting this alone anymore. Now I use the help of Delete.me. Delete.me is a subscription service that finds and removes personal information from hundreds of data brokers websites and continuously works to keep it off. Data brokers hate them because Delete.me makes sure your personal profile is no longer theirs to sell. I tried it and they immediately got busy scouring the internet for my name and gave me reports on what they found. And then they got busy deleting things. It was great to have

someone on my team when it comes to my privacy. Take control of your data and keep your private life private by signing up for Delete Me. Now at a special discount for Darknet Diaries listeners. Today, get 20% off your Delete Me plan when you go to joindelet me.com slash dark net diaries and use promo code dark net at checkout. The only way to get 20% off is to go to join delete me.com slash dark net diaries and enter code dark net at checkout. That's join delete me.com slash dark net diaries. Use code dark net. I'm very excited about this episode because I think this is a rare story to hear. We aren't going to hear from the hacker who found the exploit, and we aren't going to hear from the company who got hacked by this exploit. Instead, we're going to go right to the source to hear the

story of one of the most notorious bugs ever, right from the horse's mouth, Microsoft. Hello. Hey, Jack, can you hear me? Yeah, I hear you pretty well. What kind of mic are you using? This is a headset. It's actually made by Microsoft. This is John Lambert.

My name is John Lambert, and I work at Microsoft. He's been with Microsoft a long time. While he was there, he spent 10 years working on a team called the Trustworthy Computing Group. So that was a group that was created after the early worms of CodeRed, Nimda, Blaster, Slammer, and it was designed to help improve customer trust in Microsoft products by fortifying them from a security, privacy, and reliability perspective. The domain I worked in was security. And so a lot of that focus was on finding and eliminating vulnerabilities in designing and coding from the products and working across the Microsoft sort of product suite, not just Windows or Office, but all of

the products to sort of build the techniques that security researchers had familiarity with and then inculcating that sort of ethos and know-how inside of the Microsoft development cycle. Whoa, what an interesting role to have, right? His mission is to improve the trust people have in Microsoft products by making them more secure. And I just looked this up. In 2008, Microsoft had 91,000 employees. With that many employees, yeah, I guess it makes sense to make a team to improve the trust of their products. Oh, and in case you're wondering, in 2019, they had 144,000 employees. Now, of course, the biggest Microsoft product that John would work with is Windows, the operating system itself. John got the ability to examine Windows in unique ways by looking at the source code

and building relationships with developers and conducting security tests against it. It's not really his job to make sure all the bugs are squashed or to find the bugs, but he was certainly involved in getting all the teams together to help find the bugs and get them fixed. Okay, now, when a team at Microsoft finds a bug in Windows, they give it to the developers who then work on a fix and create a patch for it. And they issue these patches on Tuesdays. Every patch Tuesday, which is every second Tuesday of the month, we will create a number of security bulletins.

The bulletins essentially are the advisory that describes, here are the vulnerabilities that we are fixing in, say, Internet Explorer or Windows or Office. And every bulletin may have one or more vulnerabilities that is being fixed. Those have individual CVE identifiers that security professionals sort of are familiar with. But the bulletin is essentially a grouping of them for a product. And the bulletin is rated, you know, critical, important, moderate, a level of severity. Now, these advisory bulletins put out on Patch Tuesday might have a name like MS-07-029 or MS-08-067. And if you see something like this, MS07-029, it means the advisory was published in 2007, and it was the 29th advisory of the year. So for this story, we're going to be talking about

the security bulletin MS08-067, which means we're talking about this bug which was found in 2008. Vista had just come out the year before, but its adoption rate wasn't that strong. So the majority of Microsoft Windows customers were using Windows XP still. And take a guess at how many lines of code it took to create Windows XP. You ready? 45 million lines of code. And of course, that was spread out among many teams. Trying to keep a program that big bug-free is, well, in my opinion, impossible. It's just way too big to keep it bug-free.

There has to be some function or routine in the code that didn't get tested properly or coded properly and has a glaring bug. Or sometimes it gets tested and coded properly, but there's a bug that's not so glaring in there. It only appears under weird circumstances, like only when the memory is filled and it's only a certain millisecond on a certain time of day. Weird stuff happens when you have a code base that big. And all this is to say, because Windows XP was 45 million lines of code long, just by that size alone, it must have had a lot of bugs. But before we get into MS-08067, maybe we should talk about a bug found the year before called MS-07029.

Okay, so that MS-07029 was a very important, it was sort of the moment of insight, if you will, for all the things that came after it. What MSO 7029 was, that was a bulletin that corrected a vulnerability with Windows DNS. And that when Microsoft became aware of it, a customer that was being exploited in the wild contacted us and said, somehow we just got attacked. Here's an attack tool that we were able to find associated with it. We think it's exploiting a vulnerability somehow. And that goes into MSRC.

MSRC is the Microsoft Security Response Center. This is the team of security professionals within Microsoft that is in charge of making Windows secure and looking for bugs and getting them fixed. They took this report they got and tested it to see if you could hack into a fully updated Windows computer with it. And it worked. And then they investigated and found there's a zero day in Windows DNS that is being used. They put out an advisory. They put out a bulletin. And what I was doing is I was looking at this sort of crash dump system that Microsoft has, this Windows error reporting.

And I was working with some engineers that were looking at the crash reports coming in associated with Windows DNS. The DNS product was otherwise very reliable and hardly ever crashed. And so the only crashes coming in at that point, when we looked at them, were actually attacks in the wild. And when we started to examine them, we could tell that actually we could have known about this attack in the wild much earlier than the customer contacting us, if only we'd been able to pull these needles out of the massive haystack of crashes coming into Microsoft. Do you ever use Windows or Internet Explorer or Microsoft Word, and the app suddenly crashes and it says, do you want to send the crash

data to Microsoft? This is what's known as WER, WER. Yeah, so Windows Air Reporting, WER, it's also known internally as Dr. Watson. It's really a technology that goes back to the late 90s. Both Office and Windows were independently working on how do we get better data on how the products are crashing before we ship them? And then after we ship them, other than customers calling customer support, how do we get data about how they're faring in the wild? And so both Office and Windows had built features to collect crash reports at scale that could be automatically submitted by customers and then run analysis tools against them to try to root cause and bucket them to say, okay, is this a new bug we don't know about or is this an existing bug that we do know about happening again? And those two efforts came together. And that feature was built into Windows. And that was called Windows Error Reporting. So yeah, that little box that pops up when the app crashes,

will ask you if you want to send the crash report to Microsoft. And if they say yes, then that communicates with Microsoft. And the backend system decides, is this something we already know about? Or is this something new? And if it's new, it starts to prompt the user to potentially upload more data so we can root cause the issue further. Now, remember I said that XP had 45 million lines of code and a program that big is impossible to be bug free. And yet I don't think it's like just a dozen bugs in a program that big. And I don't even think it's just hundreds of bugs or thousands of bugs. I think there's more like tens of thousands of bugs in a program with 45 million lines of code in it. At the same time, Windows was installed on a billion computers in 2008.

So Microsoft was seeing millions of crash dumps a day from these computers. So let's try to be a project manager for a second here and come up with a strategy to tackle these millions of crash dumps a day. We have a few options. First, let's just filter out all the known bugs we've already fixed. People's computers are crashing, but they just need to patch it and it won't crash anymore. Okay, that's out of the way. But now when we try to prioritize what to look at next, it's not so easy. It might make sense to tackle the bugs that show up the most, but these might be very low severity, maybe not as important. Maybe there's something with a more high severity, but not as many crash reports. So then you might decide to look at the highest severity crash reports instead.

Or maybe some apps are more important than others, like if MS Paint crashes, it might not be as big of a deal as if the whole computer was crashing. Or maybe you can look at the easiest to fix problems first, get those out of the way. It's really hard to know what to prioritize here. So this might give you a better understanding of how hard it would be to look through millions of crash reports a day to figure out what to fix first. But it was still very important for Microsoft to collect these dumps and analyze them. At this time in 2007, John was starting to look at these crash reports to try to make sense of them. They go up to a massive automated system. These tools run against them at scale in a completely headless way. They are sort of bucketed and bend in a way that makes it easy

for engineers to know, is there something new coming along or how active, you know, how prevalent is this issue? And then a wide variety of engineers across the company. If you know, if you work on office, you look in the office crash buckets and see what is hot for you there. If you work in windows or any other product, you can kind of see how is my product crashing. So engineers across the company were using this to fix bugs. And then like, for example, in Windows Vista, from the trustworthy computing side, they did all kinds of static analysis tools to find and remove reliability issues. And I think they fixed 100,000 bugs through those efforts. And through Windows Air Reporting, they found another 5,000 bugs that had escaped all of those processes that they went and fixed.

And so every service pack, every new product from Microsoft is more reliable because they're finding and fixing all these bugs that are manifesting in the wild. I was there kind of on the side saying, this is a fascinating telemetry system. How do we look at this from a security point of view? You said 100,000 bugs? 100,000 changes to the product that were done because of, not all of those are bugs. One way to think about it is there's coding practices that are sort of outdated. Developers will know about these things as like calling these unsafe string functions like stir copy and so forth. And instead of trying to figure out which ones are vulnerabilities, which ones are not, they said, let an easy way to just go to go say, look, we're going to have a new standard of engineering. We're going to go get rid of that whole class of thing by tackling that.

So that's an example of some of those changes. Yeah. Yeah. But I mean, are you exaggerating when you say 100,000 or? No, no. Jeez. Can you imagine trying to keep something like this secure to fix 100,000 things in the code? It just sounds like madness to try to complete.

I guess this is why they needed 91,000 Microsoft employees to tackle huge issues like this. And these crash reports were really helping them identify the problems that needed fixing. But even though these might be bugs, they might not all be security problems. Like if you click print and nothing happens, that might be a bug in the code. But is it really a security issue? A hacker probably can't use that to take control of your computer or hack you. But John was looking at this and wondering if there was anything in these crash reports that is something a hacker could use, or maybe signs that a hacker caused a crash.

Some vulnerabilities are discovered by attackers and exploded in the wild before Microsoft is aware of them. And how could we go find that before customers contact us? And there are a lot of reasons that exploits actually fail in practice. And that was the idea, which is instead of trying to find exploits when they're working, sometimes they don't. And there's a bunch of interesting reasons why they don't. And by studying the causes of failure of those exploits in the wild, that would lead us to potentially discover them. Some of the reasons they fail are because the exploit was written for, say, an English-language version of Windows and it's run on a Chinese-language version and they're slightly different internally. So a variety of reasons that exploits would fail in practice.

And I studied extensively all the different patterns of how they fail because that was what I needed to be able to understand to find them. The hunt was on for John. He wanted to find traces of hackers in the WER logs. But like I was saying, there are millions of error logs a day. So this wasn't going to be easy. Yeah, I mean, I think back then it was over a billion a month. And recognize that it's like, how could you look through a billion a month?

Because of the way Windows Error Report buckets the issues, you don't have to look at every single one, one by one. You're really trying to find new ones, new ones that are interesting in a different way. And then another way to think about it is there's only a certain number of ways that zero days are going to kind of show up and affect an application. And so, you know, people can send malicious documents and that'll crash Word. So Word is interesting to look at. People can send exploits with the browser. So Internet Explorer would be interesting to look at. And then if you think about even inside of Microsoft Word, if an attack is going to happen, likely it's going to happen when the user is first opening the document.

You know, it's rare that some exploit is designed to work when the user's been in Word for an hour and they finally click tools, options, spell check, add in, and then it goes boom. Those are not the paths where an exploit is going to work. It's going to be in the file open code path. So that further narrowed down the places that we needed to look. And then exploits fail in certain specific patterns that are the kind of patterns that we sort of knew to go sifting over. So we were able to work that funnel down from a billion reports a month to millions that seemed interesting

to hundreds of thousands that had other clues in them and so forth and whittled the funnel down. John dedicated a lot of time to try to find this unknown vulnerability that some hacker was exploiting. And when we come back from the break, John finds something that he'll never forget. Support for this show comes from Black Hills Information Security. Thank you. give them a call. I'm sure they can help. But the founder of the company, John Strand, is a teacher, and he's made it a mission to make Black Hills Information Security world-class in security training. You can learn things like penetration testing, securing the cloud, breaching the cloud, digital forensics, and so much more. But get this, the whole thing is pay what you can. Black Hills believes that great intro security classes do not

need to be expensive, and they are trying to break down barriers to get more people into the security field. And if you decide to pay over $195, you get six months access to the MetaCTF Cyber Range, which is great for practicing your skills and showing them off to potential employers. Head on over to BlackHillsInfosec.com to learn more about what services they offer and find links to their webcasts to get some world-class training. That's BlackHillsInfosec.com. BlackHillsInfosec.com. On September 25th, I remember opening a crash report for the service host process that we have seen many millions of crash reports for already. Because a couple of years earlier, there was this other vulnerability, MS-06-040, that had been adopted by bots and worms to spread.

And it was causing millions of crashes against machines that had not put on that patch. But this one was a little different. So first of all, it had an exploit. It had exploit code in it. Okay, that didn't mean it could be new. It could be an old one. It was an exploit on the stack, which is a critical part of memory that tells you that exploit was trying to be activated right now.

It had a difference in it, what I call an egg hunt, which was an exploit technique that I had never seen before for any exploit in MSO 6040. So this was either a new strain of that or something new for that service host. And this ACONT is an exploit writing technique where, as the bad guy, imagine you're going to go scale a wall and you put all your attack tools, your thieving tools in a bundle and you throw it over the wall. And then later you go scale the wall and go get that. And ACONT breaks the exploit into two pieces to serve a purpose. And it had that technique in it. So that alone drew my eye to look at it. And then the odds of there being a buffer overrun and an exploit in the same area as this MSO-6040 just seemed unlikely.

And yet I felt like if it was new, this was really important. And so I just tried to stick with it and do what I could to rule in or rule out whether this was new. And one of the most stubborn clues was in a crash report, it has information about every DLL that is loaded into it. And the DLL that had MS06040 was loaded in it. That is this NetAPI 32. And the version information told me it was fully patched. So it clearly could not have been exploiting that vulnerability because that vulnerability did not exist in that version of the product. John looked at the logs here, and it looked like a hacker was exploiting two processes.

One was basically injecting some hacker tools into the system, hiding an egg, if you will. And the other was going in and using those tools. A strange combo, but like John said, it was sort of like throwing your tools over the fence and then jumping over the fence to get them. The two processes being exploited here were SVC host or service host and NetAPI 32. Yeah. So sometimes when you are writing an exploit, you have constraints that you have to work within. And once your exploit starts to run, you have a number of steps as the attacker you're going to want to do. Typically, that involves downloading some external piece of malware to that system, and then launching that, and then the rest of your attack. Then you're resident on that computer, and you can do whatever attacks that you want to do further from there. But you have to get it

going. And sometimes that shellcode, as it's called, to get going, can't fit within the constraints of the vulnerability. Like the buffer, for example, needs to fit in is just too small a box to package all those instructions with it. And so an egg hunt is a technique that is designed to solve that by first sending over some data and interacting with the vulnerable program in a way that doesn't use the vulnerability at all. It just tries to get data and memory that is that bag of tools, that shellcode that they are going to use later. And then the only thing that they need to get when they actually run the attack is a very small piece of shellcode that basically goes and searches memory to find that bag of tools. What a tricky exploit. And because there's two parts to this bug, it makes it really hard to

find. The more John looked at this particular crash report, the more he started to realize this was actively exploiting an unknown vulnerability in Windows, which makes it a zero-day bug. This, in a way, was the hardest moment of this entire thing for me because I clearly had enough evidence to say this was a new attack, a new vulnerability that we did not know about. And to get the company to act, you need to actually pinpoint the vulnerability. We have to know what's the code that's wrong. Otherwise, nothing can happen. And I poured over the code and poured over the code and I could not figure it out.

At one point, I decide the clock is ticking. This is a potentially really bad situation. I need to go ask for help if I'm going to figure this out. And I walked over to the office of Andrew. Andrew was a colleague of mine. We had worked together previously, actually. Andrew had looked at many crash reports for security objectives for a long time. And he knew what this hunt was like, which is, in a lot of ways, it can be a very frustrating hunt. And you can spend hours looking at something you think is real and then it's not. And so what I wanted is hand this to one of the MSRC engineers and they will go figure this out. And he was

reluctant to take an engineer off of an existing confirmed vulnerability that somebody else had reported, you know, and was in the middle of to go potentially chase a ghost. So he took it on to go look at it to, I think, in his mind, say, look, this is a false positive. This is not a real issue. And case closed. And so there was some tension in the air with me really pushing a busy team to go look at something that likely was not going to pan out. But I felt like it could be important enough. It's worth doing that. And Andrew wanting to protect his team, but do the due diligence to make sure we got the answer right. So Andrew got a copy of this crash report and the code for these processes, and he started analyzing all this. He spent a couple days looking it over and working on it,

trying to find what the crash report means. There's an egg somewhere in the code, but neither of them could find it. But this error report did say that there was something causing a crash. Now keep in mind, John had only seen one crash report from this. This wasn't a widespread issue. So both John and Andrew weren't entirely sure how much effort they should spend into looking for something that only happened once on one computer. But if there was an unknown vulnerability, both John and Andrew wanted to find it and fix it.

So Andrew kept looking. And then at one point he stops by my office and the look on his face told me everything. The look was the look that a security researcher has when they found something. There's a goofy, happy smile that is also full of, holy cow, I can't believe how serious this thing is that I just found. He said, I found a vulnerability. And when I heard that, I knew everything was going to, the next two weeks of my life were going to be completely different because this vulnerability was in an area that would allow an internet worm to be written against it. The vulnerability they found allowed an attacker to take remote control of a Windows computer. No need for a username and

password. No need for RDP to be enabled or anything like that. Just give me full control of that computer and now I can issue my own commands on it, see any files, do whatever I want on that computer as if I'm right in front of it. And of course, to top it off, it's a vulnerability in a fully updated version of Windows. In the security world, we call this kind of vulnerability an RCE, which means you can do remote code execution. Like, a hacker can execute whatever commands they want on the victim's machine, and this is the worst kind of vulnerability.'s the most critical the most severe because you absolutely never want just any joe schmo executing commands on your computer which has full defenses up if you can run whatever

commands you want on someone else's computer it pretty much means it's your computer. But to top it off, as if this highly critical and severe vulnerability wasn't enough, this exploit was wormable. Wormable means that you have a vulnerability that an attacker can write an exploit for, and it can propagate across the internet, exploit that vulnerability, and then turn around and continue to repeat the process and propagate further. So it becomes a viral outbreak. And it is the most damaging, devastating, disruptive kind of attack that can take place. And the world had seen many worms in the form of Blaster, Code Red, NEMDA, Slammer, Zotop. And we knew how devastating these attacks can be. Entire businesses are disrupted.

Systems are taken offline. Network traffic gets clogged with worms replicating out of control using a ball available bandwidth. And so we knew what the potential was. We didn't think that that state had been happening yet, but we knew that was possible. So when Andrew came to John's office and told him this vulnerability is real and wormable, the two immediately jumped up and sprang into action. So Andrew and I both are on the engineering side, and we knew we needed to go activate the crisis response part of Microsoft. So he and I immediately leave my office.

We walk down the hallway to the office of the crisis manager. His name is Philip. Philip has worked on many of Microsoft's most important security crises. And he's chatting in his office with a colleague. We show up. He knew that wasn't a good sign. We look at him and say, we need to talk. He looks at his colleague and says, I'll talk to you later. And then I said, we have a zero day. And he just knew by the two of us showing up in that fashion that something bad was happening. I am mostly have had two emotions going on. One is we need to get all of Microsoft engaged on this that can do something about this. So that is an impetus to go make sure you're informing people so they of the world, or this is something that's just getting started.

So I immediately wanted to go and get some better situational awareness about scope and scale to know what we were really dealing with. John and Andrew briefed the crisis manager on what's going on, that a very serious, wormable, and extremely critical vulnerability is present in Windows and needs to be fixed immediately. And while they found this in Windows XP, the vulnerability existed in more than just XP. Yes, and it affected every version of Windows up until we had it at that point. So Microsoft has a crisis response process that they invoke when one of these things occur. It's called a SERP internally.

And then a call bridge is stood up, and then all of the kind of crisis partners across the company join that call. And then Philip would take them through, here's a summary of the situation. And generally speaking, if there's a vulnerability in Windows, you know, teams know what to do. This would involve a lot more teams, because they had to be ready for customer support calls. What is if there's an attack, what is the malware? What is the threat side of it look like? And the anti-malware team will start building signatures for that. We knew we had to prepare data for all of the security partner companies across, you know, Symantec, McAfee or whatnot for them to help go protect their customers and the ecosystem.

And then the engineering team needs to go, okay, what do we have to do to fix this vulnerability? And are there any others just like it waiting there that we need to fix at the same time so we get the patch dead right? A huge conference call was set up with pretty much someone representing all the different departments of Microsoft. The goal was to get everyone engaged in helping solve this as quickly as possible. This vulnerability was much more severe than any of the others they found. Once I knew that the right people were engaged

and working to get the right patch out the door, the thing I could go help on was, how often is this happening and where is it happening? And I called that dialing into the crash buckets to find any information I could about how often this was occurring. And I was able to bring back the situational awareness to the crisis response that said, okay, I saw it five more times today. It just spread from Malaysia to Japan Singapore, to whatnot. And they could kind of get a sense of, okay, it's the same attack payload

communicating with the same IP address on the internet to pull it down, but it's spreading from geography to geography to geography. So it didn't look like a worm, but it looked like some set of hackers that were going around using this tool across a variety of targets,

expanding its scope every single day. Yeah, see, that's fascinating. So somebody knew this vulnerability. They found this before Microsoft did and was using it. This wasn't just some accidental bit flip or something that caused the crash report. This was actually somebody had coded this and was using it in Asia. That's right. So somebody found this vulnerability, probably got the bonus of the year for finding it. And then it was weaponized. And then some group, unknown to this day, was using it in those geographies where we saw these crash reports coming. And the tool was not completely reliable.

And so that's the crashes that I were seeing. That's when it failed. But obviously, it worked and it succeeded a bunch of the time. And I didn't know how often. For every crash, is that one in 100 times that that it's crashing or is it one in a thousand? What shadow of the real activity am I seeing? That was an unknown that we were working against. And so, but I could use this to sort of get a sense

of if it's spread and it's reach and it's pace. Now, this became a top priority for the development team to fix. They dove into the code and fiercely started to rewrite the code to fix this problem. And they did. They fixed the vulnerability. But this is not the end of our story. No, no, no, no, no, no. This is just the genesis.

And stay with us, because after the break, we'll hear how this went on to become one of the most notorious bugs ever to be found in Windows. This episode is sponsored by SpyCloud. With major breaches and cyberattacks making the news daily, taking action on your company's exposure is more important than ever. I recently visited SpyCloud.com to check my darknet exposure and was surprised by just how much stolen identity data criminals have at their disposal, from credentials to cookies to PII. Knowing what's putting you and your organization at risk

and what to remediate is critical for protecting you and your users from account takeover, session hijacking, and ransomware. SpyCloud exists to disrupt cybercrime with a mission to end criminals' ability to profit from stolen data. With SpyCloud, a leader in identity threat protection, you're never in the dark about your company's exposure from third-party breaches, successful phishes, or info-stealer infections.

Get your free Darknet Exposure Report at spycloud.com slash darknetdiaries. The website is spycloud.com slash darknetdiaries. The website is spycloud.com slash darknetdiaries. Now that a fix was written and a patch was ready, a decision had to be made about how to release this to the world. Right, so there's a decision point as part of this response process, which is, okay, when is the patch going to be ready? And the ideal time to release a patch is on Patch Tuesday

because every IT team in every company knows on the second Tuesday of every month there will be a set of security patches from Microsoft and they know to organize their patching of their fleet of systems and take their downtime and their maintenance time to go put them on. And they know some of them will be severe and they should do that right away.

So IT teams are best ready to consume those updates and put them out at that time. The other option is you just ship it when you're ready. We call that going out of band. That gets the solution out there sooner. But at the same time, none of those IT teams are necessarily ready and poised to go grab it. And when you, we had a situation where there obviously was an attack in the wild that attackers were doing, and we wanted to go stop that from happening. But we knew as soon as we put this out, copycat attacks would happen by people reverse engineering the patch and understanding, and that would spawn a whole

other wave of attacks. And so there was this weighing of, if you go out of band, you could spawn a bunch of copycats of a very damaging vulnerability when IT teams are not ready to go put it on. And there could be more victimization, theoretically. The bottom line in terms of our calculus on it was we had a very severe vulnerability. It was wormable. It affected every version of Windows. We had a solution and we went out of band. This vulnerability was so severe that they decided to not wait until patch Tuesday and just push this out immediately, as soon as they got it. And this patch went out in 2008, and it was the 67th patch of the year, which famously made this MS-08-067.

Yeah, so there were two most interesting things that happened after we went out of band. The first one was that attacker where I was seeing crash reports still coming in every day until we went public. As soon as we went public, that attack disappeared, and I never saw him again. How can you trace that? Because, I mean, so you have like a signature for certain machines that kept getting attacked? Well, part of the attack details that were hidden. So one, the shellcode and egg hunt were kind of a server in Japan and downloading a payload from it. And so all of that was consistent. And then the day the patch goes live and we announce it, I see no more crash reports from this anymore. And then there's this period where for a number of hours or a day, nothing is coming in anymore.

And then we start to see new crash reports for the same issue that were clearly security companies reproducing this vulnerability. They had reverse engineered what we fixed in the patch. They were writing their own proof of concepts to crash it, not write exploits for it, but just to probe the vulnerability to make sure they understood it. And we started seeing those crash reports come in from the security researcher side. And soon enough, a few days later after that, we started to see new attacks from the first wave attackers that was clearly different and new than the original ones that look like bots or botnet programs adopting this as a spreading mechanism. And we started to see that wave as well.

See, this is what I mean by this being only the genesis. Now that the patch is out there, both white hat and black hat hackers analyzed the patch to figure out what this exploit was and how to run it. Which means now this tool went from being just used by a single hacking group somewhere in the world to now being known by the general hacker community. And within a short time, it would become available for anyone in the world to just download and use. Isn't that a strange dilemma or decision to have to make, though? Knowing that if you put a patch out, this reveals the vulnerability to the world for any hacker to use.

But Microsoft has a duty to make secure products, so they absolutely have to release a patch whenever they do find a vulnerability like this. Because this has far-reaching effects on helping people stay secure. In the first week, it was on Windows Update, and within seven days, I think we had patched 400 million machines. This is sort of the awesome part about Windows Update. It's a system that had been built to patch the Windows ecosystem at scale. And this is one of those times when you really needed it. And it came through in terms of our ability to essentially inoculate a huge swath of the world against this vulnerability in a very short period of time. And so it was very effective about that. It's hard to know for sure, but my best research tells me there was about 1 billion Windows computers in the world at that time,

and this vulnerability affected all of them. So by having 400 million of them patched in the first week, that was a huge win for helping the world become more secure. 40% of the Windows computers in the world were no longer vulnerable to this right away. That's awesome and amazing. So all of this was in October. And sometime in early December, the Conficker worm that had been this sort of small-scale thing for a time adopted this vulnerability as a spreading mechanism,

and then it began to use that against systems around the world that had not put this patch on yet. And if you think about, well, what if only 1% of the Windows PCs in the world don't patch? That's still millions of systems. And so a lot of the damage that was done by Conficker, which was significant, imagine what would have happened if it had a half a billion more PCs that were vulnerable and not patched against this issue. Conficker was the first big attack to use this exploit. Just months after John discovered this vulnerability, Conficker figured it out and used it to arm itself to use this exploit. Just months after John discovered this vulnerability, Conficker figured it out and used it to arm itself to infect Windows machines.

Because the vulnerability in MS-08067 was so effective, Conficker spread rapidly worldwide, ultimately infecting computers in over 190 countries worldwide. It would eventually infect millions of computers. Conficker was spreading in a terrible way. I mean, think about how horrible this is for some hacker group to have full control of millions of Windows computers worldwide. I'm talking about government agencies, businesses, home PCs. The hackers could see all the files on them, run any commands they wanted, install key loggers, take screenshots, install root kits, or do whatever they want on these computers. It's so frightening to think

about that. Configure continued to spread, seemingly unstoppable. By January, 30% of Windows computers still did not apply the patch to protect themselves from MS-08-067 or Configure, which means hundreds of millions of computers were still vulnerable to this. Conficker had a field day with everyone who didn't bother to patch and eventually was able to infect 10 million Windows computers worldwide. 10 million! As far as I know, this makes Conficker the largest worm ever, all thanks to MS-08067. It was disappointing because, at the same time, a strong lesson, because we'd put the patch out, right? The job is done. It's time for the world to put the patch on their systems. That's the step they have to do. There isn't anything further that we thought we

could do. And yet, look what happened and what came after that, all of this damage from Conficker. Now, Conficker had other ways to spread through USB drive infections and scanning file shares and so forth. a bit of a lesson for just how damaging things can become and how much of the world, you know, can be exposed to these attacks, even once we think we've done our part of the job. And it's fascinating to me to see that John here was the one who decided to take it upon himself to look at that crash report and to discover this. If it wasn't for him, who knows how long this would have stayed out in the wild before being discovered. Yeah, I think these operators using this tool thought they had a really great new attack and probably would have used it if it was more reliable and we had not seen this for potentially years. If they were stealthy and choosy about how they

used it. And sometimes these exploits we do learn about Zero Day have been in use for years in very disciplined ways. The noisier operators are with it, the more likely that some victim is going to find out about it. And somehow they'll get a whiff of how the attack is working and the thing can get patched. But, you know, I think if we had not seen this one in this way, very likely this operator could have continued for a very long time with it. This makes me think about how stealthy some hackers could be. I mean, imagine if the hackers disabled WER or blocked all connections to Microsoft. This would have been an effective technique to keep Microsoft from seeing these crash reports

and discovering this. But maybe that's going too crazy with it. But see, sophisticated hackers take extreme measures to hide their tracks. I mean, that's almost one of your adversaries, right? Are these APTs like the NSA? It's like you're battling with them. Like it's an arms race between you as the vendor and them as the aggressor. Do you feel that way sometimes? It does feel like that's a way that sometimes in the sense of there are people that have weaponized vulnerabilities and are using them in the wild. And in a way, the defenders that are at Microsoft, Google and other companies don't care who these people are or why they're doing it.

We just want to find what they're doing and take those tools away from them. And so every time a zero day is found in the wild by some defender organization and it gets patched, that is a happy day for me. Ooh, what a strange thing to think about. Does that put you in deep thought too? I mean, the truth of the matter is that the NSA is actively looking for vulnerabilities in Windows so that they can use that against their adversaries. And then here's John actively trying to figure out what the NSA is up to so he can basically expose their secret weapons. I don't know what to think about that. I don't know who the bad guy or the good guy is in this. NSA is supposed to be working towards keeping our country safe. But at the same time, they have to develop cyber weapons to attack other nations. So it

almost seems like the NSA would see John and Microsoft as the enemy. And John might see the NSA as the enemy. And I just never thought about how these two would be battling each other like this. It's just wild to think about this relationship. Yeah. And then in many ways, I feel like I relive this whole moment a couple of years ago when the Eternal Blue exploit was discovered. This is one of these NSA tools that the Shadow Brokers group leaked onto the Internet. A patch was produced for it. That was MS-17010. And a couple of months later, the WannaCry worm was unleashed and spread in a very similar way.

It had a more destructive payload and ran across the globe against systems that had not patched. Now, if you recall in earlier episodes, NSA actually did tell Microsoft about EternalBlue just before the shadow brokers published it to the world, giving Microsoft an early warning who were able to move quickly and patch this before it was released. I'm guessing the NSA knew it was going to be published and wanted to help the world just stay slightly ahead of the game. But that must have been an awkward phone call or whatever. For Microsoft to get the memo that NSA has found a devastating exploit in Windows and this exploit got leaked

to shadow brokers, and they're about to leak it to the world. I don't know. I don't know. It's a weird, tangly mess when you get into the relationships between NSA and Microsoft. And just to be clear, there's no link between MS-08067 being found by the NSA. Oh, something happened last week that's kind of interesting too. Last week was Patch Tuesday. And boy, was it a doozy. There was a bug patched. It was a cryptographic API bug in Windows 10,

which basically allows an attacker to pose as someone they aren't, and your computer could send trusted information to it. But here's the thing. This bug was reported to Microsoft by the NSA. In fact, it was so important, the NSA even held a press conference to urge people to patch. This is very rare. I mean, we don't know how many times the NSA has reported bugs to Microsoft. They could be doing this all the time. But we do know for sure that there were two times that they did do it. Once when Shadow Brokers got the NSA's exploits, and now again last week. The NSA says they told Microsoft because they want to build more trust with people and help keep computers secure. Hmm, really? It could be that. A world is changing and new things are happening, and the NSA might be doing this from now on to try to keep the country safer by working with vendors to get things patched. And in fact, they have done stuff like that for a while. But it's all been small potatoes versus

like the God mode bugs that the NSA keeps to themselves. So my theory is this. The NSA gave this bug to Microsoft. Why? Maybe it was just to build better PR. Okay. But then maybe the NSA knows something we don't. Like maybe they they uncovered a huge man-in-the-middle campaign that some foreign government was doing to many Americans and thought it could have devastating results, so this was their way to stop it. Or maybe the NSA lost another exploit and didn't want their enemy to have it. I don't know. But I have a feeling there's more to this story, though. So, back to Conficker.

You might be wondering what happened there. Like I was saying, Conficker infected 10 million computers, and it was growing. However, it was a mystery as to what Conficker actually did, or who made it, or what it was supposed to do. While it was infecting systems worldwide, it was apparently not doing anything once it was infected. On a periodic basis, the computer that had Configure running on it would reach out to certain domains to receive instructions on what to do next. But it just didn't get any instructions to do. Security teams all over the world feared that maybe there were instructions to do on a particular day. And in fact, for some weird reason, we thought that on April 1st, 2009, there was going to be some big surprise

that Conficker was going to give us. I remember being in the office that day and setting up a conference bridge in a war room with everyone from IT and looking for signs of Conficker kicking up or something. But nothing happened. A few companies got really fed up with Conficker spreading everywhere,

so they decided to do something about it. In February 2009, something called the Microsoft Cabal was formed. This included people from Microsoft, Verizon, New Star, America Online, Symantec, F-Secure, researchers from Georgia Tech and the Shadow Server Foundation, and so many more organizations. It was a huge list of companies that came together to figure out a way to stop Conficker. They would do things like reverse engineer Conficker to see how it worked and then write fixes to block Conficker from

spreading more. But then whoever made Conficker would change how it was infecting machines so it could keep infecting machines, creating a new variant of the worm. It became a game of cat and mouse with the security professionals blocking it and the worm creator getting around that. At some point, Microsoft said they're willing to pay $250,000 for information that leads up to the capture of whoever created the Conficker worm. They were taking this pretty serious. The FBI started getting involved, and the hunt began to look for whoever was running this worm. The author, Mark Bowden, did some extraordinary research into this worm in his book, which is just titled Worm. He writes there are a few theories in what Conficker was.

One is that it was just a security researcher playing around, making a crazy worm, but never intended to do any harm with it, just trying to see how big it could get. Another theory was that a government created this worm and was waiting for instructions to maybe attack on command or spy on people or something. But as more organizations joined the Microsoft cabal, there was more effort put into looking into this, and they may have found the answer. They reverse engineered the code and did everything they could to trace it back to their creators. And they handed this information over to the FBI, who then arrested three Ukrainians, Sergei, Yevgen, and Dmitry.

These Ukrainians all were millionaires. They drove black Porsches and they lived in penthouse apartments. And their story was that they ran a website with employees and everything, and they paid themselves. But they weren't paying themselves very much. And so they were arrested on tax evasion charges. And the feds seem to have found some evidence of Conficker code on their work computers. But I don't think we have any idea what happened to the next. It doesn't seem like the FBI was able to extradite them to the. and they just disappeared into the Ukrainian courts. But the FBI also arrested one other guy, a Swede named Mikhail. Mikhail was arrested in Denmark in connection with this and extradited to the U.S. The court records don't say anything about Konfeker though. Instead, the FBI

found evidence that Mikhail was infecting computers and putting scareware on them. This is where he would infect a computer and say there's a virus on it and you need to buy this antivirus. But when the victim buys the antivirus, nothing actually gets fixed. The FBI claims Mikkel made $71 million from his scareware campaigns, which is a really big haul. To get that much, you must have had a lot of infected machines. And there was evidence in some of the variants of Conficker that it was capable of running scareware. So it might have been getting ready to launch a big campaign to do just that, but it never did. Mikhail got two years in prison for his scareware scams that he was running. And it's alleged that he had ties to those Ukrainians who also got arrested over Conficker.

So it seems like the best theory now was that Conficker was made by a group of Ukrainian cyber criminals who may have been planning on using it to send spam emails or running scareware to scam its victims, but they never got to it. And what's truly fascinating about Conficker is that it's still out there infected on a ton of systems. Even though MS-08067 was patched in 2008, there are still computers out there that are running systems older than that, that haven't

been patched still. And the latest estimate is that Conficker is still present on 400,000 computers today. MS08067 will go down in history as one of the most notorious vulnerabilities in Windows ever. And the reason for it is because of how effective it is. I personally love playing around with this vulnerability and exploiting Windows computers with it because it's so easy to do. And I want to walk you through how I've done it. First, you need a version of Windows from before 2008, which is actually quite easy. You just install Windows XP on a computer and don't patch it. This will be vulnerable to it. Then you need to run this exploit against it. Now, instead of knowing what shell code to send to the computer and work all that out, there's a crazy shortcut. It's called Metasploit.

Metasploit is an incredible hacking framework. It has over a thousand exploits, all pre-programmed and ready to run. So you pick the MS-08067 exploit, then point at your Windows XP machine, type run, and boom, you're in. Now when, I mean you're in, you're really in. Metasploit has tools to allow you to use that computer

you just infected as if you're right in front of it. You can tools to allow you to use that computer you just infected as if you're right in front of it. You can run any command you want on that computer, all through the command line, take screenshots of the desktop, enable the mic, enable the camera, run a keylogger to watch what someone types. You can do all that and more. Metasploit is an amazing hacker tool, which is a standard for any hacker to know how to use today. And the best part about Metasploit is that it's free and open source. Anyone can grab it, study a few commands, and have over a thousand exploits ready at their fingertips. It's really powerful and fun to play with. And if you attend any ethical hacking training, chances are you'll be given Metasploit and a system vulnerable with

MS-08067 as one of your first hacks you'll do using it, because it's pretty easy to use and you can see how effective Metasploit can be. So because of that, penetration testers all over the world are very familiar with MS08067, and all have that number memorized. However, it's now 2020, so MS08067 is a dozen years old now, which means there are far fewer computers running Windows XP or that are vulnerable to this attack. So this bug is losing its notoriety. It's much more rare to find a system vulnerable to this today, but they do exist. This story is another example on why it's so important to update your software as soon as you can. However, it's not always that easy. Some networks have very strict controls, like they can't patch. A patch might break everything. The applications and software running on the computers doesn't always

work with the latest OS updates applied. And this quickly becomes a nightmare. I recently updated my home computer and a lot of the software that I was running stopped working. So I had to wait for each application maker to put out an update for me to be back up and working again. Something like this is totally unacceptable in critical networks like hospitals or power plants. So it's not always as simple as just patching it. Like I was saying at the beginning, it's a lot of different people's jobs to keep our network secure. In this case, Microsoft did their job at finding a bug and issuing a patch for it, and now that fix needs to trickle all the way down to everyone's computers. Because by being on the most up-to-date version, it is like giving yourself a vaccination to render yourself safe from the known attacks in the world.

Updating your apps, operating systems, and programs, in my opinion, is the single most effective thing you can do to protect yourself on the internet today. A big thank you to John Lambert from Microsoft for coming on the show and sharing the story with us. I found it to be really cool. Also, if you want to know more about Conficker, check out the book Worm by Mark Bowden. It goes into a lot of details about it. It's pretty good. Hey, if you're all caught up with this podcast and want more episodes, check out the Darknet Diaries Patreon page and you can find bonus episodes there. This show is made by me, the Cyber Ghost, Jack Recider. Music in this episode was special. Typically, I grab songs from all over, but in this one, every single song was created by the top talented Breakmaster Cylinder.

And even though hoodies go up and drawstrings get pulled tight every time I say it, this is Darknet Diaries. We'll see you next time.