Darknet Diaries - 91: webjedi

Episode Date: April 27, 2021

What happens when an unauthorized intruder gets into the network of a major bank? Amélie Koran aka webjedi was there for one of these intrusions and tells us the story of what happened.You can find more talks from Amélie at her website webjedi.net.SponsorsSupport for this show comes from IT Pro TV. Get 65 hours of free training by visiting ITPro.tv/darknet. And use promo code DARKNET25.This podcast is sponsored by Navisite. Accelerate IT transformation to respond to new demands, lower costs and prepare for whatever comes next. Visit Navisite.com/go.View all active sponsors.Sources https://www.foxnews.com/story/0,2933,435681,00.html https://w2.darkreading.com/risk-management/world-bank-(allegedly)-hacked/d/d-id/1072857 https://www.washingtonpost.com/nation/2020/05/18/missionary-pilot-death-coronavirus/ https://webjedi.net/

Hey, it's Jack, host of the show. Now, a lot of you write to me and tell me your favorite episodes are the ones with social engineers or penetration testers. Yeah, sure, being on the red team is fun to break into things. But my heart is with the blue team, the defenders of the network. Because that's what I did for 10 years professionally. I was configuring firewalls, intrusion detection systems, and reviewing logs to find threats in the network. I felt like it was my job to stop or restrict bad things from happening in my clients' networks. It was a game of cat and mouse. I had to learn what the bad guys knew so I could stop them.

And I'll tell you, it was exciting. At times, it felt like the battle at Helm's Deep with a never-ending onslaught of attackers, and I had to embody Legolas to defend them off one at a time. Now, this story is about a defender and how she uncovered a serious breach in the network of a major bank. These are true stories from the dark side of the internet. I'm Jack Recider. This is Darknet Diaries. This episode is sponsored by Delete Me. I know a bit too much about how scam callers work. They'll use anything they can find about you online to try to get at your money.

And our personal information is all over the place online. Phone numbers, addresses, family members, where you work, what kind of car you drive. It's endless. And it's not a fair fight. But I realized I don't need to be fighting this alone anymore. Now I use the help of Delete.me. Delete.me is a subscription service that finds and removes personal information from hundreds of data brokers' websites and continuously works to keep it off. Data brokers hate them because Delete.me makes sure your personal profile is no longer theirs to sell. I tried it and they

immediately got busy scouring the internet for my name and gave me reports on what they found. And then they got busy deleting things. It was great to have someone on my team when it comes to my privacy. Take control of your data and keep your private life private by signing up for Delete.me. Now at a special discount for Darknet Diaries listeners. Today, get 20% off your Delete.me plan when you go to joindeleteme.com slash darknetdiaries and use promo code darknet at checkout. The only way to get 20% off is to go to joindeleteme.com slash darknetdiaries and enter code darknet at checkout. That's joindeleteme.com slash darknetdiaries. use code Darknet.

Support for this show comes from Black Hills Information Security. This is a company that does penetration testing, incident response, and active monitoring to help keep businesses secure. I know a few people who work over there, and I can vouch they do very good work. If you want to improve the security of your organization, give them a call. I'm sure they can help. But the founder of the company, John Strand, is a teacher. And he's made it a mission to make Black Hills Information Security world-class in security training. You can learn things like penetration testing, securing the cloud, breaching the cloud, digital forensics, and so much more. But get this.

The whole thing is pay what you can. Black Hills believes that great intro security classes do not need to be expensive and they are trying to break down barriers to get more people into the security field. And if you decide to pay over $195, you get six months access to the MetaCTF Cyber Range, which is great for practicing your skills

and showing them off to potential employers. Head on over to blackhillsinfosec.com to learn more about what services they offer and find links to their webcasts to get some world-class training. That's BlackHillsInfosec.com. BlackHillsInfosec.com. All right, so you ready to get into it? Yeah, sure. All right, so let's to get into it? Yeah, sure. All right, so let's start with what's your name?

My name is Amelie Curran, and that wasn't always my name, but that's a story for another time. What are you known as online? My handle's Web Jedi, although back when I was in the BBS world, it was Thunderball because I was a big fan of James Bond. But the Web Jedi moniker has been mine for probably a quarter century. So I'll stick with that. That is a really cool name. How did it come along, Web Jedi? So I entered college back in the early 90s, in 93.

And being the nerd that I am, I'm a big fan of science fiction and particularly Star Wars. And about that same time is the birth of the web. So, you know, 93, 94, you saw the first couple of websites show up online. So this was back in her college days. She was going to school at Carnegie Mellon at the time, where she was studying electrical engineering

and computer science. And she was building this website on the school's computers. It was the Star Wars Multimedia Archive. And she was building this website on the school's computers. It was the Star Wars Multimedia Archive. As she was building this site, she was looking around at what other websites did. And there weren't actually many on the internet at that time. So I, you know, busted my chops to create a fan website, you know, one of the first thousand websites on the internet. One of the things she noticed is people who created websites

were referring to themselves as webmasters. And she thought about this term, webmaster. What's better than a webmaster? So I figured, you know, Jedi master. So I took WebJedi. And so WebJedi became her screen name. And that still carries over to today. I mean, WebJedi is her Twitter name now. And for the longest time, my email address was jedi at cmu.edu. So I kind of stuck with that for the time I was there. She was really fascinated with the internet, learning all kinds of stuff while at CMU, even doing things that weren't taught in class. But yeah, no, it was a lot of the cases of, you know, learning a lot of technology

stuff that wasn't getting taught in classes. So I use this as my testbed to learn how to set up a web server, a mail server, you know, security permissions for files, you know, setting up network ports on shared services. It was a great learning tool that, you know, definitely got me interested in stuff well beyond what I was being taught in school. She was getting really geeky with it, practicing doing things on computers, on her free time, and picking up all kinds of new skills. But since I was such a poorly skilled programmer,

I failed out of my CS classes rather hard and switched into a social distance sciences degree. And that's what I graduated as. But I actually spent more time as a computer engineer than I actually did as a social scientist. So I had a good mix of, you know, the policy theory and information system stuff from the social sciences team. And then, you know, knew all the technical stuff of hardware and analog circuits from the ECE courses. So it was a really weird thing to kind of graduate as skill wise. But her heart was in tech and computers. So she pursued jobs as a computer engineer. At first, she got a job as a user interface designer at Xerox. Then she got a job at a different company being a system administrator, where she was taking care of the servers in the network, updating them, configuring them, keeping them going.

Then she moved on from there. And just kind of worked up from there. You know, it was securing web servers for the American Chemical Society and running those. Eventually, I ended up to go work for Stan Lee of Marvel fame, you know, running his IT shop when he left Marvel. She then moved out of California and got a job at a utility company. They provided gas and electric to people. It was during a very interesting time because I think within the first year I was there, we had a major hurricane roll through and knock out a good swath

of power up into the Chesapeake Bay. Shortly after that, we then had the Northeast Blackout, and it taught a lot about designing for resiliency, but also just generally, you know, how do you work at scale when you don't have all the resources that you typically would have? She says she was impressed at how this company learned from their mistakes. Yeah, sure, a hurricane is not a normal event. But they knew that another one might happen again someday. So it's best to build a more resilient network in case it does happen. So they redesigned the network and built out a pretty robust data recovery center,

a secondary place that can handle the full load in the event that their main data center would go down. And I think it came into full effect is that they had a bathroom explode one day and actually flooded parts of the training floor. So that practice kept them running without having to worry about, you know, if those procedures had actually worked again. But, you know, working for a critical infrastructure company such as a power company, the mantra was just be a less appetizing target than the next network down the road. So a lot of what we built was on detection, not necessarily response. And I learned a lot there, and I carried that forward to where my career is now.

It was here where she really got into DFIR. So DFIR stands for Digital Forensics and Incident Response. This is the team to be called in when there's an incident, and they'll handle the situation. I sometimes like to think of the DR team like Winston Wolfe in Pulp Fiction. You're Jimmy, right? This is your house? Sure is. I'm Winston Wolfe. I solve problems. Good, we got one.

See, in big companies, incidents are never handled by one person. First, you have the people in the operations room who first saw the alert and notified somebody. Then you have network engineers and system administrators who are engaged in investigating it further. You might get someone from leadership entering the room, asking tons of questions and wanting updates, and is there to make decisions. And then you might also have a bunch of angry customers who want to know why their power is out. And so customer support might be looking for updates, too. The operations center quickly becomes a mess during large incidents.

And so the DFIR team steps in to get things under control. They get the latest updates and then disseminate that information to everyone who needs to know. And they'll get the right teams engaged to get things under control and work with leadership to present the details and work out any big decisions that need to be made. So while Amelie was doing this type of work at this utility company, she was also a security engineer and an architect there too. She learned a lot from there, but then left that place to get a job somewhere else for a bit. She got bored there and then went to go work for Mandiant. As their first full-time IT manager. Mandiant is a security company focusing on incident response and threat intelligence. And recently they were acquired by FireEye. They were not the Mandiant that most people

know about today. They were small, if you call like 60 people small. But we had, you know, three or four people sharing a cube desk. We had, you know, a lab that was literally a closet. You know, we're working a bunch of different major cases that people look back at. And those are the things that, you know, you remember Home Depot, you remember TJX, you know, those were cases that they worked, but this is back when they were much smaller. After Mandiant, she went to work for the World Bank. With thousands of employees and 189 member countries, the World Bank is one of the most powerful institutions in the world, funneling billions of dollars every year into ending global poverty. So if you're not familiar with what the World Bank is, let me give you a quick catch up. It was created after World War II to loan countries money to help rebuild after the war. After that, they continued to loan countries money to build bridges and other

infrastructure for the nation. So countries around the world owe money to the World Bank. World Bank is actually an NGO, a non-government organization, and because of that, it falls under different regulations and laws. And now their mission is to help people in extreme poverty. They help fund projects around the world to try to combat extreme poverty. But it's also still a bank which issues bonds and loans and stuff to people with interest. And that's how they make their money off this is because all these loans have interest. Now, the headquarters of the World Bank is like five blocks away from the White House in Washington, D.C. And while the World Bank is like five blocks away from the White House in Washington, D.C. And while the World Bank is a non-government organization, they also have connections with the government in some ways.

For instance, the president of the United States of America can nominate who the president of the World Bank should be. And if the board agrees, then that person becomes the president. So on March 16th, 2005, Thank you for giving me a chance to come by and say hello. then that person becomes the president. So on March 16th, 2005, Thank you for giving me a chance to come by and say hello. President Bush holds a press conference. Preparing for my trip out of town for Easter. And nominates Paul Wolfowitz as the president of the World Bank.

Paul is committed to development. He's a compassionate, decent man who will do a fine job in the World Bank. Now, let's back up a second. During 9-11, Paul Wolfowitz was the Deputy Secretary of Defense. That's the second highest ranked person in the entire Department of Defense. He was an early advocate for the U.S. to invade Iraq under the belief that Iraq had weapons of mass destruction. And so it was a little odd to hear about this nomination. I mean, on one hand, you have Wolfowitz attacking Iraq. And then what? On the other hand, he might loan them money to rebuild the country?

It just seemed really odd. But here's how one reporter asked a question to President Bush. Paul Wolfowitz, who was a chief architect of one of the most unpopular wars in our history. That's an interesting start. It's your choice to be the president of the World Bank. What kind of signal does that send to the rest of the world? First of all, I appreciate the world leaders taking my phone calls as I explain to them why I think Paul will be a strong president of the World Bank.

Well, Paul Wolfowitz didn't last long as the president of the World Bank. Two years into this role as president... President Paul Wolfowitz was at the center of a scandal over alleged favoritism. By arranging a promotion and pay raise for his female companion, Shahariza, his credibility on this and other issues was undermined. At issue for many member states was Wolfowitz's effectiveness in the wake of the scandal. Former World Bank official Gene Rotberg. If the countries say your effectiveness is damaged, and if the staff say your effectiveness

is damaged, then by definition you are damaged. From the start, Wolfowitz was the center of controversy as bank chief because of his role in planning the Iraq war when he was a top Defense Department official. Much of the bank staff held this against him, according to Rotberg. It is Iraq. He is looked upon as one of the persons who formed both the intellectual and practical support for that war, that it had cost hundreds of thousands of lives. That is what the staff thinks.

Yeah, for reals. If the World's Bank mission is to help people in extreme poverty, having a president who architected the Iraq War and now at the center of a scandal where he helped use his position as president to get his girlfriend a job in the World Bank? Yeah, he was fired. Or I guess the term is that he was forced to resign. I regret that it's come to this. I admire Paul Wolfowitz.

Anyway, that's what the World Bank was like when Amelie got a job working there back in 2008. A new president had just come on board and she was hired on to. I was just an information security engineer. I was just basic utility player. I was a contractor, so I was just kind of like plug you in here, plug you in there. Now, at the time, Amelie was a meticulous note taker. Well, you know, this being back in, you know, pre-iPhone, pre-iPad or anything like that, you know, no one had like tablet computers. So, you know, it was just normal for an incident handler to walk around. Every time they had a

case, it was their paper notebook, a Stenopad that you were frantically taking notes on. There was no real easy way to do this and secure it. I think that's one of the other challenges, too, is a handler's notebook is their Bible. And I just wanted to mention her notepad here, because for her to retell this story that happened back in 2008, she brought these old notepads out to confirm the story. You know, flipping back through some of them, and I don't know why I ended up having these, but it was just given to me as my box when I left out and they didn't ask for any of the stuff back. So it is what it is.

For this story, she finds the date in her notebook where all this started to happen. I think when I look through my notebooks, you know, stuff I was looking at before was, you know, some full disk encryption stuff. But, you know, my notebook ends, you know, looking at BitLocker and then immediately the next page is, here are all the notes that just got handed to you about this incident. And it was like, okay, I'm drinking from the fire hose here. What was handed to her was a very serious security incident going on in the World Bank. So what had happened from my best memory, and this is mainly because I was on the outside for the first two weeks of this, was something got triggered on a log. Someone saw some weird traffic happening and that spun up some folks

to investigate it. And this was a team of probably about five or seven people that were in a conference room down the hall for me. You were in Washington, D.C. at the time? Yeah, this is Washington, D.C. But that's where you were, right? Yeah. Yeah. Okay. And, you know, they just noticed there was some weird traffic and, you know, they saw some stuff through, I believe, some basic integrity checking that some stuff had changed on a server that was not supposed to generally be touched by people. Yeah, file integrity monitoring. This is a helpful tool that companies use to monitor security problems. This is where a system checks all the important servers in a network to make sure nothing has changed on it that shouldn't have changed. So like if there was a configuration

change, the file integrity checker would notice that and create an alert. And the monitoring team would now have to go to the system administrator and ask, hey, did you make a change on this server? And if that system administrator did, then everything would be okay. But in this case, the file integrity monitor triggered an alert, and it showed that someone had made changes to a server which wasn't made by the system administrator or anyone else in the IT team. This meant that some unauthorized person had been inside one of their servers in the World Bank and was doing stuff to the servers.

This is where Amelie started getting pulled into the incident to see if there was a way she could help. She took a look at the systems that had unauthorized changes. Some of the triggers were which machine was hit. And one of the machines that got touched was our HSM. An HSM is a hardware security module. It's a device that specifically does cryptographic computations, which means this is the device

that does their encryption, decryption, and authentication. Essentially, our locker for all our cryptologic material. That server was shown to be touched. And that was like, okay, they were going after the crown jewels. And that, I think, was probably the triggers when that machine name came up in that list, that was immediately like,

this has gotten bigger than just like a couple database servers being touched. The company quickly put everyone in IT to work in this incident. But the amount of work that everyone needed to do was staggering. There were tons of logs to look through, systems to analyze, connections to review. And this is where Amelie started getting involved. They had one full-time forensics guy. And the folks who were running the incident handling group before I was called in were basically saying, image everything. Which is a good first thing to do in this situation. It's really just like making a copy of everything on that server to analyze offline. Because an attacker might erase their tracks or systems

might change. So having a copy of an infected machine is great, but there's a sort of fog of war when you're dealing with an incident like this. You can't really see all of what the attacker did, so it's hard to know how big of an issue this really is, and it's hard to know where to even look for clues. After looking through the logs and alerts, they had evidence that this attacker had accessed and changed configurations on 30 different servers in the bank. And so they were taking images and snapshots of these 30 computers. But they only had one guy with one computer

who was capable of analyzing these images to look for clues of malicious activity. And to analyze one machine might take hours and hours, but that's just for a computer to analyze it. For a human to analyze it, it takes even longer. So the process was going very slow. And at the same time, some of the employees were in a panic, stressed out to the brim over this incident. People from management were also filled with anxiety when things were not moving as fast as they wanted. Emergency meetings were spun up to try to get people to

move faster and leadership was freaking out. I was coming back from lunch one day and got a conference call about this thing. And I, as a contractor, and you had the CIO and the CTO on the call and the CISO. And as a contractor, I said, calm the fuck down. And literally the entire phone line just went silent. And that's what is needed sometimes. A good incident handler and learning from these experiences is to maintain calm. You know, throughout the story, you know, their initial two weeks, hair on fire. But when you have somebody who is giving them good information, giving them actionable things they need to do, helping them solve their problem, that's the best thing an incident handler can do is to maintain a sense of calm and transparency. And if there's anything

that anybody walks away from this with is, you know, that's the thing that makes a good incident handler. Not what cool tools you know or anything like that is just, you know, speaking the truth, sharing evidence and being having a cool head. I've often find that when people are handling incidents like this, they sometimes go through all the stages of grief. You know, first, you're shocked that a hacker got into your network. And then you might deny that it happened like, whoa, no, no, no, no, no, no way they got into my server. That's crazy. And then when there's no denying it, you might feel angry that it happened. Like, whoa, no, no, no, no, no way they got into my server. That's crazy. And then when there's no denying it, you might feel angry that it happened. And then you might bargain like, well, at least they didn't get into that database over there. But then when the

reality hits that all this is really happening, you might feel depressed. Like this is such a big problem that maybe we'll never solve. And once you process all that, can you really accept the situation and move on? And as Amelie puts it, your lunch has already been eaten. And because she's dealt with this kind of thing so many times that she can quickly move to this acceptance stage and just start working on solutions while others might still be busy dealing with their emotions. So Amelie was now fully immersed in this problem. If you're an incident handler and you're thrown into it, if you're not the one who's actually on the detection, and a lot of times that could occur with another team. So, you know, network team or server team or something like that. Yeah, you have fire hoses aimed at you of information coming from

every which direction. And it's a matter of which one you're going to turn and open your mouth to. And in these cases, you know, coming in two weeks late, it was like, give me the dump and give me an afternoon or a day to kind of sort through this. And if anybody's got any inferences, you know, try to summarize them. Unfortunately, this team didn't have it, so it was literally like, give me what you got. And I sat down in my office and tried to pour over as much of the data I could and try to make sense of it. So it was literally like, give me what you got. And I sat down in my office and tried to pour over as much of the data I could and try to make sense of it. She knew just what to do in a

situation like this and started asking for forensic images and logs to review. And being that I was, you know, called in two weeks late, you know, getting a memory image was near impossible. So we were working with a lot of imperfect information. By the time I got a chance to call for stuff from, say, ArcSight, which was their log repository, and even Mazu, which I think was their NetFlow logging tool, they had such little online storage that, you know, most of that evidence was gone. So it was playing catch up with stuff that was disappearing, trying to grab sand as it was flying through your hands.

So time was very much of the essence, and trying to narrow down what we needed to pull was also very important because, again, time was of the essence. Because when this was handed off, I had no idea if the aggressor, the attacker,, you know, whoever was in the network was still on the network. On top of her doing all this incident handling, the bank realized they needed even more help. So they called Mandiant up, which is where she used to work. But they called them up just to help with incident response too.

Mandiant had a whole team of incident responders ready to be deployed on site to help troubleshoot major attacks like this. But it would take them a few days to arrive. So Amelie stayed busy working on the issue in the meantime. I was literally trying to map out a picture of what the entire incident looked like. So looking at log files, looking at servers that they may have already identified and drawing out a map. Like where the hops were, who, you know, who was affected, when did they do this, what was the timeline. And I spent most of that weekend, I got approved for extra hours to work that entire weekend.

And I think I was, I think I was doing this on a Mac. So it was like probably like the first version of OmniGraffle that was ever released. So I'm doing, you know, all the stuff in Visio and OmniGraffle, you know, mapping out all the paths. So it was just like, it was like a board game. You just kind of watched, you know, they got on this server and then they went lateral to these two servers and they touch these files. And, you know, it really does end up occasionally looking like a digital version of that yarn and pushpin thing. By that time, the World Bank was having daily meetings, a war room, if you like, to bring everyone up to speed on the latest with this incident and to make decisions on what to do next. While a lot of people were working on this, only a small group had access to the details of this incident.

As we were initially handling this response, and it was probably timed about that two weeks when I was called in. There was a story that ended up getting leaked to Fox News that had particularly detailed recounting of a lot of what was going on with our incident response. As if someone was in the room was also leaking stuff to the press. That's not good. Typically, when you have an intrusion like this, you want to be very careful how you publicly disclose this.

The wording needs to be precise. And at the very least, you want to be able to control the messaging that the press knows. But on top of that, they just notified the hacker that the bank is on to them. There's just a lot of pressure of trying to stem the bleeding and make sure that, you know, the message is controlled. So someone had spoken to a reporter at Fox News and told them about this incident. Here, I'll read the article for you. The headline says, World Bank under cyber siege in unprecedented crisis. Then the story reads, it is still not known how much information was stolen, but sources inside the bank confirmed that servers in the institution's highly restricted treasury unit were deeply penetrated with spy software last April.

Invaders had full access of the rest of the bank's network for nearly a month in June and July. In a frantic midnight email to colleagues, the bank's senior technology manager referred to the situation as an unprecedented crisis. Hmm. That's some pretty specific information that this inside source had leaked. Like they saw that email from the senior manager and they know information that was only discussed in that war room. Something strange was going on here. So there's an internal integrity group at the World Bank, which basically is a watcher for the watchers, I guess. They audit how the bank does programs. They're kind of like internal review groups and whatnot. And part of something I had to do when I first signed on as a contractor there was ask to go and investigate some news stories that were leaked out

that seemed to be originating from data that was passing through some of the executive branches there. Ah, so Amelie had already been looking for this leaker before this news story even hit. And now, even though the bank is in the middle of an unprecedented crisis, she's got to find out who this person is that's leaking information to Fox News. I think that earlier case involved

the Wall Street Journal as well. So there was an intersection between Fox News and Wall Street Journal. Trying to figure out who the leaker is, is like a game of Among Us, where you're trying to figure out who the imposter is. Who would have the motivation to talk to Wall Street Journal and Fox News? Who would have access to this kind of information to be in the war room where this stuff is discussed or to see those emails? Amelie started becoming very observant of everyone in the war room where this stuff is discussed or to see those emails. Amelie started becoming very observant of everyone in the IT department, trying to figure out who this leaker was. And stay with us, because after the break, she sets a trap.

This episode is sponsored by SpyCloud. With major breaches and cyber attacks making the news daily, taking action on your company's exposure is more important than ever. I recently visited SpyCloud.com to check my darknet exposure and was surprised by just how much stolen identity data criminals have at their disposal. From credentials to cookies to PII. Knowing what's putting you and your organization at risk and what to remediate is critical for protecting you and your users from account takeover,

session hijacking, and ransomware. SpyCloud exists to disrupt cybercrime with a mission to end criminals' ability to profit from stolen data. With SpyCloud, a leader in identity threat protection, you're never in the dark about your company's exposure from third-party breaches, successful phishes, or info-stealer infections. get your free Darknet Exposure Report at spycloud.com slash darknetdiaries.

The website is spycloud.com slash darknetdiaries. Amelie starts making a list of names in her notebook of who this leaker might be and crossing off names that just don't seem possible for it to be that person. She had already started researching this before. This was actually the second case of someone leaking stories to the press. So, you know, that original case, and I had notes for that one too, we're looking at some linguistical analysis of all things, finding out what people, how people wrote certain things, you know, looking at quotes

from the Wall Street Journal and Fox News, and then looking at email, as well as documents that may have been, you know, accessed and, you know, seeing who had accessed them. And then with the emails, seeing how these people's quotes were. So when you'd see stuff that was called out, you know, we could search the mail system and try to find out, you know, where those particular quotes came from. And, you know, with this incident, you know, the data that was getting leaked out was, you know, making it out to be like, you know, we had Keystone cops running this response. So parallel to all this, because obviously we knew that the incident response team was potentially compromised, it was almost like incident response inception. So I was actually

investigating the investigators who had, you know, kind of tripped upon the fact that we had this data getting leaked out. Imagine being in that incident war room and your eyes come up over the laptop and you gaze around the room, sizing up everyone, wondering if they're the leaker. And at the same time, they're looking back at you, wondering if you're the inside source. Amelie got an idea. We set a trap because we had kind of narrowed it down to a few suspects. You know, due to the prior investigation, we had, I think, probably about five or six people that we knew possibly were the leakers. Her suspicion was that it was someone inside the IT department. We decided to set up a honeypot.

So we had a conference room where a lot of this outbriefing was done to the CIO and the CTO and some of the senior IT people in the World Bank. The debrief meetings were with leadership and senior people, and it was only a few people. Her hunch was that it wasn't anyone in these meetings, but it was someone who might be snooping in on one of these meetings, trying to find the latest news to give to a reporter. So when the meeting was over and everyone left, Amelie planted some fake information. And started writing some fake stuff on the walls. You know, put up some papers on the table and whatnot. And we kind of waited for those notes to start showing up somewhere. I love how low-tech of a honeypot this is.

Just a few notes left in a conference room. Well, now that the honey trap was set, they just had to wait. We saw some stuff that had popped up with the Fox News story a day later. So we'd done the honeypot. We laid it out and then we cleaned it up. And, you know, when we knew one of our suspects was in the office and then a day or two later, we saw it show up in a Fox News report. So we knew what they were reporting was wrong.

And we had narrowed it down to, I think, like two or three people. And eventually it was narrowed down to this one person. And we honeypotted a second time. And we actually had somebody kind of walk by the office of the suspect to make sure they were there at that time. And, you know, we got a message back. I think we were still using, you know,

Palm Pilots to send email and stuff at the time to tell you how long ago it was to let them know that, yeah, that person was in the office and they were on their computer at the time. And they were the only one that was alive in that hallway. So we kept an eye on him because, you know, sometimes the phrase goes, there's a useful idiot. But at the same time, they wanted to build this case further

to prove it was him. So now we had a suspect in a machine. You know, we could push out our remote forensic imaging elements out to them and grab an image of their hard drive. And when we eventually pulled it through, using stuff like NCASE and some other tools, reconstruct the web

caches so you could see the Yahoo emails that went out. So we had evidence that they were using their work computer to leak the information out, and it wasn't because we weren't detecting it in the mail system, which was Lotus Notes. We knew it was through webmail. So we were able to then, you know, kind of dive through the entire history of what he had sent out and found out that he was connected not only to the leaking of the information regarding this incident, but was also tied to some of the connections

with the prior leadership. Wait, the prior leadership of the World Bank? As in Paul Wolfowitz? Yeah. So when Amelie analyzed this IT person's computer, the leaker, she found that he was also working with another person about these leaks. This person who was helping him was an internal investigator also with the World Bank. They had an investigator that was, you know, one of the internal integrity investigators. You know, I had done an analysis on their hard drive. So I was the watcher watching,

the watcher watching, the watcher type kind of thing. And, you know, saw some of his personal items on there. What she found was evidence that this investigator was being blackmailed by the former leadership, which was Paul Wolfowitz or his team. And they somehow figured out a secret about this guy. Who was gay, but wasn't openly gay. And they were basically going to use that information as a leverage to out him. Remember, this was during the Bush administration, and it's a very conservative organization overall within the World Bank.

And that's what they were using to leverage. And I let my boss know that this is most likely the reason why they were blackmailing him. So to tie it all together, what Amelie believes is that Wolfowitz seemed to be a little upset that he had to leave the World Bank. And we believed he was trying to make the current leadership of the World Bank look bad. So his people were blackmailing this investigator to somehow find a way to make the bank look bad. And the investigator was just using the IT guy to get this information and to leak it to the press to make the bank look bad. And the investigator was just using the IT guy to get this information and to leak it to the press to make the bank look bad. In places like Fox News,

we're adding in all kinds of extra narratives, like talking about how the current president isn't doing as good of a job and stuff like that. D.C.'s very much along smear campaigns and whatever kind of leverage, no matter who it can hurt, this is how this town operates. It's pretty sad. So she prepared all this for HR to handle and her boss. The investigator, I think, still was there, just probably waylaid a little bit. I didn't see much

of him. It was more or less that I'm sure the pestering by the outside folks went down quite significantly, but I was told that that was handled at a level above me. The IT person was eventually let go. Okay, problem solved. What a relief. Time to take a break. Oh, wait a minute. No, we can't. This bank is under attack. Remember, there's a hacker that got into 30 servers in this network. And this was all going on at the same time. What a mess to try to handle two different incidents at once. All this was just putting a

lot of extra stress on people. That first weekend, you know, it was me going, you know, working as much as I can and then going and getting something to eat and maybe going home. I needed to sleep in my own bed, but I think the next day when I came back I brought a pillow and a blanket to sleep under my desk so I could just continuously work through the day.

Eventually, I just left a pillow and a blanket there because I just figured this was just what life was going to be like for a while. Sleeping under your desk is not fun at all. Okay, so back to the network intrusion. We eventually got to the point where we're actually, you know, pinpointing machines of interest. So we had 30 machines involved. But to kind of only kind of forensically look at a couple, I think we narrowed it down to like seven machines. And, you know, I had a list of, you know, these are the ones that you have images for and these are the things we're looking for.

A lot of this was done in parallel. So we had the forensic images of, I think, those five to seven servers. But at the same time, knowing that, you know, we instrumented relatively well, but there was a lot of data that started to disappear, like the Mazu gateway data, that was the NetFlow stuff, and that the Arc site was, you know, no longer online. You know, it was trying to piece together the story from logs,

trying to figure out, like, all right, so here's the map of what happened. And what do the logs tell us about this and how they got in? So it was more or less, a lot of the forensics were confirmation of our inferences. So as we started to pinpoint what got accessed and whatnot, we still didn't know the motivation.

And if I remember correctly, we didn't actually end up getting to where we found the motivation as to what they were interested in. Most of this came down to, you know, what tools did they use? How did they gain access? And, you know, what machines do we need to reimage, what data do we need to be on the lookout for if it's going to be used by an adversary, and so forth.

What was it that was compromised? As it came down to, it was some machines associated with the HR system. Hmm. Okay. You know, a bank has a lot of money, so you'd think that a hacker going through all the effort of getting into a bank, it's a kind of surprise that they were going after HR. Yeah. The thing was with the data that was on there,

there was a little bit of the HR stuff, and when you start asking questions as to figure out like what all the connections are, you know, they're shared databases. We considered it was probably HR, but there may have been some other databases that they were interested on there. At the point in time is that we were mainly, the leadership was mainly interested in how they got in and if they could clean up. I'm curious about this too. How did the hacker get in? One of the issues was with the bank is that they had abandoned a multi-factor access program.

So not only were they not using like, you know, a hard card or a smart card for credentialing for all users, the users between Enterprise Admin and the regular old, you know, every, you know, Jane or Joe user on the World Bank Network, you know, it was just a matter of your user ID and your password. So our Enterprise Admin, one of our three Enterprise Admins for AD's account was compromised. All it takes is the right username and password to get in sometimes. And you might ask, why was multi-factor authentication turned off?

Well, they did try multi-factor authentication, but for whatever reason, they didn't like it. It was too slow, too complex. It was impacting how business worked. So they removed it. And we're in the process of switching everyone to use smart cards for authentication. This is where you have to insert a little credit card like thing and then type in your password to get into a computer. And so they turned off the token method for authenticating users. And we're in the process of switching everyone to use these physical cards. So it was just really bad timing that the hackers

got in during that transition. Once the hacker got in, they tried running a hacker tool, but the antivirus on that computer blocked it. So they tried another exploit, a newer one. Even though this computer had antivirus running, it didn't trigger on this newer exploit. Once they were in one computer, they were able to traverse the network to get into other computers. The hacker eventually got their hands on password hashes.

Now you need to run a password cracking tool to figure out what the password is once you get the hashes. But Amelie saw that they had gotten these password hashes, and she knew they had the Enterprise admin's password. So she decided to get the hashes herself and see if she could crack the password. And was able to pull out the passwords for the enterprise admin rather quickly, I think in about five minutes. Aha. So if she can crack the password that quickly, that meant that the password was not very strong. This helped her connect a bunch of pieces together to know how the hacker moved around and got different things. So when she went to the closeout meeting with the head of IT to wrap this whole thing up, she decided to do a sort of magic trick for them.

I had Joyce Lin ask the enterprise admin for his password. Joyce Lin was from Mandiant, who was called in to help with this whole incident. And so the admin wrote the password on a piece of paper and then folded it so nobody could see it and put it on the table. At the same time, Amelie started John the Ripper, a tool used to crack passwords, and everyone in the meeting watched. And quite quickly, she had managed to crack the Enterprise

password. And then turned the screen around and I said, is this your card? And sure enough, that was the password that was written on that sheet. And Joyce showed the screen and that password. And it was a really simple password. I think it was his daughter's name with a year or something. It was a really bad password policy. So that instantiated some policy changes. They started doing some user account separation. A couple of months later, we had called in Microsoft to go through an entire analysis of the Active Directory

forest management for the entire bank. Yes, nothing like a good old-fashioned password audit. This is where you take the hash dump for everyone's passwords on the entire network and see how many can be easily cracked. And so you see some pretty bad behavior. Like a lot of people use the company name inside their password. And that kind of stuff might show up on the audit, which then might allow the security team to make new rules to restrict certain passwords in an attempt to make things harder to crack. On top of that, they finished

rolling out that smart card authentication for everyone. So this was a kind of a come to Jesus incident for them to try to get serious about increasing their security. They also brought in more FTEs or full-time employees to help do security. They built out a SOC, they developed an incident handling playbook, and they improved their security overall for the whole company to keep this from happening again. So as much as they were doing everything you could potentially do wrong before this, that it was enough of a punch that it got them to really kind of try to do something better. Now, the biggest question that always comes up from a hack like this is, who did this? And they weren't able to figure that out for sure. But there were a few clues that suggested that this attack came

from China. So they packaged up these findings and sent it to the World Bank leadership. The initial response was from the bank's CIO was, oh my God. And I think there was an explicative in there. She was very much more reserved than most, but she was literally ready to march up to the Chinese embassy or up there and around UDC and basically chew them out and say that they were going to pull all the bank funding from all of their projects. Yeah, this is interesting because the World Bank was loaning money to China for various projects. I don't know what exactly, but maybe a loan to build a bridge or maybe to help people in extreme poverty in China. But it doesn't matter what project. The thing is, is that the World Bank was directly helping China

and didn't like that China was behind this attack. Yeah, we don't know if they actually spoke to anyone in China about this, but it's certainly interesting if they did. Why that was significant was, is that the challenge with anything like this is attribution. There was stuff that wasn't shared with us once we, you know, as I mentioned, you asked like, well, why HR systems?

There was definitely something that triggered in the back of the mind of those executives that when we showed them that evidence, they knew that it was China or some, you know, Chinese asset that was looking to get this information because there was something higher up at the bank that was happening that us lowly contractors and other employees weren't party to. So it wasn't necessarily, you know, the way that typically you would, you know, map your TTPs to a particular actor.

So, you know, China has their threat actors with their names and stuff, and then Russia has theirs, and Iran has theirs, and North Korea has theirs. But, you know, some of that too, the reason we also were able to, you know, be somewhat confident other than the fact that the bank executives were like, yeah, we kind of, this makes sense. Where the Mandiant, you know, as part of the, some of the malware stuff that we had, we'd looked at using, using their mirror tool, which was the first release of their mirror tool

at the time, was that some of the, some of the stuff that we'd found on the system started matching some of their early work they'd had to map to some of the Chinese threat actors. So, you know, it all kind of came together. But this is very much in the early days, you know, like I said, 2007, 2008, very early days of being able to have some level of confidence that, you know that these were the teams that were particularly poking and prodding. So that concluded the investigation. It was time for Amelie to bring her pillow back home and sleep in her own bed again.

But honestly, Amelie loves handling incidents like this. It's the chase. It's the finding something new, finding something clever. The thrill of the chase. It's the, the finding something new, finding something clever. Um, uh, the, the, the, the, the thrill of the chase overall. Uh, you know, I kind of sometimes thrive on, on stuff. I've done forensic stuff where we had to do an entire imaging of a lab, uh, while people were out from the time work closed to the next day they walked in, uh, you know, trying to do it on the sly and just the challenge of like, we didn't have the right tools and walked in, you know, trying to do it on the sly. And just the challenge of like, we didn't have the right tools and we're improvising and that kind of stuff. You know,

you're running on adrenaline and endorphins and whatnot. But, you know, the whole thing, like somebody inside, this is an inside job. And then you have the realization, like, this is somebody I'd been investigating. So this is a much bigger picture. And then you start thinking about the political intrigue and you're just kind of like, I this is a much bigger picture. And then you start thinking about the political intrigue and you're just kind of like, I can't believe I'm here. And it just, it's really weird to be, you know, at the right place at the wrong time or the wrong place at the right time or whatever it is.

It just seems, you know, for me, I kind of run on that, I guess. I guess that's my MO. I was at the, you know, at the White House when we had the OPM breach and heart bleed and, you know, working at a power company during a blackout and a hurricane. And it's just like, I don't think anybody would ever want to hire me

because just like trouble follows. And it's not my fault. But, you know, the idea is you can kind of help these organizations fix stuff. And the emotion after that is like, all right, well, we did this cleanup. Your lunch was eaten. You know, we told you the truth. This is what happened.

But what can we do to get better? And, you know, if you're not somebody who loves the thrill of the chase, and a lot of IT people are builders, building back better, which is the mantra of the current new administration as of today. You know, the build back better is the thing that also thrills people like they get, you know, they get to tool up, they get to construct stuff, they get to do the things that really are exciting, they get to put good stuff into practice. And that's, that's another emotional high. So if you get to play all of that, man, it's great as a blue team person. You know, it's very frequently I ever got to red team. So maybe capturing flags and stuff are the big emotional high for red team people. But for blue team, it's like, yeah, I kept them out.

Or, yeah, we fixed this thing. And, you know, go ahead, try me. So, yeah, that's kind of how it feels. So that's it. This incident was all wrapped up, all with the help of Amelie and Joyce Lin. So, yeah, that's kind of how it feels. So that's it. This incident was all wrapped up, all with the help of Amelie and Joyce Lin. So Joyce Lin, she was the project manager from Mandiant. She unfortunately passed away in an aircraft crash.

She died in May of 2020. She was delivering medical supplies, medical and food supplies to, I think, somewhere in Southeast Asia. And her plane crashed. She was an Air Force reservist. The funny thing was is that after that, I ran into her when I was stationed at the Defense Cybercrime Center. And she was doing some reserve duty up there. And it was just like connecting old times.

So it's a small world. You never know who you run into. But when I saw her walk in the halls up there and I was like, I just smiled and she smiled back and we just knew, you know, we've been through hell. It wasn't necessarily in a foxhole anywhere, but you know, we knew and trusted one another. And it was a shame when I had heard from some Mandiant people that she had perished, but she'd led her life pretty well. Amelie kept working at the World Bank for a while, but something awkward happened, which made her leave.

What we had found out, because I was working late, late hours for a lot of these incidents that we had, and I usually just like working late in general because it was quiet, ended up starting to get suspicious of my boss. He would show up at weird hours and stuff like that. And he was having some issues with his Mac and the like. And we ended up kind of finding out that he was, after I left, found out that he was cheating on his recently pregnant wife with a co-worker.

And the fact that I was there late and seeing his coming and goings, I think he felt kind of threatened. So I was, my contract wasn't renewed, which is, you know, kind of a shame. I mean, the fact is, is that, you know, going through all this, this work and whatnot, and then, you know, getting let go because you're getting too close to something else that was, you know, trying to be swept under the carpet was a little annoying. But, you know, I felt that, you know, they'll do what they need to do. Then I became the chief enterprise security architect for Department of the Interior. And I was there for nearly five years.

Chief security architect for Department of Interior. That sounds huge. Yeah, yeah. Yeah, it was a big project to work on. And I took over, helped develop their mobile security program, some of the remote work policies and stuff like that. Lots of different things. And during my time there, I did a leadership rotation

as part of the President's Management Council at the White House at the Office of Management Budget and worked for the Chief Information Officer, the Federal Chief Information Officer. Oddly enough, I think within a week of me getting there is when they had Heartlead, the SSL, the OpenSSL incident, which was an interesting experiment in trying to explain to senior political officials how open source projects were governed.

Shortly after we kind of handled the Heartlead data call, there was some news that came out regarding USIS, which is one of the companies that was contracted out to do background investigations for the Office of Personal Management, OPM. And the news that came out was just that their contract was put on hold or terminated. And they performed about 50% of the background investigations. And then later on in the summer, Keypoint was also terminated.

And then a few weeks later, DHS released the notice that they had found some intrusions on the OPM network. But that wasn't necessarily when the OPM breach was disclosed. As you know, the timeline, the OPM breach was disclosed, I believe, in March of 2015. The fact is, is that things like a history of incident handling and response, when you start to look at what these companies did, how they were connected to the network over VPN and whatnot, back into OPM, it started looking like they were using these companies as a way into OPM's soft underbelly on their network. And that's exactly what ended up happening. So I had mentioned to the federal CIO at the time,

Steve Van Roekel, that this looks like you have a breach in progress at some point. Unfortunately, my rotation ended, but I at least let them know, like, this looked bad. So just be prepared that, you know, in time, this will probably get much, much worse. And at that point in time, I think they started the DHS work with OPM to do the investigation. And then that was disclosed in March. This incident where the Office of Personnel Management was breached was a major incident that I'll have to cover in another episode someday. But handling all these crazy incidents just made Amelie a pro at incident response.

Well, you know, then I helped found the U.S. Digital Service when I was there because no one else, you know, the person who was originally working on it decided they were thinking about leaving government. So, you know, it was just random stuff I, you know, got assigned to go and do and, you know, kind of put the feather in the hat. But I think of, you know, a federal government career, like you're at the White House, you know, this is like the Super Bowl of federal employment. And, you know, I was to return to Interior to my old position there. And I was really excited about bringing, you know, what we're doing for the U.S. Digital Service back to the agency. And we just switched to CIO in that time. It really kind of was saying, hey, you know, you haven't had this chief technology officer position filled in a while. Like, are you guys going to fill it? I really would love to have

that. If not, I would really like to, you know, kind of push to do digital service stuff at the at Interior. And they just didn't act. And I was just feeling like, you know, kind of pent to do digital service stuff at Interior. And they just didn't act. And I was just feeling like, you know, kind of pent up. So I decided to go work for Disney. I left federal service for about a year to go work as an enterprise architect doing technology strategy for Disney. Spent our year in LA, hated Los Angeles,

moved back to DC, went to go work for Treasury in the GSOC, the government SOC in Vienna, Virginia, and lead the continuous diagnostics and mitigation program for all of Treasury. And then most recently, I got offered a position to work as a deputy chief information officer at the HHS Health and Human Services Inspector Jemble's office, which I really always kind of wanted my career to go and do. And, you know, learn stuff like budgeting, which most techies don't learn, learn how to do HR, which most techies don't tend to learn. Got a chance to kind of lead teams. And when our CTO left, I was dual-hatted as the chief technology officer leading up development efforts,

as well as the deputy CIO until we hired a new deputy CIO and I stuck as a CTO for a year, you know, doing a lot of cool stuff there. And I'm currently at Splunk as a technology advocate because every manager needs a break. And I like not having to manage people for the last year or so. But the cool thing is, as a technology advocate, I get to kind of go out and speak about ways that you can do things better.

And she has given a lot of talks. If you go to her website, webjedi.net, right on the front page, you see a link to 15 different talks she's given at places like DEF CON and SHMOO CON and other DevOps conferences too. So if you want to hear more from her, definitely check out her talks. You know, this is my job now and I really kind of enjoy doing it because it's a way to maybe spread the knowledge around to people that may not get experience to it and hopefully make their lives a little bit easier. A big thank you to Amelie Caron. You can find her on Twitter, which is WebJedi, or visit her blog, which is WebJedi.net. I bring you this show free of charge every two weeks. And one reason I can keep it going is because of all the wonderful people who give to the show through Patreon. This is the most direct way to show support for content you appreciate. So consider donating by going to patreon.com slash darknetdiaries.

The show will continue to be free whether you give or not. I'll still be here making the show because I don't want to leave you hanging. And I hope you don't leave me hanging on the other end. This show is made by me, the ID10T award winner, Jack Reisider. Sound design this episode by the digitized Andrew Merriweather. Editing help this episode by the 3D printed Damien. And our theme music is by the never cold, always hot, Breakmaster Cylinder. And even though when a sequel query walked into a bar,

it went up to two tables and asked, Can I join you? This is Darknet Diaries.