Darknet Diaries - 103: Cloud Hopper

Episode Date: October 26, 2021

Fabio Viggiani is an incident responder. In this episode he talks about the story when one of his clients were breached.SponsorsSupport for this show, and for stretched security teams, comes from SOC.OS. Too many security alerts means alert fatigue for under-resourced SecOps teams. Traditional tools aren’t solving the problem. SOC.OS is the lightweight, cost-effective, and low-maintenance solution for your team. Centralise, enrich, and correlate your security alerts into manageable, prioritised clusters. Get started with an extended 3-month free trial at https://socos.io/darknet.Support for this show comes from IT Pro TV. Get 65 hours of free training by visiting ITPro.tv/darknet. And use promo code DARKNET.Sources https://www.reuters.com/investigates/special-report/china-cyber-cloudhopper https://www.reuters.com/article/us-china-cyber-cloudhopper-companies-exc-idUSKCN1TR1D4 https://www.fbi.gov/wanted/cyber/apt-10-group https://www.youtube.com/watch?v=277A09ON7mY https://www.wsj.com/articles/ghosts-in-the-clouds-inside-chinas-major-corporate-hack-11577729061 https://www.technologyreview.com/2018/12/20/239760/chinese-hackers-allegedly-stole-data-of-more-than-100000-us-navy-personnel/

Who's the person with the most power in the workplace? You might think it's the CEO or owner, since they can call all the shots and make policy changes that everyone has to adhere to. But I think the most powerful person in the workplace might be the sysadmin, the person who has administrative access to the core machines that are required for the business to operate. They can see what's in the database, and they can read anyone's email in the whole company, and they can see what files are on your computer,

and they can sniff all the network traffic from your computer to see where you go and what you downloaded. Now, not every network is set up like this, where someone can see everything about everyone, and not all networks have one person who has all of this access. But some networks are set up like this, where one person has control of everything. With the press of a button, they can bring business to a halt, or potentially reroute customer payments or paychecks to them. It's crazy how much power they have. And so it goes without saying, you never ever want some unauthorized person to have admin access to your

network. Because using this power maliciously can be incredibly destructive to your business. But there's another person who also has a lot of power that we sometimes forget about. That's the overnight janitor. The person who has a key to the building and every room in the office, including the CEO's office. And on top of that, they're always there when nobody else is, which gives them the opportunity and capability for some serious spying. The only thing they'd need is the motivation. And what's even crazier is that some of these janitorial services have many businesses that they service each night. So that's quite the key ring to have access to, especially in the right parts of town.

Imagine if the janitor's key ring got into the wrong hands, into the hands of someone with a lot of motivation and malicious intent. And what if that someone was extremely skilled at computers and hacking? That would surely be trouble. These are true stories from the dark side of the internet. I'm Jack Recider. This is Darknet Diaries. This episode is sponsored by Delete.me. I know a bit too much about how scam callers work. They'll use anything they can find about you online to try to get at your money.

And our personal information is all over the place online. Phone numbers, addresses, family members, where you work, what kind of car you drive. It's endless. And it's not a fair fight. But I realize I don't need to be fighting this alone anymore. Now I use the help of Delete.me. Delete.me is a subscription service that finds and removes personal information from hundreds of data brokers' websites and continuously works to keep it off. Data brokers hate them because Delete.me makes sure your personal profile is no longer theirs to sell. I tried it and they

immediately got busy scouring the internet for my name and gave me reports on what they found. And then they got busy deleting things. It was great to have someone on my team when it comes to my privacy. Take control of your data and keep your private life private by signing up for Delete Me. Now at a special discount for Darknet Diaries listeners. Today, get 20% off your Delete Me plan when you go to joindeleteme.com slash darknetdiaries and use promo code Darknet at checkout. The only way to get 20% off is to go to joindeleteme.com slash darknetdiaries and enter code darknet at checkout. That's joindeleteme.com slash darknetdiaries. Use code darknet. Support for this show comes from Black Hills Information Security. Thank you. I'm sure they can help. But the founder of the company, John Strand, is a teacher.

And he's made it a mission to make Black Hills Information Security world-class in security training. You can learn things like penetration testing, securing the cloud, breaching the cloud, digital forensics, and so much more. But get this, the whole thing is pay what you can. Black Hills believes that great intro security classes do not need to be expensive. And they are trying to break down barriers to get more people into the security field. And if you decide to pay over $195, you get six months access to the MetaCTF Cyber Range, which is great for practicing your skills and showing them off to potential employers. Head on over to BlackHillsInfosec.com to learn more about what services they offer and find links to their webcasts to get some world-class training.

That's BlackHillsInfosec.com. BlackHillsInfosec.com. Start with your name. What's your name and what do you do? Yeah, so my name is Fabio Avigiani. I'm an incident responder, threat analyst, and red teamer at TruSec. Red teaming means simulating an attack to make sure a network's defense actually works, which is fun to come in and attack the network. Fabio does that, but that's not what this story is about. Right.

So I work a lot with incident response. That's one of my primary areas that I work with. Fabio works for a company called TruSec, which is based in Sweden. When businesses or organizations get attacked, they can call TruSec up to come investigate and remediate the issue. That's when Fabio will go on site to a customer's location to help them out. He's done a lot of this type of incident response work. So basically anything from ransomware to espionage, I'm a technical lead for the forensics team. When we do incident response, we have a forensics team. That's basically the investigation to understand what has happened, what the threat actor has done, how they've come into the

environment, and if there is any persistence, anything to remove and how to clean that up. Another team is the recovery team where they do all the infrastructure work. So in a ransomware case, for example, there is a lot of recovery and rebuilding to be done. So we work very close with them to tell them like, these are the things that need to be cleaned out. These are the things that are safe to restore.

This is a date that the system can be restored because we have verified there is nothing there. Fabio has been with this company for eight years now. And in that time, he's seen lots of network intrusions and handled many incidents. But there's one incident in particular that he'll always remember. Yeah, so it was summer of 2016, and we got a call from this customer, from this company here in Sweden. Now, Fabio had actually done work for this company before, advising them on how to secure their network better. And reviews of their security,

and penetration testing of some of their applications, things like that. He didn't want to say what company this was, because companies really don't like talking about that time when they were attacked. And it's also not cool for him to go and handle an incident for a customer and then blab about it. It's other world on my show. So he can't say who this company was. But I do know enough about this company that I want you to picture a large, typical office type business. There's many offices, they're all pretty big, and what's notable here is that they have thousands and thousands of computers in the network. What we knew from the beginning is that they had been contacted by the Swedish security service, because one of their systems had been talking to a command-and-control server somewhere on the internet. It was located at a foreign state.

That's basically all we knew when we got that call. Now, this is actually a big deal. When the Swedish security service calls you up to tip you off of a potential problem, you should definitely sit up straight in your chair and ask for help. Because the Swedish security service is the government agency in Sweden that investigates espionage and counterterrorism and any threats against national security. It's sort of like the FBI in the US. And so you can imagine if the FBI calls you to say, hey, one of the computers in your network is reaching out to a really bad computer on the internet, you're going to want to spring into action. What the Swedish Security service was saying was a computer at this company was talking to a known bad actor,

a command and control server, and they gave the IPs that were involved with this. But that was about it. Just the fact that this is information coming from the security service tells you that, you know, it's not just random, you know, command and control server by, you know, some whatever criminal group that is doing ransomware. They don't tell that. They focus on nation state. So when you get a call from them, it's probably related to something bigger. But even though this isn't much information, it's coming from such a reliable source that you can assume a few things. First, they said a computer was reaching out to a command and control server.

When malware infects a computer, it needs instructions on what to do once it's there. Sometimes it's built into the malware. Other times it calls out to another computer and says, what should I do now? Or here's what I have. This is what a command and control server is. Something that can interact with an infected computer. The fact that a computer is reaching out to a command and control server at all means it's probably infected with something like malware that you definitely want to remove.

But this is the Swedish security service notifying them, which might mean that this is either a very serious threat actor, or it could mean that other companies in Sweden have been hit by this too, and notified the Swedish security service, who then looked into it and found this company may also be infected. Anyway, all that is to say is that this was such a reliable tip that it really did warrant calling up Fabio and telling him to come immediately. And that's just what he did. He quickly started shoving gear into his bag.

Well, a couple of laptops with all the tooling and everything needed. And then, you know, all kinds of equipment like, you know, external disks, different type of USB devices for transfers, and usually a lot of storage is needed. But it's also, in this case, it was actually physically close by. So we didn't have to plan so far ahead. So, you know, just the initial things, a lot of storage and a lot of tooling. He packed it all up, jumped in the car and drove to this customer's location. Initially, they told us that they got this call from the security service because they had this connection to a malicious command and control server on the internet.

And the rest of the detail, we would get there. So when we got there, we got in a room with them and they gave us the information that they had received, which was three things. Two IP addresses of command and control servers that their infrastructure has been seen communicating with. Time windows, so when those connections started and ended. And an internal host name of one of their servers that apparently has been communicated with those IP addresses. This is a good set of clues to start with, especially having an internal host name

that's suspected to be infected. Fabio honed in on that server first. So obviously the first thing is asking about that server and understand, first of all, what type of server it is. And then, you know, get access to it in whatever state it is and take it from there. Turns out it was not just some random server. It was a pretty important server because it was one of the jump servers

that the MSP used to manage this customer. Oh, well, that's interesting. So MSP stands for Managed Service Provider. This company outsourced the monitoring and management of most of their servers to another company to take care of, which is this MSP. The MSP is who will keep servers patched up and make configuration changes for this company. And they'll also monitor for faults and incidents. So if one server in this network had a high CPU, it would alert the MSP, and then someone from the MSP would log into that server and fix the issue. But this MSP managed all these servers remotely, from another country even. So they needed a good, reliable way to access all these computers in this network. And to do that, they set up a VPN to a server that they could use to jump off of when they needed to get in this network, which is a jump server. So when someone in the MSP needed to check out a server in this network, first they connected into this jump server, which then had access to all the servers in the network. And it was this jump server that was reaching out to a known command control server. And since this jump server is used by an MSP, it meant it had access to pretty much every important server in the network. Of all the servers to be infected, this was probably

one of the worst possible ones. That is a really good place to be, you know, for a threat actor. You know, first of all, we started asking for access because the system was up and running. So we just asked, can we get access to it? But the problem was that server was fully controlled by the MSP. So Fabio had to call them to get access to it. You know, they were very reluctant from the beginning to give us access. Saying things like, you know, we have SLAs with the customer.

We can't just give access to anyone. You know, if things go down, then it's our responsibility and all that stuff, which, you know, stuff we hear, but it never works out because ultimately it's the customer system. So we say, okay, fine, we'll get back to the customer and figure that out.

But in the meantime, if you don't want us to access the live system, you know, which is fine for now, if you can take want us to access the live system, which is fine for now, if you can take a disk image and a memory dump so we can start from that. Now you might think this is suspicious for the MSP to not help Fabio get access to it. But that's typical in situations like this. Managed service providers provide service to handle computer problems for the customer. And Fabio was not the customer. He worked for a different company, TruSec.

Normally, any type of service providers to an organization that is breached, they may tend to be defensive because, you know, they don't want to, because they might be the reason or they might be indirectly the reason. Maybe they haven't really done what they were supposed to be doing. They haven't been respecting, you know, the agreements with their customers. So they feel like they feel a little bit threatened. If we find something that shows they've done something badly, then, you know, this could be bad for them. So that's why some people may be defensive. And that's another reason for their hesitation.

Once you start throwing around the B word, breach, the MSP is going to perk up and be extra careful about what they're doing. Because for them, it's a bit alarming to hear that their customer may have been breached. But ultimately, this was the company's server, not the MSP's. So the company simply demanded that the MSP give Fabio access. And they let him in. And with that, he was in the system and was capturing the data he needed to do his investigation. We've got a disk image and a memory dump, which is really all you need, right, for doing a forensic investigation.

A disk image is an exact copy of the entire hard drive. And a memory dump is a copy of what's currently stored in the system's RAM memory, which will tell you what programs are running, including any malware. Okay, so Fabio has been doing this type of work for a while and gets right to work analyzing the disk image. The disk image was put onto an external hard drive, and so he just mounts it to his computer as an external drive. But what do you do with this? Where do you even look on this hard drive to try to find malware yeah sure you could run an antivirus scan on it but this jump server already had antivirus running on it and all was quiet nothing had triggered so now what well this is why you need someone who's

trained in digital forensics and what makes a good digital forensics analyst is the ability to spot things that aren't normal. But in order to know what's not normal, you really have to know what is normal. So it's incredibly important for someone who wants to be good at digital forensics to know how computers normally work inside and out. What processes are normally supposed to be running? Where do those programs typically live? What stuff belongs in the Windows directory and what stuff doesn't? This jump server was a Windows computer and Fabio is pretty familiar with Windows. So he got right to work looking through files manually.

You just mount it. You do a read-only mount on your computer and you just have a quick look at it. Because, you know, just from experience, you know where this type of things tend to be. The first place he checked was if anything was in the temp folder. He likes checking here often because this is where intruders like to stage files and put things. The temp folder is a nice spot to stash stuff temporarily. And that is what I mean. Fabio doesn't start with some elaborate scan that might take hours. He manually checks a few places first, just to see if he can spot anything himself right away. So he checks the temp folder straight away. C colon backslash temp.

And there was a bunch of files there. And we found a file with a pretty obvious name. It was called the hostname of the server.mimikatz.hash. Uh-oh. Within the first minute of having access to this server, he already found terrible news. Mimikatz was executed on this computer, and that's bad. See, Windows has a major flaw in the way it handles passwords. When you log into a Windows computer itself, you have to enter a username and password. And that username and password that you just typed in gets stored in memory, often in clear text.

Mimikatz is a tool that goes to the exact spot in memory and grabs the password so that anyone can see it and clear text. So if you can successfully run Mimikatz, it means that you can see the username and password of every single login to this computer since it was rebooted last. There is no reason for this MSP to have run Mimikatz, which means this was a smoking gun, that yes, a threat actor was here and tried to get usernames and passwords

of the users of this machine. That was the output file of the Mimikaz execution. So we opened it up and it contained nearly 100 credentials of users that had been logged on to that system. And the credentials were in clear text. They can be hashed, but if the system is a little bit older and is unpatched and there is no protection for caching passwords in clear text in memory, then Mimikas would be able to extract it in clear text. And that's what we saw in that output file.

So we had about 100 users with their clear text passwords in that text output file, including several Active Directory domain administrators. Which, you know, immediately kind of escalated the whole incident, right? Because then you have evidence that someone had access to, well, the highest privileged credentials in Active Directory. Mimikatz found a lot of passwords, and that is not good. You can assume that all these usernames and passwords are now in the hands

of whatever attacker that got into this computer. But again, this was a MSP jump server, which had connectivity to pretty much every important server in the network. Now, a finding like this is scary. It means this has gotten very serious. And it's like finding a bomb in the building. When Fabio and his two colleagues found this, the temperature in the room went up. Well, it definitely did. And that also made it so we moved to a much bigger room, like an actual war room, you know, with screens and everything. Initially, you know, you never know what you come across. So we were, you know, in this small room,

like, you know, three, four people just looking at this. And, you know, when this type of things happened, then we, you know, make sure this gets escalated and we establish a proper working environment for a big incident, because this was obviously going to be a big incident. Again, we're talking about an organization with thousands and thousands of systems

and you have just identified that they are very likely fully compromised. So you know that you're going to directly or indirectly have to go through everything. It's going to take a while. It's going to take a lot of people. It's going to take some time. Stay with us because we're going to take a quick break. But after we get back, Fabio gets some answers. This episode is sponsored by NetSuite. What does the future hold for business? You don't know? Me neither. But what I do know is

that you don't have to be months ahead of your competitors to be more successful. Just a few days or even a few hours can work wonders. So until someone brings you a crystal ball, NetSuite can give you an advantage. More than 38,000 businesses have future-proofed their business with NetSuite by Oracle. It's a cloud ERP service and one that I'd be using if I needed the help. NetSuite brings accounting, financial management, inventory, and HR into one fluid platform. When you're closing the books in days, not weeks, you're spending less time looking backwards

and more time on what's next. Whether your company is earning millions or even hundreds of millions, NetSuite helps you respond to immediate challenges and seize your biggest opportunities and make use of real-time insights and forecasting, allowing you the opportunity to look into the future with actionable data. Speaking of opportunity, download the CFO's Guide to AI and Machine Learning at netsuite.com.

The guide is free to you at netsuite.com. Finding evidence that an unauthorized person That's netnetsuite.com. But how did it get there? And what did it shoot at? The business leaders needed to be called in at this point to be made aware of this, because this could potentially have big consequences that can disrupt business. Fabio stuck his head, nose and hands back down into his laptop like a dog trying to dig a hole in the ground. Then, of course, you still need to go through the thorough process. Then you use tooling, like you build a timeline of all the files that have been created, modified, accessed on the disk, and you correlate that with the time of connection to the command and control server and say, okay, what files were created or touched around the time that the connection started. Then you narrow it down to all the new files or newly modified files on disk around that

time, and then you just go through that. Well, he makes it sound easy, but that's actually a long, arduous task. A thorough scan using a tool can take hours or more just to go through all the files and check each one. Then when you have it narrowed down to a few directories or files that were changed during that specific time, you need to analyze those files more carefully, either by hand or using other tools. Now, it's one thing for a digital forensics analyst to be able to find problems on a computer, but what makes a really good analyst is the ability to communicate the issues to people effectively. Fabio had to give instructions to people on what to do next. First of all, from our side, we need more people to get ready to start looking into a lot of other systems.

So this has to scale somehow. Because you know, like already looking at that, you know there's going to be more systems affected. It's not just going to be the server. So then at a higher level, and that's not something that I would do directly in that case, for example, then I would get back to our incident manager and say, hey, this is what we found. So this needs to be communicated to the management level at the customer to make sure that they understand what we are finding here and that there will be consequences of this certain you know a different type when you when

you know even if you don't know exactly what happened yet uh you know that they had that level of access inside your organization so management needs to know that right away because you know they need to start working on controlling the situation from all different perspectives, from a business perspective, from a marketing communication perspective, and all that stuff. So they need to know as soon as possible. So we had a couple of parallel activities going on.

One was looking at the disk, and one was looking at the memory. We had the IP address of the command and control server. Something you can get out from memory is network connections. Current or historical, if they're still left in memory somewhere. So we did look for that IP address and we found a process that had been connected with that IP address. And the name of the process was vba328rkit.exe, which is not something I recognized immediately. But, you know, we took the hash of that binary and checked it and it turned out to be a legitimate software

called VBA32 Anti-Rootkit Scanner, which is ironic in a way. It was a legitimate software looking for rootkits and malware on systems. It was scanning the system looking for rootkits and malware on systems. It was scanning the system looking for malware. And that process, it was a signed binary by this company producing this software. Signed binary is a way to show authenticity of a file. That file really was actual software that detects malware from a legitimate company. And it specifically looked

for rootkits, which is malware trying to get access to something it's not supposed to. But it was this anti-rootkit software that was connecting to the bad IP, the command and control server that the Swedish security service told them about. That is very strange, but it's also a clue as to what might be going on here.

Right, so we looked at this file a little bit more, and there were a couple of things sticking out immediately. First thing is that it was in a very unusual location. It was under C Windows Web, which is a folder that exists, but it doesn't have that type of software in there. So just having that binary located in that directory was strange. And next to that file, there were a few files, a couple of DLLs and another couple of files.

So that immediately smelled like DLL sideloading. DLL sideloading is an interesting attack technique. Here's how it works. In Windows, programs often require more than one file to run, like a driver or a config file or a DLL file. A DLL file is just some extra data that the program needs in order for it to load properly. And when the program tries to load, it'll try to find the required files. And this process can be manipulated by placing a malicious DLL file in a certain place so the program will load it into memory. Programs have sort of an order of operation of how they look for their needed DLLs. And this can be exploited.

So this particular DLL had instructions to communicate with an outside server. And this type of attack is much harder for antivirus scans to pick up. It's a very well-known technique and very effective and also very easy to do because there is all kinds of software vulnerable to DLL site loading because it's practically a very hard issue to fix. Not technically. Technically, you just need to verify you're loading the right DLLs. But in practice, when you have software with all kinds of DLLs, all kinds of updates, just maintaining that is really challenging and expensive. So there are a lot of products that don't do it right. So Fabio examined this DLL. And yes, sure enough, this normal and benign program was loading this malicious DLL file. It was very simple. It only had one job.

When it was loaded, it would read another file from disk, which was just a binary blob. It was actually encrypted data. It would decrypt it with a key that was stored into it. It was stored inside the DLL. It would decompress it, and then it would just load it and execute it in memory. So again, this would be done within the context of the legitimate binary.

So if you look at who is doing what on the system, you would see that it's this process that now is executing this code. Now that they know this threat actor likes to inject itself into known good processes, they start looking for more instances of DLL side loading. And we found three more instances of DLL side loading implants. And they would start the same type of malware, but connecting to different command and control servers. And you could also see that they had been started at different points in time.

And there was actually weeks in between these executions. So we suddenly got our timeline a few weeks back, which means the first instance of this rat had already been running for, I think it was more than a month at that time. So actually, this investigation is going very well so far. Yes, it's always bad to find that someone came and ate your lunch when you weren't looking, but now they have lots of pieces of evidence to go with. And so they take what they've learned from this and start spreading out their search

to find out what other computers might have these same indicators of compromise. Yeah, so there are two directions you move from here. And we normally have different people working in parallel on different tracks. One is to figure out what happened after that, what other systems have been affected after this server had been affected, and the other track is how they got in the first place. So they start searching everywhere to see if these DLL files

were on any other computer in the network, and see what connections were to and from this computer during that time. They basically were just following the path of evidence. But wait a minute, hold on. Discovering these malicious DLL files means, for certain, there was an unwanted intruder in this server doing things, pushing buttons, executing programs. So wouldn't you want to immediately kick out all users and lock this system down so that whatever malicious person has access to this can't do anything else that is always a call and it's to be made and

there is no default answer it depends but looking at the situation here that had been running for weeks right so if it would be up for a couple more hours while we investigate, the chances that something specifically happened within those couple of hours is low, provided that you haven't given away that you're onto them. That's why it's so important

that the right actions are taken from the beginning, and especially that the wrong actions are not taken from the beginning when you communicate this. Because then you're going to have to take this type of decisions say okay do we think that they know if they do then we may prefer to shut this down right now so they can't hide better now or if you if you have a feeling like we've been very stealthy in our investigation, they probably don't know, then it's actually easier for us to work with something that is ongoing.

Given the time window here, it's been going on for weeks. What are the odds that it's going to happen within the next couple hours? In this case, the call was to allow the investigation to continue a little while longer without wiping this compromised server down and disinfecting it. Because the main thing they still needed to figure out was, how did this malware get on here? It didn't show up by itself. Someone put it there. So they looked through the logs and pretty easily discovered that someone simply logged into this computer, normally through a remote desktop, and put it there, which is not an exploit or a hack at all. It means someone had the username and password to get into this server. They logged into it and put the malware on. Okay, so they know how it got on and they know what

files it left on the system, but they're curious to see if there's anything currently running now. And that's where memory analysis comes in. Because whatever's in RAM is what's actively running. So we were looking at the memory analysis. We found a few interesting things in there. They found that, yeah, there was malware in the memory. But as they looked at it closer, they found a note in the malware, which was odd. This appeared to be a note for the forensic investigators that were looking for this malware, like Fabio. Which said like this, I have it written here. It said, have your bosses given you the space to try to be a hacker? Come on, man, don't kill me.

That's what he said. Have your bosses given you the space to be a hacker? Come on, man, don't kill me. What in the world does that mean? It's not clear. Does it mean to tell your boss you want to be a hacker? Or to leave the malware here and just ignore it? This message is confusing. Whoever wrote it missed the chance at saying something effective. But if it does mean to leave the malware there, The message is confusing. Whoever wrote it missed the chance at saying something effective.

But if it does mean to leave the malware there, then it totally reminds me of this scene from the TV show Mr. Robot, where Elliot is investigating an infected computer after an attack. I'm going to take a look at the infected server, okay? Give me a minute. They must have left a mark or something. Every hacker loves attention. They don't just do DDoS attacks for no reason. This is it. Is that supposed to be a joke? This was way too easy. They didn't hide it well at all. Elliot looks at the message and it says,

leave me here in all uppercase. And see, that's a clear message. This note is from me. They're telling me to leave it here. But why? It's funny, that's my wallpaper on the desktop right now. As he analyzed things closer, he saw some more interesting evidence. On that particular server, in the same directory where we had the DLL siloing malware, there were three more files that kind of changed the perspective of this whole thing. There were three files. One was executable. It was called nbt.exe, which is a tool. It's a legitimate tool called NBT scan. It's a NetBios named network scanner. Basically, NetBios is how

Windows computers connect to shared network drives. So this NetBIOS scanner can scan a whole network and find what servers have shared network drives on them. And then if a computer has a shared network drive, you may be able to connect to it to see what files are on that server. Then it was a text file called p.txt, which was empty. And then you had a batch file called pp.cmd. And pp.cmd had something like 33 lines. And each line was a command that was executing nbt.exe, so the NetBIOS scanner, followed by a public IP range.

And then putting that output into the p.txt file. So you had those 33 or something public IP ranges in that batch file that were scanned. And obviously the first thing you do is you start looking into what are those public IP ranges, who owns them and what were they scanning? And, well, I think 19 of those public IP ranges belong to the U.S. Department of Defense. Whoa, that's interesting. This threat actor was using this server and this company's network to scan the U.S. Department of Defense's servers to check if any of them have open file sharing connections with this company.

Now, the Department of Defense is huge. It's the military, so Navy, Air Force, Army, but also the NSA is part of DoD., saw that the output was blank, and then logged out. And then there was no activity on this server for quite a while. Right. So this kind of tells you that our customer was most likely not the primary target for this, right? They were trying to see from the network they were in, can they go into what was maybe their final target? And once they realized that's not the case,

they just dropped it, at least for a while. Which suggests this is likely a nation state actor they're dealing with and not some criminal group or hacktivist. Because look at what they

immediately went for. It wasn't the customer's data, or money, or ransomware. They immediately went to scan the DoD. Fabio checked with this company to make sure there aren't any connections with them and the U.S. Department of Defense. No, this company was not connected with them in any way. After the scan, there was no malicious activity on the server for two whole weeks. Then this threat actor logged back in, but installed all new malware and all new tools, which talked to a totally different command and control server. They didn't use any of the tools that were already there. In fact, they brought in a known malware called PlugX.

PlugX is a known RAT. RAT stands for Remote Access Trojan, and it's a type of malware that can control your computer. That has been used over many years. It's still used by many different threat actors groups that are all based out of China. Looking at the forensic data, it's as if there were two or three different teams that were part of this attack. One team to just establish initial connection to the system.

Once that happened, there was an immediate scan of the DoD's network to look for shared connections. Then, two weeks later, another connection into the server where they brought all new malware and tools. And it was then when Mimikatz was run, where they grabbed all the credentials and started pivoting and traversing to other systems in the network. In fact, they got into the domain controller of this network and had full admin access there, which pretty much gave them control

over the whole business. And this is consistent with how nation state attackers work. There's sometimes one team that's just there to get initial access, and then another team takes it from there and carries out objectives. And now Fabio is suspecting that this threat actor might be from China. But he can't tell for sure, and there's still more research to be done.

Fabio wanted to know more about how they initially got into this jump server to begin with. They looked at what user logged in and placed the initial malware on the system. And it was a username of someone who worked at the MSP, the company that managed the computers in this network. The source of that logon was an IP address at the MSP side. So it was a logon from the MSP infrastructure into the customer infrastructure. So we checked with the MSP and we asked about that user and if that person was working that day, if that person had logged on at that time.

Turns out that that person wasn't even working that day. With some more questions to the MSP, Fabio concluded that this was a malicious logon. That employee at the MSP did not log into the server and place the malware on there. Someone had stolen their credentials and did it. But wait a minute. In order for this malicious actor to get into this jump server, they connected into it from the MSP's network. This means the attacker had control of a computer inside the MSP. Oh man, this just made the incident so much bigger because the MSP has more than just one customer.

In fact, this is one of the biggest MSPs in the world. They have hundreds, if not thousands of customers where they're able to get into networks and manage all those computers too. So we just had to ask questions at that point. I mean, we said, look, we see this malicious logon of this user that dropped malware and it comes from that particular IP address located within your infrastructure.

So they took in that information and they were doing their own investigation. It took, I think, three weeks after we've given that evidence for them to get back to us and say, yeah, we see malware on our jump station as well. So within the MSP infrastructure, same malware. The MSP had been hacked into and didn't know it until Fabio showed them the evidence. For them, this was much worse than one of their customers being breached.

They were breached now too. And they may have facilitated a breach on many of their customers. This must have been a really bad day for the MSP to discover this. After another few weeks, then we also got to know that more of their customers had been compromised with the same malware. So our customer was not the only one. And we also know that they found keyloggers on the JUM servers at the MSP site.

Those were, you know, just to give the picture of the infrastructure here, the MSP has a lot of customer to manage. We're talking about a global one. So, you know, a name that everyone knows about. They manage a lot of customers and they have an infrastructure as a jump layer between their internal infrastructure and the different customers infrastructure. And jump servers are used to access more than one customer. So the one that was used to jump into the customer we were handling was also used to jump into a lot of other customers and that system had key loggers on it.

So the threat actor was able to see the credentials for the different MSP customers and were able to jump into multiple customer environments from there. I mean, if you put this together with what we found in our investigation, I could only imagine like they did the scan on the server that we were investigating. They scan the DOD ranges. I would expect they've done a similar scan from other customer environments as well. And then just prioritize the ones that had trust with DoD. Oh, interesting. This is now starting to come

together for me. If a Chinese threat actor wants to get into the U.S. Department of Defense's network, how could they do it? Well, they might have intelligence that says, well, some companies do have a shared connection with the DoD, maybe because they're outsourcing something or connected with them in some way. And so the threat actor might know that the DoD allows some companies to connect to it through NetBios, only specific companies or countries. So their thought was maybe they could find the network or a company that does have access to DoD's network. And to do that, they could just hack into an MSP who has access to lots of networks and then spider into each of the customers' networks and run scans on the DoD's IPs to see if there's any shared folders open to that company

or network. Wow. This is what an advanced persistent threat is. An APT. Whoever was behind this had quite the resources to try to penetrate DoD's network and had no problem hacking into potentially hundreds of networks around the world to try. Unbelievable. As this incident winded down, Fabio still had no idea who did this. And that mystery held up for years. But a few years after that, news hit that told him exactly who did it. Here's a clip from a press conference where the U.S. Deputy Attorney General Rod Rosenstein addressed the public. Good morning. Today, the Department of Justice is announcing a criminal indictment of two hackers associated with the Chinese government. The charges include conspiracy to commit computer intrusions against dozens of companies in the United States and around the world. This case is significant because the defendants are accused of targeting and compromising managed service providers, or MSPs.

MSPs are firms that are trusted to store, process, and protect commercial data, including intellectual property and other confidential business information. When hackers gain access to MSPs, they can steal sensitive business information that gives competitors an unfair advantage. The indictment alleges that the defendants worked for a group known to cyber security experts as APT10. These groups are designated as APTs, or Advanced Persistent Threats, because they use malware to gain access to computer networks and to exfiltrate or steal data over an extended period of time. These defendants allegedly compromised MSP clients in at least a dozen countries,

the United States and 11 other countries. The victims included companies in banking and finance, telecommunications and consumer electronics, medical equipment, packaging, manufacturing, consulting, healthcare, biotechnology, automotive, oil and gas exploration, and mining. The defendants allegedly committed these crimes in association with a Chinese intelligence agency known as the Ministry of State Security. There is no free pass to violate American laws, merely because they do so under the protection of a foreign state. Later on in this press conference, Jeffrey Berman, the U.S. attorney for the Southern District of New York, had some remarks. The defendant's hacking campaigns also targeted U.S. government agencies, including the laboratories of NASA,

the United States Department of Energy, and the U.S. Navy. Members of APT10 stole personal confidential information, including social security numbers and dates of birth from over 100,000 Navy personnel. Whoa, they did it. Those crazy hackers did it. They found a way

into the Department of Defense, specifically the U.S. Navy's network. If there's one thing the history of hacking has taught us, it's that data will not be contained. People will break in and expand to new territories, and they'll crash through barriers

painfully, maybe even dangerously. But, well, there it is. The hackers found a way. They got into the U.S. Navy and stole 100,000 records of Navy personnel, including Social Security numbers. Incredible. Well, once this indictment came out, more details started to emerge. Reuters journalists Jack Stubbs, Joseph Mann, and Christopher Bing did an investigation and found that seven different service providers were compromised, and they listed Hewlett Packard Enterprise, IBM, Fujitsu, Tadak Consultancy, NTT Data, Dimension Data, and Computer Sciences Corporation.

And yes, all these provide IT services to other companies. So if someone hacked into any of these, they would probably be able to get into their customers. And the Reuters article goes on to list some of the customers that were hit by this, which includes the telecom giant Ericsson, a Navy shipbuilder, and the travel Service, Sabre. Now, some of these companies listed do have contracts with the U.S. Navy, especially that Navy Shipbuilder. So it's quite possible that one of these companies did have privileged access into the U.S. Navy's network, which is a fascinating attack, right? Don't come in through the fortified front door when you can just disguise yourself as a caterer and just get

welcomed in through the side door? This was obviously a massive campaign, which seemed to have a primary objective of getting into U.S. government networks. And that's kind of what we expect espionage to be, right? When one government wants information on another government, they'll use electronics or computers to carry out their data collection and spy on the enemy. But the concerning thing here is that the Chinese government hacked into U.S. companies in order to complete their mission. On top of that, when they got into these companies, they sucked up any intellectual property they found along the way.

And that's straight up theft. That a foreign government stole proprietary information from a corporation is astounding. Because that kind of thing just doesn't sound right to me. But it doesn't sound right to the U.S. Feds either. Here's the Deputy Attorney General Rod Rosenstein again. In 2015, China promised to stop stealing trade secrets and other confidential business information through computer hacking, with the intent of providing competitive advantage to companies in the commercial sector. But the activity alleged in this indictment violates the commitment that China made. That was a commitment they made to members of the international community,

to the United States, to the G20, and to APEC. It's one thing for governments to spy on each other, but it's a totally different thing when a government hacks into a private company to steal data from them so that they can benefit from it economically. But really, the rules of cyberspace have yet to be fully formed. The way this space is innovating and changing every day makes it extremely difficult to lay a set of international laws down and actually enforce them. The people who were here in this space early were able to sneak by because

there weren't any rules, and the advanced players today surely would only make rules which allow them to continue to have power and control in this domain. But regardless of what rules are made in cyberspace, it'll only work with nations who agree to abide by the rules. Well, Fabio and his team at TruSec were able to clean up this client's infection, which was not easy. I mean, one thing they had to do was change every single active directory password in the entire company. And there were thousands of passwords. But not only that, there are lots of computers that have accounts that talk to other computers. So these services all had to have their password changed too. And that took a long time because there were so many things that

would break along the way. I mean, just think about all the old servers in a network that nobody has touched for 10 years. And the person who set them up is long gone from the company. Yeah, well, suddenly it's not working now. And the current admins have no idea where the credentials are stored in this custom application that was made it's a mess which causes businesses to be impacted for quite a while there is a lot of consequence when you need to do a full proper active directory set proper Active Directory reset. And then on top of that, we introduced active monitoring and EDR tooling because you can never be 100% sure

that the investigation has found everything. So you still want to have your eyes on everything that is happening at least for a while after this. Ideally forever, right? You always want to have your eyes on things. In addition to that, this company cut ties with the MSP that got them infected. They were already in the process of

renegotiating a contract with them, and this just made the decision easier to not go forward with them. They got a different group to come manage their servers after that. This is an interesting story, since the threat actor targeted MSPs to go after their customers and then carry out their objectives from there. And MSPs are pretty common. More and more companies are outsourcing their IT infrastructure. So to target them makes a lot of sense if your goal is to steal intellectual property. It's sort of like going after the janitor's key ring,

which can get you access into many buildings in town. So far, the people indicted have not been arrested or brought to court. They're still hiding out somewhere. But they have been named and identified and are considered fugitives in the eyes of the U.S. If they're ever caught, they're going to have to go to New York to face their charges. Big thank you to Fabio Pagani for sharing this story with us. It's crazy to think that as an incident responder, you might wake up someday and go face off against a Chinese advanced persistent threat.

Yeah, that happens sometimes. You know there are bonus episodes of Darknet Diaries, right? There's also an ad-free version of the show, too. And there's two ways to get this. If you're an Apple Podcast user, you can sign up to Darknet Diaries Plus right there in Apple Podcasts, or you can visit patreon.com slash darknetdiaries. By joining either of these, you will directly be supporting the show, and it'll give you a better listening experience. And I really have to say thank you to all the people who joined,

because they really do make the show much better. So thanks. The show is made by me, the Cloudy Dragon, Jack Recider. Sound design and editing by the Hidden Tiger, Andrew Merriweather. Our theme music is by the Fiery Crane, Breakmaster Cylinder. Doing math in binary is slow. You have to go bit by bit. This is Dark Knight Diaries.