Darknet Diaries - Ep 42: Mini-Stories: Vol 2

Episode Date: July 9, 2019

Three stories in one episode. Listen in on one of Dave Kennedy's penetration tests he conducted where he got caught trying to gain entry into a datacenter. Listen to a network security engineer talk about the unexpected visitor found in his network and what he did about it. And listen to Dan Tentler talk about a wild and crazy engagement he did for a client.GuestsA very special thanks to Dave Kennedy. Learn more about his company at trustedsec.com.Thank you Clay for sharing your story. Check out the WOPR Summit.Viss also brought an amazing story to share. Thank you too. Learn more about him at Phobos.io.I first heard Clay's story on the Getting Into Infosec Podcast. Thanks Ayman for finding him and bring that story to my attention.SponsorsThis episode was sponsored by CMD. Securing Linux systems is hard, let CMD help you with that. Visit https://cmd.com/dark to get a free demo.This episode was sponsored by Thinkst Canary. Their canaries attract malicious actors in your network and then send you an alert if someone tries to access them. Great early warning system for knowing when someone is snooping around where they shouldn't be. Check them out at https://canary.tools.For more show notes and links check out darknetdiaries.com.

Hey, it's Jack, host of the show. A long time ago, I set up a file sharing website at home on a Raspberry Pi. I set it up to make it easy to transfer files between me and anyone I needed to send files to. It was a simple website. Drag and drop the file onto the webpage, and boom, it's hosted on my website for like a week, and then it gets deleted. I knew it wasn't secure, so I never posted anything that was sensitive to it. But I also took this opportunity to see if I could detect anyone trying to hack into the thing. I set up all my best sensors I had at home, a firewall, an intrusion detection system, full packet captures using Security Onion. I turned on tons of logging and watched.

But nothing happened. Nobody knew my site existed to even think about trying to exploit it. Oh well, yeah, I forgot about that little website for years. But last week, I went to check on it, and there was a suspicious file uploaded. Not by me. I checked into it, and whoa, someone uploaded an exploit and gained access to my Raspberry Pi. A hacker was in my house. Okay, uh, geez, uh, quick, what do you do? Now, perhaps some people would feel freaked out, violated, or get anxiety, because it's scary knowing someone is in your computer looking at your stuff, and you have no idea who they are.

But me? Well, I stayed calm, because I expected this to happen, so I isolated the whole thing on its own network, and it just wasn't possible for them to move to any other computer or get anything good off this Raspberry Pi. You could say this was sort of a honeypot. I traced their footsteps and looked at everything they did. Psh, amateurs. They used an off-the-shelf PHP script to exploit the thing. They didn't cover their tracks. They checked a few directories looking for anything good. The server had nothing, not even a database. They tried getting to root and hopping to some other devices in the network, but yeah, no luck. This system wasn't even allowed to connect to the internet, so they left.

And so, yeah, not really that exciting. I turned that Raspberry Pi off and reformatted the SD card. But you know what? I did learn something cool along the way. And we're going to get into a similar story today that I think you'll learn something interesting too on what to do when this happens in an important network. These are true stories from the dark side of the internet. I'm Jack Recider. This is Darknet Diaries. Okay, so this is another mini stories episode. There are three stories in one.

These are stories that are too good to pass up, but not long enough to make into a whole episode. And there are a few cuss words in this one, just to let you know. All right, so let's call some hackers. Hello? Oh, can you hear me? This is Dave Kennedy. He's quite known in the InfoSec space.

He's built some highly popular hacking tools and helped start DerbyCon, which is a popular hacking conference in Kentucky. But probably the thing he's most proud of as crowning achievement is this. How are you? One sec, I'm just finishing up an email. This is a clip from Mr. Robot. Elliot, the character in the show, is trying to hide from someone and slips into a conference room and tries to social engineer his way into a meeting that's in progress.

We should get started. I think you're in the wrong room. I'm sorry, you are? Sean, head of sales. Sean, of course. Dave Kennedy. I work with Craig on the Q4 push. I had longer hair then. There's no coincidence that Elliot uses Dave Kennedy as his fake name

while trying to social engineer his way into this thing. It's because Dave is a social engineer master. Dave's reputation precedes him. So how does a big time InfoSec guy like this get started? Playing video games in high school. I was programming MUDs back then, and I was one of the guys that ran the actual MUD and kind of promoted and grew it and everything else. And that's where I started learning some C and C++.

MUD stands for multi-user dungeon. Think of it like World of Warcraft, but with absolutely no graphics. It's all text-based, but still online where you can group up and quest and raid and fight everything. He realized college wasn't right for him after high school, so he decided to join the army. all text-based, but still online where you can group up and quest and raid and fight everything. He realized college wasn't right for him after high school, so he decided to join the Army. He headed down to the Army recruiter's office. The guys just didn't seem very happy there. And I'm like, man, why would I really join this if the folks that are trying to recruit me aren't happy about their jobs or what they're doing?

And I was actually walking out about not ready, not even going to join the military. And I saw these four really buff Marines walking in walking in, you know, sink and they just, you know, wearing the dress blues and they look just look sharp as heck. And I was like, man, I want to be like that. Walked into the Marine recruiting station and I was a really overweight kid and, you know, didn't have a lot of physical fitness or anything like that. And I said, hey, I want to I want to become a Marine. And I tested very highly on the ASVAB, which is the aptitude test for, uh, for the military. And, uh, what was

great about the Marine Corps is, uh, they guarantee your position. And I wanted to do something, uh, like hacking, uh, and, and, and wanted to get into more of the, uh, intelligence side of things. So I was able to go into, uh, uh, the military intelligence side and work in signals intelligence, which was a ton of fun. He was stationed in Hawaii and Fort Meade and did two tours in Iraq. He got to do fun stuff like forensics and research and cyber warfare. He got out of the military and joined a small consulting shop. Back then, penetration testing and security in general was in its early stages.

Social engineering, the deceptively benign sounding name for tricking people into giving up their passwords, that really wasn't that big of a thing yet. Web applications weren't really getting that much attention from the security professionals. Dave headed up the penetration testing division, then eventually became the VP of consulting. And it was at this, his first job, where he learned a lot of new skills in different programming languages, like Python.

And then I had a really great opportunity hit me to be the chief security officer over at Diebold, which at the time I think I was like 26 or 27 years old, which was awesome. You know, being a VP of security of a Fortune 1000 company, I really had no idea what I was doing, but it turned out to be a really, really awesome position. I learned a ton from that. 27 and the VP of a Fortune 1000 company? Whoa. He was young and motivated to learn. He picked up all kinds of skills that he used to then start up his own company, which he called TrustedSec, and then he started Binary Defense.

So TrustedSec is an information security consulting company. I started it literally in the basement of my house and Binary Defense as well. I started Binary Defense and they're both two different companies, two separate companies. And I did that for a very specific reason. Consulting is very specific and I didn't feel like we could be the same company

doing the same work and also doing the monitoring detection of an organization as well, like giving heads up or making ourselves look good when we're doing an assessment. So I really split the companies up early on.. I think we have about 162 employees now. The story we're talking about today is about an assignment with TrustedSec.

For this engagement, the client was a large retail company with retail stores all over the U.S. and they wanted Dave to test the security of the store. And, you know, we had a few objectives. One is to be able to steal stuff from the store. The other objectives were to get access to the corporate headquarters. Steal stuff from the store is actually going into the store and grabbing like stuff off the shelves. Oh, yeah. Yeah, absolutely. As well as as well as could you get access to the back store area where they have like the point of sale systems and the like the base servers? Can we get into that and plant stuff in? So it's a lot of fun. If you think about it, this type of work is simply quality assurance.

Companies have been doing quality assurance testing for decades, making sure their product is within spec. And now in the modern age, the way some companies test quality assurance is to hire a bad guy to see how good their security is. We did some reconnaissance ahead of time. We went to the store, purchased a couple of things legitimately, went to a different store, looked at who the employees were, how they operated, when they took lunch breaks, leased out a person out during times. We had all of that kind of mapped out for when we were actively going after this organization. While he was there, he noticed

these stores all have LP. You know what LP is, right? It's loss prevention. And it's typically a person standing near the front door of the store watching every customer coming in and going out to prevent people like Dave from stealing things. So first, Dave got to test how good their LP is. So if you come in wearing a suit, you're pretty much not going to be looked upon. You come in dressed up, you know, as a, you know, ripped jeans and, you know, dirty hair or something like that. I don't know, you know, looking suspicious, you know, looking to your left and right. And, you know, maybe that's a way that you get identified. But, you know, for us, you know, we usually come in looking professional, you know, and looking in a way that we're not suspicious. We're not

looking over our shoulders. We're not looking nervous. We're looking like a customer. We might actually buy some things with cash, you know, just to kind of throw everything off. It's more so just trying to be and act like you play a part of that role and that you fit. So we just started grabbing a bunch of things from the store, you know, shoving them into our backpacks. So during this time, the LP is looking for shoplifters, but Dave brought help to handle that. A second person to distract the LP. It's very difficult to keep an eye on everybody that's in the store. So, you know, there's only a finite amount of personnel. If you can do some distractions in different locations that have much lower levels of personnel,

even much higher percentage of being successful and things that can take time away from the person. Like, you know, if we have two people, you can do a diversion for one, have them communicate and talk, and then the other person's doing nefarious things. I think that works out really well with us when we have two people you can do a diversion for one uh have them communicate and talk and then the other person's doing uh nefarious things i think that that works out really well with us when we have two people kind of uh doing it at this point dave has a bag full of stuff he stole and is walking around the store this is a multi-floor store so it goes up to the second floor and even goes up to the third floor we walked into the store as just a regular person and when they weren't looking we just went into the back we were basically in back for like i don't know 20 minutes 30 minutes we have these little devices

that we call tap devices um that have cellular cellular communications so we don't have to worry about the firewalls but it still allows direct access to their network so we plug that into other network it has two ports on it um just unplug the one ethernet cable plug the other one in and then um was able to basically have direct access to their back end infrastructure, their cardholder environment and the retail's enterprise network. OK, he's stolen stuff and now tapped into the network and got access into their back end infrastructure through this network port. It had iPads for kind of catching people out and things like that. And we took like two of the iPads.

Jeez, Dave is on a roll here. So now they have access to customer credit card information and internal company data that came out from the back room to see what other things they could take from the store. And then we saw the cash register and it was on this podium. And so we just had one of our folks, there was two of us, one of them was just basically asking about a bunch of stuff. And he was distracting the LP.

I basically took a screwdriver and removed it from the, actually bolted it in from the thing and walked out with the cash register. This is a big cash register. This isn't like a small thing. I mean, I took the whole cash register with all the money inside of it. I mean, it's extremely heavy. So I carried that out literally and walked out of the store without anybody. And one of the employees looked at me and kind of looked at me weird, but then I just kind of waved and walked out. And that was kind of that was the end of the story. We walked out and drove in our car and drove off. Dave walks out of the store with a big old heavy cash register full of cash, two iPads in his backpack, and a ton of other store merchandise. Unbelievable.

You know, it's a rush. I get nervous every single time I still do it. Dave now tries to test another store to see how they'd handle him. And for one of the stores, we called ahead and we spoofed our number coming from the corporate offices and claimed to be one of their main IT folks and that we're going to be doing an upgrade to the store location for faster bandwidth and everything else. And they were super excited about that. So they let us write in. We had big business cards. This is one of Dave's specialties, social engineering, spoofing phone numbers, acting like IT from corporate office. He's a master at this. When he did this, it worked like a charm. They escorted him right into the

back room, showed him the computers and left him there unsupervised for 30 minutes while he hacked into the network. Again, unbelievable. Dave explained to the head of security how they could get into everything so easily. This kind of shocked them. So they wanted Dave to now test the security in their corporate headquarters to see if they could break into the data center there. And here's where we actually got busted. It wasn't the store locations that had the most amount of security. It was the enterprise location that didn't have much security at all. First, they had to figure out how to get into the building of headquarters. And what we did is we looked at the front location. The front location, you had a lot of people badging in.

However, one of the side doors, people could just walk out. You didn't see anybody walking in, but you could see people walking out, especially during lunch and dinner and things like that. And so during lunchtime, we waited outside and saw somebody walking out and we just pretended to be on the phone. We're dressed up in a suit. And as soon as the door was about to close, we grabbed and we walked right in. So it was really easy to get into the building itself. It's easy for Dave because he knows all the tricks and has done this a bunch of times. When you do something a lot, you get pretty confident with what you're doing.

You just walk around like you belong. You kind of just walk around. You pretend that you're on the phone. You're with somebody else. You're pointing at something or pretending that you're having a meeting. And you just keep walking around the building until you find the objectives that you need. We found the data center. The data center was locked in.

There wasn't a lot of traffic, especially during the lunchtime. We went to this conference room, which was basically like conference room tucked away on the side. They sat down and acted like they belonged there. And from here, they planned their next steps. They wanted in the data center, but that door was locked. And chances are it's harder to piggyback into a data center and just follow someone else in. But a good social engineer doesn't always have everything planned out. Sometimes they just have to take it step by step,

see how far they can get, and then figure out what they can do from there. So they looked around to see what they could use in this conference room. And there's a conference, you know, like a bridge on there, like a phone there. And we called from the bridge, and I called the data center number. And the way that I was able to do that was first calling the receptionist first, saying, hey, what's the data center's extension? They gave it to me and then I called the extension. And then before we called the extension, we did some research on individual people in the company.

And I found a person in IT that had access to the data center. And so I called this phone. I'm like, hey, I'll just say his name is Bob. I'm like, hey, it's Bob. I'm here with a bunch of auditors for for PCI work. Just I'm going to do just a quick site audit of the data center. Could you let them in just so just so we can get this last part of this, this, you know, compliance thing taken care of for for for the payment card industry? I just threw out a bunch of acronyms, things like that. And the person at data center was like,

hey, who'd you say you were again? I'm like, oh, hey, it's Bob. Just trying to get this audit done. He's like, I'm best friends with Bob and you're definitely not Bob. I don't know who you are or why you're calling from a conference room that's downstairs, but something's not right here.

Shoot, he's been caught. Of all the people to impersonateate he picked someone that person on the phone actually knew quick what do you do we rushed out of the building really quick before he got busted dave escapes and that's always the second objective to a social engineer if they get caught to try to escape because part of this is testing their incident response. And their response was pretty poor if they let Dave get away. But Dave's objective was not complete. The company tasked him with getting into the data center, so he needs to go back and try again.

But now, on one hand, he knows more about the location. And on the other hand, it might be trickier because maybe they're on high alert now. We rebroke back in two days later. Same method for piggybacking. And then we waited past launch until everybody came back and we kind of just sat. There was like a little break room right outside the data center. And we just sat and watched who had access to the data center, who didn't. They noticed to get into the data center, you need an RFID badge. This is one of

those proximity cards where you swipe a credit card looking thing near the door and it unlocks the door. Well, they came prepared for that. We've created our own custom backpacks that are over amplified and we can usually get a little bit of distance, a few inches away from an individual and their badge and be able to collect. So we can just walk past somebody and clone their badge and be able to replicate it. We can clone as many as you want to. We can actually imprint new badges and what we usually do is we'll get pictures from outside the facility of their badges and then we'll we have a printer in our cars like a portable printer. We'll print badge IDs that look like theirs as well with our pictures on it and then we'll just imprint those badges with their identification and their badge cloning.

So just by walking past them, you can literally just clone a badge as many as you want to. So they do just that. They prepare to use their RFID keycard cloning machine to walk past someone coming in or out of the data center, clone it, and then go make a copy off-site. As they're watching people go in and out of the data center, they pick someone, a mark. And we were able to walk past the person, kind of grab him and say, hey, you know, I'm a new employee here, blah, blah, blah, blah, and just ask a bunch of questions. And we cloned his badge at the same time.

Success. They got the digital key they need to get into the data center. So they need to leave and go print it on a badge. They pack up and head out. Came back at night, badged ourselves in, and got into the data center that way. We signed in, as we were supposed to, because there was somebody in there. Didn't even question us or ask us. Just kind of looked at us. We signed in. They went back to their computer. And then we essentially had free access to roam the data center.

And what we did is we placed another tap device in one of the core networking switches, which gave us confirming a DHCP. And we were able to communicate with different things. And so once we had that, we essentially had direct access to their internal into their entire environment and kind of took pictures and selfies and things like that. There was actually a bathroom in the data center, which I thought was really weird. So we use the bathroom and then we walked out. Mission accomplished. Feels good. But Dave is in a funny place

because when he's successful, it means his client's security wasn't strong enough. It sort of means he has to go to them with bad news. It's almost always shock. They assume that they have problems or exposures, but they probably don't realize to what extent that is. And our job isn't to say like, listen, you're doing all this wrong.

Our job is to highlight the things that they're doing well as well. So, you know, here's the things that you did well. Here's the things that actually thwarted or stopped us. Here's the things that, you know, you do very good. And here's some of the things that we identify that are really good for you to address based on criticality or risk towards your organization. And here's how you address them. Here's how you fix them. It's not just about,

you know, smashing and grabbing and being an awesome hacker and doing all those crazy things. It's really about making the customer better, making the people that you're testing better in the long run. I think that's really important that we lose a lot in this industry of is that, you know, most folks just focus on, hey, I'm the best hacker in the world. I just destroyed everything. Good luck, you know, and kind of leave it there. Whereas, you know, as an industry, we really have to focus more on the teaching aspects around, hey, how do they actually fix this? How do they actually address it? What are the things that we can do to get better and make it harder for attackers to get in? And that's really our

ultimate goal. Dave met with the company and coached them how to shore up their defenses. As you may have guessed, this episode and past episodes, those RFID badges, yeah, they're vulnerable to cloning, which makes it easy to bypass those locks. Some companies have moved away from using badges like this and have switched to something else, like maybe a magnetic stripe card, which has its own weaknesses, but it makes cloning it a little bit harder. Other companies require a biometric ID to get into doors,

like a fingerprint or an eye scanner. And I've been in a big data center that did all this and more. An RFID badge just to get into the parking lot, a pin to get into the building. Then to get into the data center area in the building, you had to swipe a magnetic card, enter a little chamber, which weighed you, and then did a retina scan, and then allowed only one person through at a time with a guard watching every single person coming in and out. Then to top it off, I needed an old-fashioned regular key to get into the actual cage where my client's servers were. Oh, and as a side note, I thwarted all this security a few times and snuck my girlfriend in without going through any of this, but that's another story. So Dave gave a bunch of tips to this client.

You know, when we debriefed them, we worked with them again the next year and they had really taken the results and addressed them. They ended up switching to a different solution and away from proximity cards. So they actually did a technology improvement and enhancement and put also additional controls in place. Like instead of that back area being there, I get to go through kind of like man traps and things like that to get in and out of the building.

So they did a really good job. And we actually got busted the year after that. So in both the retail location store as well as the corporate headquarters. So it was kind of a good success story. This episode is sponsored by Delete Me. I know a bit too much about how scam callers work. They'll use anything they can find about you online to try to get at your money. And our personal information is all over the place online. Phone numbers, addresses, family members, where you work, what kind of car you drive.

It's endless. And it's not a fair fight. But I realized I don't need to be fighting this alone anymore. Now I use the help of Delete.me. Delete.me is a subscription service that finds and removes personal information from hundreds of data brokers websites and continuously works to keep it off. Data brokers hate them because Delete.me makes sure your personal profile is no longer theirs to sell. I tried it and they immediately got busy scouring the internet for my name and gave me reports on what they found. And then they got busy deleting things. It was great to have someone on my team when it comes to my privacy.

Take control of your data and keep your private life private by signing up for Delete.me. Now at a special discount for Darknet Diaries listeners. Today get 20% off your Delete.me plan when you go to joindeleteme.com slash darknetdiaries and use promo code darknet at checkout. The only way to get 20% off is to go to joindeleteme.com slash darknetdiaries and enter code darknet at checkout. That's joindeleteme.com slash darknetdiaries. Use code darknet. Support for this show comes from Black Hills Information Security.

This is a company that does penetration testing, incident response, and active monitoring to help keep businesses secure. I know a few people who work over there, and I can vouch they do very good work. If you want to improve the security of your organization, give them a call. I'm sure they can help. But the founder of the company, John Strand, is a teacher and he's made it a mission to make Black Hills Information Security world-class in security training. You can learn things like penetration testing, securing the cloud, breaching the cloud, digital forensics, and so much more. But get this, the whole thing is pay what you can. Black Hills believes that great intro security classes do not need to be expensive,

and they are trying to break down barriers to get more people into the security field. And if you decide to pay over $195, you get six months access to the MetaCTF Cyber Range, which is great for practicing your skills and showing them off to potential employers. Head on over to BlackHillsInfosec.com to learn more about what services they offer and find links to their webcasts to get some world-class training. That's BlackHillsInfosec.com. So yeah, might as well start out with your name and what do you do? All right. So my name is Clay. I'm an Infosec engineer.

So I work at a university. So I'm Infosec for an entire school. Yeah, we have a lot of Linux machines. We've also been migrating a lot to the cloud. Clay does a lot of IT work for this school, this big university, ranging from anything from coding to system administration, web app security, setting up the network, and even doing penetration tests. I also help generate best practices. If sysadmins or programmers have

questions, they can come to me. And if I don't know the answer off the top of my head, I do the research and get back to them. One of Clay's responsibilities is to take care of threats that are found in the network. And one thing he battles a lot with is crypto miners. Being in this environment, in academia, it's really hard to have all of the systems on the network managed a managed system is just a computer that clay is aware of and can access and somewhat control so an unmanaged computer on the network clay has no control over it and may not even know it exists obviously if you're a system admin you want to be able to access all the computers on your network but at the same time it's impossible to manage every computer at a university.

Students bring their own devices into the network all the time. But yeah, something Clay battles with frequently is crypto miners. And this is where a student might install a Bitcoin miner on a computer in the lab or in a research center. And then the Bitcoin miner will consume a ton of cpu or graphics processing to try to generate some crypto coins and automatically get deposited into the student's wallet and we'll actually get an alert when this happens uh we have an ids in place um so that's typically how we'll we'll be notified of these events an ids stands for intrusion detection system this is a device that inspects every packet coming in or out of the network and checks to see if that packet matches any known signature for some kind of security issue.

In this case, it matches the signature for crypto mining because when it connects to the blockchain or pools or whatever, it then recognizes this as a miner and triggers an alert. Yeah, then the fun begins. We can isolate the machine, usually myself and assist admin, just so we have two pairs of eyes. It's always better than one. We'll go and we'll start the investigation. We'll look at running processes. We'll look at the

bash history, things like that. We'll look at open ports, if it's running netstat, so we can see if it's listening or if there's a connection that is established there isn't always but yeah we look at all those things it's fun because when detecting something is wrong in the network and then you find it and isolate it and squish it it's just exciting and as a sysadmin most of your job isn't tackling live security issues so when

it's happening yeah it is exciting. And honestly, it's always fun to catch someone in the act that's doing something they shouldn't be doing and go and bonk them on the head and tell them not to do that anymore. Because they're usually blown away that you figured out it was them. So this paints a picture of what kind of stuff Clay works on. But Clay also sometimes does this on the side a little. He has a few clients and he helps secure their network. And when they have an issue, they call him up.

One day, they give him a call. So this was just a normal day at work. I got an email from the client. Something doesn't seem right. Something doesn't look right. The application is acting kind of funky or, you know, something's off. She's like, okay, well, let me, you know, grab a cup of coffee and come and check it out. Basically, one of the faculty or staff at the school was complaining about a slow website,

which was running Linux, which is a server that Clay can access and check into. The Linux server runs this website, and Clay looks around at the thing. He's checking things like, does the website load? Yeah, it does. It's working okay. Is the server running high CPU or is low on disk space? No, that's fine too. Things seem okay, and maybe a junior level sysadmin would stop here and just try to let it sort itself out, reboot the machine and be done. But Clay is not a junior sysadmin. He's a senior security engineer. So he takes another look. I want to see who's logged in, if anyone is logged in. He checks here to see if any developers are in there messing around or another sysadmin doing something or anyone fiddling with this. He doesn't see anyone else there, so he does his usual rounds. Is the database up and running?

Is the VPN up and running? How does that look? So just standard stuff, right? Looking over the whole application stack, making sure things are running. Doing a quick top, making sure nothing is running extremely high, taking up a lot of load, using a lot of memory, those sorts of things. At first glance, all this seems okay still. But then, a second look through everything, he finds something. I found that there was a root shell open. A root shell is open on this server. Let me explain.

On Linux, the super user or administrator is called root. This user account has full privileges to everything on the server. What Clay sees is that someone is logged in as root. Having a shell is another way of saying someone's logged into the command line. Now, you and I might think, oh, it's just another administrator doing work. But the school has set up the network correctly. See, it's not good to allow anyone to log in as root

because you have no idea who that is logged in as root. And every hacker on the planet knows this username exists and will try to brute force the password to it if you give them a chance. So the school set it up so that individual users like Clay's username has access and admin capabilities and super user privileges. So Clay knows that under no circumstance should anyone ever be logged in as Root. But here there is. Someone is logged in as Root. I immediately start thinking to myself, oh crap, we do have a compromise. It is a root level compromise. So now my heart starts pounding

a little bit more stronger and I start thinking, oh hell, what the hell do I do next? Okay, so in the physical world, this is equivalent to coming home and seeing your front door is wide open and there are muddy tracks leading into your house. The feeling of discovering someone is in your server

that shouldn't be there and for them to have root level access to it is really, really scary. Do I look to see how they got in and block it? Do I just sever their connection and hope they don't come back

before I can patch the stuff? Or what? So all of these thoughts are just racing through my mind. Clay takes a step back and a deep breath. All of a sudden, he's hyper-focused on this issue now. Anything else that he was thinking about doing that day is no longer in his thoughts. This is all he can think about. So I said, well, the best thing to do is determine how the hell they got in.

Right. And, and try not to make a lot of noise on the system while I'm doing this, because I don't know if they're active, if they're like sitting at the shell, actively looking at stuff, or if they're, they just they're like sitting at the shell actively looking at stuff, or if they just have a shell open and it's in the background, or maybe that shell is just waiting for a command or something. I don't know exactly what's going on. So I want to be careful, and I want to go slowly, and I want to find out what the hell happened.

So being a web application, I knew it had to be probably SQL injection. Cross-site scripting thing probably wouldn't lead to this level of compromise, at least not right away. So I started looking at the database. This web server was running an SQL database. And this is where all the data is stored for the website. Clay was looking at the history of commands executed in the database, trying to find anything unusual. And I started looking at some of the pages that use the database more heavily than others. And I did start to notice some weird shit in the database, in some of the tables. And I was able to isolate it to one of two pages that had this vulnerability. And so I visited those pages and they looked okay. Nothing was out of whack or funky or no errors were being displayed or anything like that.

So I thought, let me just move these files and get them, just move them out of the way so they're not accessible anymore. Clay determined that a couple of pages on this website were probably where the hacker got in. So he just took those pages offline, making it so further intrusions couldn't occur. Now, removing how this person got in is one thing, but it doesn't remove them from your server. The root user was still logged into the server. But if Clay kicks them out now, they probably couldn't get back in.

I tried to SU to root at some point during this whole thing, and I couldn't. So I knew they had changed the password. Clay knows the root password to this machine, but it wasn't letting him log into it. Yikes, this just got scarier. Not only is there someone in his server, but they're actively changing the passwords on it.

Yes, yes, exactly, yeah. So now I'm freaking out because this system might have to be completely torn down and rebuilt. Yes, we have backups, thankfully. I'm not sure, but I don't think you can kick out the root user unless you are logged in as root yourself. And when you know you have an active hacker in your network, it's hard to know if you're cleaning up everything when you do kick them out. They might have an open back door or pivoted to another computer. It's so stressful.

So this box is hosed. I'm going to have to call the data center and have someone go to the machine, physically unplug it and call it a day and like figure out what the next steps are. Right. So to rebuild. So that's when I said, well, I really don't want to go down that route. What, is there anything I can do? Clay is logged into the server and decides to look at the files located under etc, password, and etc shadow. These files contain a list of hashed passwords for each user, and Clay is able

to see the hashed password for root. Now, this isn't the actual password. It's a representation of the password, a long string of crazy characters that you get once you run it through an algorithm. When you type your password in, it runs it through that same algorithm again, and if it's a matching result as that long crazy string, then the passwords match. And that's when I started to run John the Ripper on it. John the Ripper is a tool used to crack passwords. It'll try thousands, millions of passwords and run it through that algorithm to see if it has a matching hash. At this point, Clay has become a hacker himself and is doing exactly what a hacker would do to crack passwords to break into a computer. It's just that Clay is trying to break into his own server.

Now, to run John the Ripper, this takes a while. Clay doesn't have a beefy cracking station, so he goes on to investigate more about what this guy's doing and starts looking at database tables and other stuff. But within a short time, there was a hit, a match on the password. John the Ripper found what the root password was and surprised Clay how quickly he actually found it, which usually means it's not that complicated. And yeah, so the password was Mark 2002.

I will never forget it. He won't forget it because it's awesome to use hacker tools to outsmart a hacker and for it to work so effectively. Great job, Clay. Yeah. So now I'm starting to feel good. I'm much more optimistic at this point. So I start thinking I need to boot this guy off. I need, I've moved the files out of the way. I can lock down the database. I can just shut off the database, right? I'll shut the database down. We'll put up a notice we're down for maintenance, not a big deal. And then I can get back to that later, later in the evening or whenever.

But how am I going to boot this guy off? What can I do to lock it down further? Monitoring can I put in place that isn't already here to help me, to help throw an alert if the person should get back? So I start developing a plan. So it's not so easy as just kicking the hacker out of your network. You have to make sure you have everything in place. If you kick the hacker out and they just have a way to come right back in, you essentially did nothing. Because if they set up a backdoor and they just come in through that, you might not even know they came back.

Clay has a plan to kick this person out, but he's going to have to do it quickly so the hacker doesn't just come right back in. He starts to look at the server to plan every person out, but he's going to have to do it quickly so the hacker doesn't just come right back in. He starts to look at the server to plan every step out. I need to SU to root, so I do that. Okay, so now I'm root. Are there any cron jobs in place? I have his shell process, I can kill it. Great. Let me look at IP tables and only allow SSH access to, you know, or from one or two IP addresses. That's it. So I had that lined up. I need to take down the database, put up a maintenance notice and change the password again. And also look for evidence of files that may have been dropped or evidence of other back doors that might be listening or ready to listen. So I'm looking in, you know, slash temp, looking in roots directory, looking all over for clues like that. Just doing it quickly, but definitely coming back to that

once I kill the process and change the password. Okay, at this point, his definitely coming back to that. Once I kill the process, it changed the password. Okay, at this point, his plan is all sorted out. Each command is typed out on a notepad, just ready to be pasted in. He double checks that there's not anything else that he missed, and he thinks he's all ready. Three, two, one, go. Kill the login for root, change the password, turn off the database entirely, put up a maintenance notice, and block this IP from ever connecting to this again. He thinks that's it.

That's all the commands. He's watching the connections, but not seeing anyone try to come back in. Warped. Clay feels... Amazing. Fucking amazing. Yeah, my heart was still racing. I wasn't sure if it was all going to work, right?

Because I don't know what everything that had been done. The techniques weren't like very advanced. There wasn't a bunch of like cleaning up things or, you know, cleaning up the bash history or, you know, scrubbing the logs or anything like that. I saw no evidence of that. So it didn't seem very high-tech. So I was optimistic. Regain control of the machine and keep the person out. I'm still in response mode. So I had to reach out to the owner of the system,

let them know what has transpired, and then I immediately start planning next steps. Yeah, I want to run to the bar and have a beer real quick, but there's really no time for that. I also reached out to the data center just to let them know what had happened. I just felt like that was a responsible thing to do and briefed them on what I did and the steps I had taken and that the vulnerability was identified and essentially fixed. Yeah. And then at that point, I thought I had done my due diligence of informing all of the stakeholders. And now I could take a deep breath and start focusing on forensics a little bit because I wanted to save all the things, right? I wanted to save the SQL logs.

I wanted to save bash history. I didn't find any funky binaries, so that was good. And then I had to start cleaning up the database, which was a task and chore. Yeah, and then I had to start locking down, you know, thinking about other ways to lock down the system and other monitoring to put in place. Clay went through as much of the logs as he could to retrace every step the hacker took, because that's

important, and it's the right thing to do. If you can figure out how they got in and how long they were there and what they did while they were there, you can improve security immensely. He determined the hacker had only been in there for a few days, and they got in by using an SQL injection through the webpage that he found. He figured this out by looking at the SQL logs, and through this, they were able to get a shell on the server, which then they escalated their privileges to root. And once he figured all this out, Clay rolled the server back and database back

to the day before the hacker got in, so that if the hacker left anything behind, it was completely gone. And they fixed the SQL injection that this website had. I think Clay did a great job handling the situation. Besides doing sysadmin work and chasing hackers out of the network, Clay also is an organizer for the Whopper Summit. The goal is to really bring together different communities that are all really involved with hacking and making, building and breaking things. It's going to take place at the end of March next year, and it's more than likely going to be right outside the Philadelphia area. Yeah, so if you're around Philly next March, go to the Whopper Summit. That's W-O-P-R, which is the name of the computer in war games. It just sounds so much fun.

This episode is sponsored by SpyCloud. With major breaches and cyber attacks making the news daily, taking action on your company's exposure is more important than ever. I recently visited SpyCloud.com to check my darknet exposure and was surprised by just how much stolen identity data criminals have at their disposal. From credentials to cookies to PII. Knowing what's putting you and your organization at risk and what to remediate is critical for protecting you and your users from account takeover, session hijacking, and ransomware. SpyCloud exists to disrupt cybercrime with a mission to end criminals' ability to profit from stolen data. With SpyCloud, a leader in identity threat protection, you're never in the dark about your company's exposure from third-party breaches,

successful phishes, or infostealer infections. For our third story here, we'll hear one from dan tentler which goes by this so i'm this uh i run phobos group we do interesting work for interesting people now this was working for a company a while back as a penetration tester and security consultant it was a good company but for some zany reason the company ran out of money, stopped paying the employees. So he started his own company, which was pretty much the same company. But since the people were all looking for jobs, this just scooped them up and took the talent with him, including his co-founder Ali. They called it the Phobos Group. Phobos Group was formed essentially because myself and Ali were just fed up with perpetuating the cycle of compliance is the bare minimum, so we're just going to do that.

Oh, God, we're breached. Oh, what do we do? Oh, fire the CISO. Like, rinse and repeat, right? do absolutely the bare minimum or slightly below the bare minimum, get horribly steamrolled by malware, blame somebody, fire them, give them several million dollars as a golden parachute, bring in someone else and the cycle repeats. And we're like, this is dumb. Like all of the stuff that's happening, this is entirely smoke and mirrors and snake oil.

And we're done. This is stupid. We're out. So our core offerings are simulating what real bad guys do. This is opinionated and talented in securing clients' networks and testing their security in a real and meaningful way. He doesn't sugarcoat it. He finds the dirty parts of the network and tells you how important it is to clean it up. Let's see what's a good one.

Yeah, I have one. So there's a company that came to us at one point a while back that said, we have a bizarre problem and we're not sure how to fix it. Typically, Vist's clients might ask for a penetration test to get things started. But in this case, they told Vist they have a problem with a specific employee. They are having discussions with folks or making comments about stuff that are like private personal emails that piece that people have have written uh not even on their work account and it's making these people very very nervous because they're beginning to think that they're being surveilled and to to add to exacerbate that problem um this person is also kind of a creeper and he keeps trying to flirt up the girls in the office and somehow he knows who's single and who's not single despite the fact that

nobody talks about that work they basically knew that the dude was a problem and said like we need to find a way to get this guy out of the company please help that's that's unusual okay it's very unusual but when you bill yourself as a company that doesn't do wham bam thank you ma'am pen tests and rubber stamp security you get the interesting stuff and not the boring stuff okay now i'm aware that the vast majority of security threats in the network come from the inside. Something like 60% of all attacks are carried out by people in the company because they're doing things like simple human error. You know what? I'm guilty of that. I've accidentally taken down a whole network myself. I got the ID 10 T award for that one. And sometimes

people just accidentally share their passwords, like when they're forwarding email chains. Sometimes you just have someone evil in the company, a wolf in sheep's clothing. So at this point, it sounds like this creeper in their office is somehow getting data from the employees, which is making people feel uncomfortable. It sounds like it could be an insider attack. We start asking questions and the story is basically that there's a guy that works for this company and he was like in help desk

or in like low level IT and he was your typical office creeper sociopath and he was making all the women in the office uncomfortable and he was abrasive and he was not pleasant and he was not friendly and he was not friendly he was difficult to work with but because this company the way they explained it as well we're family owned and we um we don't

want to put a bad taste in people's mouths so they tended to not fire people they wanted to try to like get people to leave on their own accord so in this in this instance their genius their galaxy brain idea and and i hope you have whiskey on hand was to put this guy in a position that he would hate so much that he would quit on his own accord and the problem would solve itself right like make the person so miserable that they leave on their own accord not an uncommon thing web sense did that to me took several years that's a whole other story but um yeah not not an uncommon thing if you have a non-confrontational not a type personality management and leadership then you want the problem to fix itself so you you orchestrate you know uh a lateral move for this person and say, oh, hey – and you dress it up as a need.

Oh, we need you over here way more than we need you over there. Like this would be so great for you, and it's all lies. So what they did was they promoted him to the head of security. What? They promoted him to head of security? They took a guy that was a problem. They took a guy that was making women in the office uncomfortable, and they promoted him to the head of security. So they gave him the keys to the kingdom.

Okay, this is going to be interesting. This is going to try to look around the network to find some kind of reason why this guy should be fired. But this guy has full control of the entire network. He has full access, administratively, to the entire infrastructure of the entire network. He has full access administratively to the entire infrastructure of the company. So you have to presume that if he's spying on other people, he would be spying on us out of a sense of self-preservation. So it's almost certainly, since they contacted us from their work email

and they had several calls with us from their boardroom, almost certainly that guy had access to those conversations. And he is almost certainly aware that he is now being investigated and we're working on it. Well, this is going to be a challenge then. It's like a backwards game of cat and mouse, where the mouse is trying to catch the cat doing something illegal or blatantly against the company rules so that he can be fired.

And I've also heard of companies that just won't fire anyone. Like state and government agencies are like this often. You almost never hear of anyone getting fired because of poor performance. So this gets to work. Step one, get out of band, out of earshot. Basically, the bosses need to get off the network to avoid the spying eyes of this guy. We had them bring in personal machines or work entirely with personal machines so that there was no way for this guy to move laterally onto their equipment. We forced them to set up 2FA. We forced them to change all their passwords. And then we were looking at their equipment to make sure that it was not phoning home. So the first thing we had to do to get that engagement off the ground was basically teach

the customer how to do out-of-band communications. And then once we got to the point we were doing out-of-band communications, they started relaying to us the ways that he was horribly breaking his own OPSEC. Okay, so even though they had stopped using the corporate network altogether and were using cell phones and text messages and a different email system altogether, this guy still ended up finding out stuff that they were talking about. For instance, they went to lunch one day and this head of security creeper guy says, so how'd that meeting with Phobos go yesterday? And they texted Viz and told him about this. And he's like, what? How did they know that?

But something like that actually narrows down the possibilities pretty well. He must know this because either one, he bugged Vis, which is not likely, or two, he bugged the boss's office, or three, he was on the call. Those are the only three possible scenarios. So the team from Phobos goes into the office and starts snooping around. He went into meeting rooms looking for any unusual equipment, anything taped underneath a table or strange devices stuffed in a potted plant in the corner of the room and he found stuff all kinds of stuff the dude was physically bugging like the boardroom and the meeting rooms he put like cheap you know

buy them buy them on amazon buy them on spy shop type deals he put them inside of the receptacles for power he like took the screw off like Enemy of the state style, like Jason Bourne mode, got some cheapy audio bug and put it in the power receptacle in the boardroom. Okay, good start. Get rid of the bugs in the office. Now the team starts looking at the network. We were granted

administrative access to stuff, and we were able to find some of his implants, which weren't even implants. They were like PowerShell scripts to do stuff. But he was using the system against the system. So like, you can configure phone systems to record every phone call, so that's what he did. You can configure Windows Active Directory to use GPO to do basically

simple surveillance on all the workstations. He had the phone system configured to record and save every phone call made so that he could review them. And he configured, I think it was a GPO that he set up to take screenshots of people's machines and send him screenshots. So he was getting screenshots of every employee

and he was recording every phone call. At the end of the day, if the objective is to surveil the office, and if you're the head of security, then you don't need to use spyware. You can use the system to surveil the system. It's been designed that way. So it's just a matter of who's driving it, right? Now, at this point, the guy is starting to become aware that Viz is on to him and investigating him. So he started trying to block Viz from doing certain things.

He was, but he was not... He wasn't a security person. He was a help desk guy that got promoted to the head of security because they wanted him to try to quit. So in terms of technical aptitude, while he was fairly technical, he was not a security guy by nature. So it wasn't very difficult to run circles around him. So at this point in the investigation, one of Vys's co-workers asked to see the data center. When they opened the network closet to see what was in there, it was all gone. And what this guy did was he took all of their hosting and moved it to Ukraine, took all their on-premise stuff from their office and took it into his garage, put it in a two-post

rack in his garage, got business cable, and then started doing things like writing off his mortgage and all of his power and all of his water and all of his utilities as business expenses because he was hosting. He basically on paper said that he was the hosting facility for the company. What a crazy weirdo. He moved all the servers to either a hosting provider he had full control over or his garage. And because he was the head of security, he had the authority to make all these decisions and execute them. Then he was issued, I don't know why, a corporate Amex. Amex is their corporate credit card, American Express. And it's only to be spent on business related things like traveling to clients' locations or buying things for work. And on that corporate Amex, he put all sorts of things like his groceries and his wedding reception.

And that's it was shortly after all that happened is when we got called in. So one of the first things we started asking is who on earth is approving his expense? Yeah. And then it turns out like it was his boss guy and his boss guy was also completely oblivious and didn't even bother looking. It was like, oh, well, this is just temporary, right? He's going to go. But he was putting almost 200 grand a year on this on this corporate Amex and like nobody in question. This is smart. He followed the money, always follow the money. And when he showed this to the executives that this guy spent in this much money, they sat down his boss and had a really difficult chat with him. At this point, the company has a solid case against this head of security creepo guy to fire him.

But maybe they should do more than just that. There was this process of producing enough evidence to basically turn him over to law enforcement. And it was just a matter of documenting all the stuff that was discovered with photos and logs and going through all the, like basically building a timeline and then turning it over to the FBI and saying, this guy is broken. We don't know how many laws. So this and his team did just that. They collected all the logs and evidence of any potential laws this guy broke, put it into a report and turned it into the bosses.

In the state of California, the guy was breaking all sorts of laws. I'd have to go look them all up to get you the specific examples, but if you just look for, like, employee privacy laws, you're going to find pages and pages and pages and pages of stuff. There are laws against how you can surveil your own employees in California, and that's where this company was. Yeah, he went way above and beyond what the laws allowed here. So Viz had lots of evidence that he was breaking all kinds of laws, and he turned this into the client. They thanked Viz for his help, and that was the laws allowed here. So Viz had lots of evidence that he was breaking all kinds of laws

and he turned this into the client. They thanked Viz for his help and that was the end of that. The company did with that guy is kind of unclear, but Viz heard that the FBI built a solid case against him and came in and arrested that guy and he left the office in handcuffs. This investigation opened the eyes of the bosses

to many other problems in the company. Like who hired this guy? And who let all this stuff just keep going on and on and on? The company eventually hired a bunch more IT and security staff that weren't toxic or crazy, and they took back control of their own network. And the company changed the way they view firing people. Now, they've learned how much it can cost a company if they don't fire certain people. The damage from just this one person was enormous. $200,000 in corporate credit card charges,

firing a lot of staff, and spending months getting the network cleaned up and back to a secure place so that they can manage it themselves. All that adds up, and it would have just been a lot better if they just fired him instead of trying to place him into a situation to get him to quit. You've been listening to Darknet Diaries. A huge thanks goes to Dave Kennedy for sharing his story. You can find him on Twitter. His name there is HackingDave. Or visit TrustedSec.com. Also, thank you,

Clay, for that awesome story. And Clay is inviting all of you to check out the Whopper Summit. That's W-O-P-R-S-U-M-M-I-T.org. And if you're in the Philadelphia area, check it out. I first heard Clay's story on the Getting Into InfoSec podcast, which is a great podcast that interviews people on how they got into InfoSec. I was even a guest once. So if you want to hear stories about how people got started doing these kind of things, check out the podcast, Getting Into InfoSec. And finally, thanks, Viss, for your story. Catch him on Twitter. His name is V-I-S-S or at phobos.io.

This episode was created by me, Venomwares. Or you could just call me VMwares for short. My name is Jack Recyder and I got some production help this episode from the modest

Michelle Martin the music was created by the Trippy Troubadour Breakmaster Cylinder see you in two weeks