Darknet Diaries - 86: The LinkedIn Incident

Episode Date: March 2, 2021

In 2012, LinkedIn was the target of a data breach. A hacker got in and stole millions of user details. Username and password hashes were then sold to people willing to buy. This episode goes over the story of what happened.For a good password manager, check out LastPass.SponsorsSupport for this episode comes from Quadrant Information Security. If you need a team of around the clock analysts to monitor for threat in your network using a custom SIEM, check out what Quadrant can do for you by visiting www.quadrantsec.com.Support for this show comes from Thinkst Canary. Their canaries attract malicious actors in your network and then send you an alert if someone tries to access them. Great early warning system for knowing when someone is snooping around where they shouldn’t be. Check them out at https://canary.tools.Support for this show comes from Linode. Linode supplies you with virtual servers. Visit linode.com/darknet and get a special offer.

Never get fascinated with the cybercrime supply chain? It's never a solo hacker doing the whole thing. There's a lot of layers to this onion. So let's say a hacker breaks into a place and steals a bunch of information from some company. Well, next he'll typically want to sell that data to make some money and do it again. So now you've got to find a buyer. But before we even get to the buyer of stolen data, there's sometimes brokers involved, people who have negotiated deals between hackers and buyers. So you might go to one of these brokers, offer a percentage for selling the database to someone. Now it's on them to find

someone. But when the broker finds a buyer, sometimes one side doesn't trust the other. So they bring in a trusted third party, an underground escrow agent, if you will, who will wait for both the cash and the database and then make the trade. Okay, but then what does the buyer do with this database dump? Well, if it's full of email addresses, they might use it to send spam to people. But of course, the spammer isn't selling anything themselves. They're typically promoting someone else's business, a porn website or a pharmacy. And it's just fascinating for me to think about that sometimes. It's never about the data breach itself, but what happens to that data after it's stolen? These are true stories from the dark side of the internet.

I'm Jack Recider. This is Darknet Diaries. This episode is sponsored by Delete Me. I know a bit too much about how scam callers work. They'll use anything they can find about you online to try to get at your money. And our personal information is all over the place online. Phone numbers, addresses, family members, where you work, what kind of car you drive. It's endless. And it's not a fair fight.

But I realize I don't need to be fighting this alone anymore. Now I use the help of Delete.me. Delete.me is a subscription service that finds and removes personal information from hundreds of data brokers' websites and continuously works to keep it off. Data brokers hate them because Delete.me makes sure your personal profile is no longer theirs to sell. I tried it and they immediately got busy scouring the internet for my name and gave me reports on what they found. And then they got busy deleting things. It was great to have someone on my team when it comes to my privacy.

Take control of your data and keep your private life private by signing up for Delete Me. Now at a special discount for Darknet Diaries listeners. Today, get 20% off your Delete Me plan when you go to joindeleteme.com slash darknetdiaries and use promo code darknet at checkout. The only way to get 20% off is to go to joindeleteme.com slash darknetdiaries and enter code darknet at checkout. That's joindeleteme.com slash darknetdiaries. Use code darknet. I'm sure you all know what LinkedIn is, right? It's the social network for professionals. You pretty much start your account by posting your resume of where you worked and what you did there.

And you can use the site to look for jobs and connect with other professionals in your field. It's pretty popular in the U.S. In 2012, a person wanted to hack into LinkedIn and get as much user data as they could. But how are you going to get into the network of LinkedIn? This is a major Silicon Valley company made by some really skilled engineers and administrators. They would certainly be following all the latest best practices for securing a network by doing things like securing the front door to the network by putting a big firewall up to block all non-critical traffic from coming in and inspecting it for malicious activity.

Then they'll conduct security audits on all the internet-facing systems to make sure there's no security holes. And of course, they'll be running state-of-the-art monitoring tools and antivirus tools to watch for any intrusions. And they did all that. The front doors of LinkedIn's network was airtight. So the hacker would have to find another way in. He knew that engineers at LinkedIn had access to the corporate network

when they were remote. I mean, today it's obvious that a lot of companies have remote employees, but back in 2012, there were LinkedIn employees who had remote access into the network. That is, they didn't have to be physically in the office in order to access the database or other critical systems. So the hacker set out to figure how exactly do some engineers get remote access into the network, and he concluded they must be getting in through a VPN. A VPN is a way to securely connect into a remote network. The traffic is encrypted from the edge of the corporate network all the way to the user's computer, wherever they are in the world. And that's just it. If there's a backdoor entrance for employees only, it would also mean that the hacker could try to get in through that.

So the hacker starts looking on LinkedIn's website for people who worked there. Engineers, system administrators, anyone who might have access into that VPN. And so he looked around for a victim. Which, by the way, this is the reason why I don't like posting my information on LinkedIn, because you can easily search for all the people who work in a specific company and then figure out who the admins are there who are probably posting things like, oh, I'm good at Cisco firewalls and Oracle databases. And they might even be posting what versions of Oracle they're good at, which is a clue to any hacker to know what to expect once they get in.

But what this means is that it's pretty easy to find a target and narrow your sights on them by just looking at who's on LinkedIn. And this hacker found a LinkedIn engineer who probably had remote VPN access as well as access to the database inside. And the hacker zeroed in on this guy. The hacker saw this engineer's LinkedIn profile. And on his profile, there was a URL to this engineer's personal website. Basically, it was like thisengineersname.com.

The hacker went to the website to check it out. It was just a basic about me type blog. It said, hello, I'm a site reliability engineer at LinkedIn, and here are my hobbies and things. The hacker poked around here for a bit, but couldn't find anything to exploit. But he looked at where the site was hosted, and it was hosted on a residential IP address. Hmm.

It seemed like this LinkedIn site reliability engineer was running a web server out of his house. This means there are open ports from the Internet into his computers. The hacker thought, well, if I can get into this engineer's home computer, this might give me a way into LinkedIn. So he looked to see if there are any other websites also hosted at this IP address. And he found one called Cockguide.com. looked to see if there are any other websites also hosted at this IP address, and he found one called cockide.com. He browsed around here, and this is a much bigger blog-type website.

Cockide.com was a site ran by this engineer's friend. He just hosted it for him. And there's videos of pranks and pictures, and it's basically a blog. But this site was built using PHP as the back-end technology. And the hacker started looking for ways to exploit this site. He found a way to upload files to the site, and he uploaded a couple malicious PHP files. One was specifically called madness.php. Now, if a hacker can upload their own PHP program to your website and get that program to execute, then the hacker can take over that

computer that's hosting the website. Because when you go to view the file through a browser, it's going to execute whatever code is in that PHP program. And you can configure it to give you remote access to that computer. And that's just what this file did. The hacker got shell access to the website cockyde.com, which was being hosted in the same place this LinkedIn engineer's personal blog was hosted. Once he got into the web server, he started scanning for other IPs on the network and found one, an iMac computer. He found that this iMac had an open SSH port, which allows people to connect to that iMac. So he started trying to brute force login to that iMac. For the username, he used the first initial and the last name of the LinkedIn engineer.

And he just started hammering away, trying thousands and thousands of passwords, looking for one that worked. And this was all happening in February of 2012. And for days, this web server was attacking the iMac all within this engineer's house without the engineer knowing it. And after a few days of trying

thousands of passwords, one worked. He got a hit for a valid username and password. So the hacker logged into that iMac with this and looked around. First, he realized this is a person's personal iMac. It's not a LinkedIn computer or even an official work computer. Then he discovered that this web server that he got into was running on this iMac. Yeah, it was actually running on a virtual machine in the iMac. And I find this fascinating because in essence, the virtual machine was the only thing exposed to the internet. But the hacker got into the virtual machine and then got into the host computer from there. And it's fascinating to me because this shouldn't happen, but the way he did it was through the virtual IP interface. Because I always

wonder how it's possible to escape out of a virtual machine and onto the host computer. And here's an example of when a hacker did that. After looking around the iMac for a while, the hacker stumbled upon the keys to the kingdom. Literally, he found a private key to LinkedIn. This was the key that the engineer could use to log into LinkedIn with. See, when you log into certain systems, you could use a username and a password. But another option is to use public and private keys, where the public key is put on the server you need to access and your private key is on your own computer. So when you connect to the server, you authenticate using the keys. This is all done automatically and saves you from having to type passwords.

But when the hacker saw the private key, he snagged it right away. But where does this private key connect to? Well, the hacker had to look around a little bit more to find out. And that's when he found a set of VPN profiles that allowed this person's iMac to connect to LinkedIn. And the profile contained everything they needed to connect. The server name, the IP address, the username. The only thing missing was the private key, which the hacker just got. The hacker then took this VPN profile and credentials

and connected directly into LinkedIn's VPN server. Now, here's where the hacker made a major mistake. He made this connection into LinkedIn from his home, which was in Moscow, Russia. And LinkedIn is in California. Well, there was nothing to stop this connection coming in from Moscow, though. So he just got right in. And from here, he was able to find his way around the network looking for the user database, and he found it. And he was able to log into that and grab the username, password hash, and email addresses of as many LinkedIn users as he could.

And after that, he logged out and disappeared. LinkedIn had just been breached, but they didn't know it yet, and they wouldn't find out for another three months. The first moment when LinkedIn learned about this was through a forum called InsidePro.com. This was an underground criminal forum where you could buy and sell stolen data from hacks. Someone was offering LinkedIn user data for sale there. The team at LinkedIn saw this and immediately sprang into action. First, they needed to verify that the data being sold online was real LinkedIn user data. They compared what was in that sample database with their own database, and sure enough,

the hashes matched. The horror and the fear you get when confirming that you've just been breached, it's indescribable. Now, LinkedIn's response for something like this is a four-step process. First, you confirm, contain, remediate, and do postmortem. They just confirmed that they were breached. Next is for them to contain the problem. Is the hacker still in the network? What did they steal? How did they get in?

Can we block them from getting in again? All these questions needed immediate answers. LinkedIn engineers and security team took over a conference room and called it a war room. Something like 40 to 60 people from LinkedIn were all working on this incident. They were flying in from foreign countries to help. They had the security team involved, hunting through events and logs, looking for evidence. They had SREs, or Site Reliability Engineers, in there combing through their systems,

looking for traces of unauthorized activity. There were lawyers present. Their chief internet security officer was present and active, doing all kinds of triage. Other executives were in the room too, because this was the most important thing going on at LinkedIn at the time. So the atmosphere was heavy and intense. The first clue they got was from the VPN logs. The LinkedIn security team saw that one of their California-based engineers had logged into the VPN

from Moscow numerous times. So they called him in for an interview. Hey, have you been to Russia lately? No. Did you use any kind of Russian proxy lately? No. Did you give anyone in Russia your login details? No. The security team was on the trail. This was a major clue. They found out that this engineer had been connecting to the corporate network from his home iMac. And I'm going to guess that probably wasn't allowed. But the security team asked him to bring that iMac in for an examination. So let's back up a month, actually. So a month before LinkedIn even knows this happened, the hacker was looking through the database that he stole. It contained email addresses, usernames, and password hashes. This isn't the password itself. It's a representation of the password after it goes through an algorithm. So in other words, you couldn't see anyone's password in this dump.

But the hacker wanted to find a way to crack these passwords. So he posted a few of the hashes to a forum asking for help on how to crack them. When LinkedIn was investigating this, they saw that old post which matched the hashes that were in their database. The situation was getting worse for LinkedIn. They're now saying that the hacker is actively trying to crack users hashes. And unfortunately for LinkedIn, they weren't yet salting their password hashes. Salting password hashes is an extra step that you do to make cracking hashes even harder. They were in the process of doing this, but when you have hundreds of millions of users, it's not easy to get it done. At LinkedIn, they have different designations for how serious an incident is. Code yellow is something that is

some kind of technical risk, like a server running over capacity, or they're not sure how to scale it properly, or degradation of service that isn't causing a whole outage but could at any moment. Code yellows happen every few months or so there. But as the LinkedIn team investigated this, they determined this incident was a code red, meaning it was business impacting. And because they were already seeing user data leaked to the internet, it meant this threat was certain. And it could be at any moment that the LinkedIn database dump was revealed to the world. I believe at the time it was only available to whoever would buy it, and it wasn't freely available for anyone to look at. And this

creates a tense and scary moment for any security team. Not knowing what or how much got stolen, and not knowing what the thieves plan on doing with it. There's a level of anxiety here, especially because this was already hitting the news media who was announcing this hack to the world. Lots of different teams were pulling logs and saving them for incident responders to comb through. It's best to have logging turned on so you can sort of go back in time and see what happened where. One problem though is that there's now a lot of logs to go through. And if you think about the millions of users who are on the site every day, trying to find a needle in a haystack is tricky. I think the day they discovered this or the day after is when LinkedIn called the FBI

to inform them of this breach. And the FBI was very responsive. They asked for logs and started interviewing people right away. LinkedIn saw that IPs were connecting in from Moscow. And so they took that IP and started tracing it through the network. Where did it go? What did it access? They were looking through SSH logs, Wikilog, server access logs, and they saw connections from that IP, which told them what user agent the hacker's computer had.

The user agent can tell you things like operating system, browser type, and version. The hacker's user agent was unique. It actually had the word Sputnik in the end, which isn't normal. Sputnik is the name of the first satellite to be put in orbit by the Russians, and this user agent just wasn't seen by anyone before that. Which makes me wonder if the hacker put it there as sort of a signature. With this extra information, engineers can now search logs for that particular user agent to see if that has any hits for it.

Because maybe the hacker wasn't using the same IP every time. Maybe they had come in from different IPs and different channels and different VPNs or something, but if there was a matching user agent, then you could know that that's probably the same person. Next, they looked at the public website to LinkedIn to see if anyone with a matching IP or a matching user agent was logged into any user's account on LinkedIn.com. And sure enough, there was some activity. The same IP and user agent was seen logging into 30 different LinkedIn accounts through the public website. This meant that the hacker had cracked some passwords that was in the data and was logging into LinkedIn with those users. This certainly elevated the concern for

LinkedIn. And back in Moscow, the hacker did in fact have a rather beefy GPU farm. See, cracking password hashes is process intensive. You have to cycle through millions of passwords, hash them, and see if those hashes match the hash in the database. And if so, you found a password. And graphics cards, GPUs, are particularly good at doing a lot of these little calculations like this. They can do a lot of simultaneous things at once. So the hacker was running the database dump through this password cracking station he had. And when he'd get a match on someone that he found interesting, he'd try logging into LinkedIn to verify it. And it did in fact work. He was logging into LinkedIn.com as different users. Back at LinkedIn, engineers started checking the database servers.

They took the information they discovered and searched the logs to see if anyone had logged into the database servers with these IPs, username, or user agent. The database LinkedIn used at the time was Oracle, which is on a Unix machine. So they looked to see if anyone connected to it using SSH. And sure enough, they did. There were logs of the hacker logging into the database server and then accessing the database and running queries in that database. To get this far into the investigation took LinkedIn six weeks.

And I'm talking an active war room, code red situation for six weeks solid with multiple teams, dozens of people looking through thousands of servers, combing through millions of logs. it just takes a lot of time. During this time, they forced LinkedIn employees to change their passwords, they created a whole new account for the engineer who initially got hacked with that iMac computer, and they rebuilt servers to make sure there were no traces left on the system. Also, it appears that LinkedIn announced this breach as soon as they could to the public to inform their users that something bad has happened. LinkedIn users, beware.

The business social network says some of its users' passwords have been stolen and leaked onto the internet. A hacking group rocked online network LinkedIn this week by publishing almost 6.5 million user passwords to the site. 6.5 million LinkedIn user accounts were claimed to be on some hacker forums. However, not all 6.5 million passwords were visible, and the dump seemed to only be in the hands of a few people at the time, with just a sample of it posted publicly. Now, the posting and the investigation all happened in June 2012, but my research shows the hacker got into the

network back in March of that year, a whole three months earlier, which means LinkedIn had no idea of this breach until it showed up on this public forum. So the hacker did this hack back in March 2012. But in May of 2012, before LinkedIn even knew they were breached, he hacked into another website too. And I'm not exactly sure what steps he did here, but I've got high confidence of how it went. So by May of 2012, the hacker had already cracked quite a few hashes in the database that he stole from LinkedIn and was testing some of these logins by logging into LinkedIn with those usernames and it was working. So my theory is that he went through the cracked passwords looking for anyone that worked for a big IT company as an engineer or admin. Because this hacker was in the business of selling massive databases to make money.

So this was his thing. So looking through his cracked passwords, he found a quality assurance engineer who worked at Dropbox. And he tested that the password worked by logging into this Dropbox engineer's LinkedIn account. And yep, he got in no problem. Now this Dropbox engineer has stated on the record that yes, in fact, he was reusing passwords at the time. His Twitter, Facebook, LinkedIn, and Google accounts all had the same password. So this hacker actually gained access to a ton of this engineer's accounts. But did this engineer use the same password where he worked to at Dropbox? I don't know for sure, but the hacker was able to log in as the

engineer at Dropbox. And I mean, he was able to get into the corporate network with this login, either through a VPN or an admin web portal, I'm not sure. And obviously, I shouldn't need to tell you this, but it's not a good idea to reuse passwords for this exact reason. Now, this quality assurance engineer didn't have access to files that users stored on their Dropbox accounts, but he did have access to users' metadata, information about their accounts and stuff, because he would sometimes need to look into issues that users were facing. And the thing is, if a hacker has this guy's login information, then the hacker can access anything this engineer can. So the hacker got in and grabbed all the user data he could

that this engineer had access to, which consisted of usernames, email addresses, hashed and salted passwords. And then the hacker transferred this to himself and got out. A month later, the FBI was investigating this LinkedIn breach. One of the things they looked for was to see if that IP address or user agent was logging into any LinkedIn accounts as users of the site. And sure enough, there were logs indicating that the same person had logged into about 30 different LinkedIn user accounts, meaning the hacker had cracked the usernames and passwords for these LinkedIn users and was testing it to verify it worked. One of those accounts belonged to a Dropbox employee, and I don't know if it was the same quality assurance engineer, but the FBI saw that connection and thought maybe Dropbox might be the next target. So the FBI called Dropbox to tell them this information and even gave them IPs and user agents to look for.

Dropbox looked through their logs and confirmed that, yeah, those IPs did connect in as a quality assurance engineer and got into the corporate network. Once Dropbox did confirm there was unauthorized access into the network, they immediately set up a war room to handle the incident. This is basically like command central, a place where all data can be combined and up-to-date information is relayed to. Now, at the time in 2012, Dropbox had just under 150 employees working there. And just like in LinkedIn, this was the biggest thing going on at Dropbox at the time, so it was practically an all-hands-on-deck response. Dropbox had over 20 people working on this incident, but they knew they needed even more help, so they started hiring more security incident responders to just come on in and help. They first discovered that someone had unauthorized access into the corporate network of Dropbox. This seemed contained, though, as the connections didn't seem to make it into the production portion where Dropbox.com was ran out of. This was in the corporate side. Well, this is

still a big deal to have someone lurking around in your corporate network. They weren't able to see the crown jewels of like what data users were storing in their Dropbox accounts. Now, specifically, they were seeing that a Dropbox engineer was connecting into the Dropbox network from Russia. Then once they connected, they went to the internal Dropbox wiki, which has information on how to troubleshoot certain things and other technical details about Dropbox's network. The Dropbox team kept examining the logs. They saw which Dropbox engineer had his username and password stolen and went through the logs to see if there was any other suspicious activity around that. This engineer had an account at Dropbox.com, so they looked at his recent activity,

and it shows that he invited another Dropbox user to join his Dropbox team. The thing is, the Dropbox engineer did not invite that user. So this means the hacker got into the engineer's Dropbox account and then invited himself to see that engineer's files. And after they had their accounts linked up, the Dropbox engineer's account transferred some large files to the hacker's Dropbox. When Dropbox looked into what these files were, it was a list of 20 million Dropbox user details. Email, username, and salted password hashes. This made Dropbox aware that not only were they breached, but the hacker stole at least 20 million user details from their customers.

But they still weren't sure how the hacker got these, or even if it was from Dropbox at all. The next victim here is a company called FormSpring. This is a social networking site which is focused on asking questions. Think of it like a place that's dedicated to ask-me-anything type interviews. In 2012, they had 30 million registered users. And in June of 2012, the hacker got one of the admin usernames and passwords from FormSpring server and logged in using SSH. My guess is that he got this login from the LinkedIn data too, but I'm not sure on that. He also logged into one of the web admin panels using the same username. He was able to install a malicious program called madness.php on the server so that he could get back in anytime he wanted. It's the same madness.php that was found on that iMac. And he found their internal wiki and did a search there

for hashed passwords. I guess what he was looking for was information about them or where they're stored or something. Using the web admin panel, he was able to run SQL commands on the database and grabbed a large amount of user info, specifically emails, usernames, and salted password hashes. Then he logged out, and that was in June. On July 9th, someone posted a database dump of FormSpring users on some underground forum. It contained 420,000 accounts. Someone saw this and contacted a journalist. The email addresses in the dump contained the word FormSpring in them a lot, like user plus FormSpring at gmail.com, that kind of stuff. So the journalist called up FormSpring to get some answers. FormSpring had no idea they were

breached and had no idea how this database got on the forum. But this turned into an all-hands-on-deck situation for them, too. They only had a few dozen people working there at the time, but everyone who was technical got involved with this investigation. And when one journalist posted about it, soon many more journalists were calling, so the marketing team had to get involved, too, trying to handle PR. First, the ForumSpring security team needed to confirm the data.

They took the 420,000 users and compared it to their database, and it was a match. It certainly was their data. Next, they started looking for anomalous activity, and that's when they found someone had SSH'd into the server from a Russian IP. They took the IP and looked at more logs, which indicated that this user logged into the web admin portal, and from there they were able to see what the user agent was for the person who accessed it. They also saw this hacker access the wiki and placed madness.php on the web server and ran some SQL queries from the admin control panel. They discovered all this in about one day. I guess their environment was just a lot smaller

than LinkedIn to be able to get through it all quicker. Once FormSpring confirmed that they had an intrusion, they needed to contain it. They changed the username that was used to log in, deleted madness.php, put more rules in place to change passwords more frequently, and set up monitoring rules to look for logins that weren't from where the admin lives, and then totally destroyed and rebuilt certain servers that they knew the hacker had been on. On top of that, they notified all their users that a breach had occurred. They told users what data had been stolen and had them change their passwords immediately. The day after they discovered this breach, the FBI called them and said,

hey, heard you had a break-in. Can we see what happened? FormSpring sent the FBI all the logs they could to assist in the investigation. And a few weeks later, FormSpring had everything back to normal and things were working just fine again. But this story isn't over. One guy hacked into three major websites and the FBI is now on the trail. You've got to hear what happens next. Stay with us. Support for this show comes from Black Hills Information Security. This is a company that does penetration testing, incident response, and active monitoring to help keep businesses secure. I know a few people who work over there, and I can vouch they do very good work.

If you want to improve the security of your organization, give them a call. I'm sure they can help. But the founder of the company, John Strand, is a teacher, and he's made it a mission to make Black Hills Information Security world-class in security training. You can learn things like penetration testing, securing the cloud, breaching the cloud, digital forensics, and so much more. But get this, the whole thing is pay what you can. Black Hills believes that great intro security classes do not need to be expensive, and they are trying to break down barriers to get more people into the security field. And if you decide to pay over $195, you get six months access to the MetaCTF Cyber Range, which is great for practicing your skills and showing them off to potential employers.

Head on over to blackhillsinfosec.com to learn more about what services they offer and find links to their webcasts to get some world-class training. That's BlackHillsInfosec.com. BlackHillsInfosec.com. So at this point, the FBI was aware of all three of these cases. Someone had breached LinkedIn, Dropbox, and Formspring, all from the same IPs and same user agents, with a trail connecting it all. Pretty much the day LinkedIn found out that they'd been breached, they called the FBI. And the FBI began interviewing people at LinkedIn and collecting logs from them.

They saw that the hacker had connected from multiple IPs in Russia. They also saw the user agent with the word Sputnik in it. LinkedIn was sending them hard drives full of logs to examine and feeding them all the information they were finding. They even supplied an image of the engineer's iMac that got hacked to the FBI. While reviewing all this data, the FBI found something crucial in the logs. They knew what IP and user agent the hacker had. So they looked at all the people who logged into LinkedIn.com, the public website, in the last few years, and they found a person named Jamiro Cuatro. This person had registered for a LinkedIn account way before this hack and had the

same IP and user agent as the hacker. This could be gold. Like I said, users of LinkedIn often post their full resume there. So if this user had information about himself posted on his account, it could wrap up this whole story really quick. But Jamiro had a blank LinkedIn account. He wasn't associated to any company, and he didn't have a single connection or friend on LinkedIn. But to create a LinkedIn account, you need an email address. So the FBI looked to see what email address registered this account, and it was chinabig01 at gmail.com. The FBI thought this might be the email address of the hacker. Now, the FBI had quite a few IP addresses that they were considering suspicious with this,

but they narrowed down their interest to five that may be owned by the hacker, and all of them were in Russia. If this had been in the US, the FBI could issue a subpoena to an ISP and get information on who pays the bill for that connection, and get answers almost immediately. But things work differently when the FBI wants information from an ISP in Russia. There is a thing called the Mutual Legal Assistance Treaty, or MLAT. MLAT was set up to allow foreign nations to cooperate in helping criminal investigations by supplying law enforcement with internet service subscriber

info. So the FBI requested from Russia subscriber records through MLAT to see whose IPs those were. But this is not a fast process. It takes eight months to five years to get subscriber info through MLAT. So the FBI had to wait for a while on this. In the meantime, they started cross-referencing LinkedIn data with Dropbox and FormSpring data. In all three attacks, they found similar IOCs, or indicators of compromise. The same IPs, the same user agents, the same browser and OS. And once again, they looked for users on those sites from those IPs who registered for an account before this hack took place. Dropbox also had a user registered with ChinaBig01 at Gmail.com.

That person was named Jamis Gurus, a bit different from the Jamiro Cuatro from LinkedIn. And FormSpring also had a user registered as ChinaBig01 at gmail.com too. Because there were so many similar indicators on all three breaches, the FBI was starting to believe that this ChinaBig01 email address might have been owned by the hacker. So the next step is the FBI contacted Google, the owners of Gmail, and issued a search warrant to get any information on that user. See, Google is a U.S. company, so it's fairly easy for the FBI to get information from a U.S.-based company. Actually, I think they have to comply with law enforcement in this kind of way.

And Google loves collecting logs on its users, so they had plenty to share. First, the FBI saw that whoever was connecting to the server had the same IPs and user agents as the intruder who got into the other companies. Next, the FBI agent was able to see what search terms this person Googled while logged into their account. And here are some of those search terms. WordPress vulnerabilities, TrueCrypt hack, Oracle export utility, EMS data export for Oracle. The user's Google activity also showed them visiting a few sites like insiderpro.com, which was the forum that these database dumps were getting posted to. The user also visited articles which talked about the LinkedIn hack.

Then the FBI took a look in their inbox and looked at what emails they had and saw a welcome email to Vimeo, a video file sharing website. So he requested information from Vimeo, which also came back with matching user agents and IP addresses. The FBI also saw evidence that this person was logging into some LinkedIn accounts, which were employees of Automatic.com. Now, Automatic is the parent company to WordPress. And the FBI contacted Automatic

and requested to see any login activity from these Russian IPs. And sure enough, there were some login activities. Someone from Russia was logging in with different Automatic employee usernames and passwords. It's unclear what exactly the hacker stole out of Automatic's site, if anything, but it is clear he got in multiple times with different Automatic's site, if anything, but it is clear he got in multiple times with different Automatic engineers' usernames. The FBI agent then saw a

welcome email to Afraid.org. This is a website which offers dynamic DNS services. It's a U.S.-based company, so the FBI agent issued a search warrant to get data on who owned the account related to China Big O-1, and the name on this account came back as ZopaKui1. And Afraid.org also showed that whoever was accessing the site had the user agent with Sputnik in it too. The FBI did a Google search on this ZopaKui1, which is a strange and unique word, and found a user registered on an online gaming site called congregate.com. The FBI requested user details here and confirmed the user agent and IPs matched

but also discovered the user had registered a credit card on that site and had purchased some game credits. The FBI tried to trace the bank details of that card but it led them to a bank in Russia which they could not get extra information on. However, they did look to see what email address was registered with this ZopaKui1 congregate account, and the email for this user was roottaka at mail.ru. Now, since mail.ru is hosted in Russia, the FBI couldn't issue a subpoena for that either.

But the FBI googled the beginning of that email address, Ruttaka, and found a Gmail user with the same name, Ruttaka at Gmail.com. So the FBI issued a search warrant with Google to get information on that Google user, and Google responded with more information. The first thing they saw was what Google searches that user had searched for, and they were searching for things like LinkedIn hack, MySQL count fields, change Mac address, Wi-Fi Windows 7. There were also some Google map searches that this user did. They saw the user search for a dentist in Moscow and some other map searches in Russia. Next, the FBI agent was able to look at emails for this Rutaka Gmail account.

He saw this person had registered an account at VKontakte, which is like the Russian version of Facebook. The site would email you anytime someone messaged you there. And there were people messaging him, asking about hacking email accounts and different relationship type stuff that was going on. But here, everyone is referring to him as Zhenya. And everyone only spoke Russian to him. By this point, the subscriber records for that IP address came back from the MLAT request to Russia. And it showed IPs were owned by two people.

And it gave their physical address. Yevgeny Nikulin had one IP and someone else had the other. Yevgeny lived on the same street that this person was doing Google map searches on. So the FBI started looking up information about Yevgeny and found photos of what he looks like. They compared those photos with the person on VKontakte's account, and they looked like the same person. And the VK account was for a person who went by the name Zenya, which is actually a common nickname for people named Yevgeny. I'm not sure how, but the FBI investigation led them to a Russian guy named Kislitsin. Kislitsin has been known in the past to broker deals between hackers

and people buying database dumps. The FBI found Kislitsin's email address, which was a hotmail address owned by Microsoft. So they issued a search warrant with Microsoft to see what was in his inbox. There, the FBI saw emails going back and forth with the buyer of the FormSpring data. The buyer ultimately decided to buy the dump for an equivalent of $7,100. I'm not sure how much data was in there, though. Somewhere between 400,000 and 30 million user records. Supposedly, the person who bought this was half Belgian and half Turkish.

What's really strange is they used a middleman for this cash deal, and he was also involved with the E-Trade and Scott Trade hacks, which I talked about in episode 76, Knaves Out. The FBI indicted Kislitsin as a co-conspirator to this, but ultimately was unable to capture him. However, the FBI was able to arrange a meeting with him in the Russian embassy in Moscow. So they visited with this Kislitsin guy and he gave them a lot of information. Not only information about this case, but information on a few other cases the FBI was investigating too. While meeting with Kislitsin, he told them that

Yevgeny Nikulin was the person who broke into Dropbox and still had access to it and had the FormSpring data, all with the goal to sell the database dumps on the black market for money. This was enough information for the FBI to issue an indictment for Yevgeny. Multiple trails led right to him. Would they be able to catch him? Stay with us through the break to find out. surprised by just how much stolen identity data criminals have at their disposal, from credentials to cookies to PII. Knowing what's putting you and your organization at risk and what to remediate is critical for protecting you and your users from account takeover, session hijacking,

and ransomware. SpyCloud exists to disrupt cybercrime with a mission to end criminals' ability to profit from stolen data. With SpyCloud, a leader in identity threat protection, you're never in the dark about your company's exposure from third-party breaches, successful phishes, or info-stealer infections. Get your free Darknet exposure report at spycloud.com slash darknetdiaries. The website is spycloud.com slash darknetdiaries. So who is Evgeny Nikulin? Well, he's Russian from Moscow, born in 1987.

So that made him 25 years old in 2012. Evgeny loves cars so much that he started a business in Moscow, buying luxury cars and renting them out to people. And it was from that where he was often seen driving Maseratis, Lamborghinis, and Bentleys around town in Russia. I don't know where all this hacking started for him. Details of his past are foggy. I imagine he got into IT and computers like anyone else. Probably started with playing video games and then wanting to hack the video games or cheat or maybe wanting to change his grades in school or wanting to just mess around with his friends by hacking into them like it was a game. Who knows why he got started in hacking? But by 2012, he was pretty familiar with how computers worked and how to exploit them.

From reading about him, I also feel like his online life overlapped with some of these pretty notorious Russian cyber criminals. He knew some of the other big hackers. Maybe they taught him. Maybe they were hanging out in the same forums or something. And maybe Yevgeny wanted to be part of that hacking world that they were part of. Because after all, he liked expensive things and was seeing some of the Russian hackers making gravy from their digital exploits. The FBI issued an indictment for Yevgeny Nikulin, but the problem was he was in Russia, and Russia was not going to arrest Yevgeny for the FBI, and even if Russia did arrest him, he's not going to get extradited to the U.S. for trial.

So the FBI had to wait. They knew the exact address of where Yevgeny lived, but had no way to go into Russia to get him. Now, up until this point, the world had thought the LinkedIn data breach was for 6.5 million users. Because after all, that's what was posted on InsiderPro.com. And what's more is that LinkedIn never clarified how many accounts got stolen. But in May 2016, someone posted that they had even more LinkedIn credentials for sale. They claimed to have 117 million user details from LinkedIn and was selling it for just over $2,000 in Bitcoin. This triggered a whole new news cycle.

This morning, topping America's money, a security breach at LinkedIn turns out to be much bigger than first thought. That's right, the social network for Business now says a hacker stole 117 million user passwords in the 2012 breach, far more than the original estimate of about six and a half million. And think about all the users of LinkedIn. Yes, of course, professionals looking to network, but also many top executives have accounts there. I mean, after all, if your business is listed there, shouldn't the leader of that business be on there too?

But on top of that, you have government officials on there, lawmakers are there, members of Congress, FBI agents, NSA agents, senators, and yes, even the President of the United States. Barack Obama made his account in 2007 when he was running for president and was president in 2012 when this happened. This news swept through lots of circles and impacted a lot of people. What's more is this new dump contained a lot of cracked passwords that anyone can see in plain text. It wasn't that LinkedIn stored passwords in plain text, but the hackers were able to find ways to crack a lot of the passwords that were in there. Oh, and in fact, we got to see what the most common passwords that got cracked were.

I'll read you the top six most common passwords that LinkedIn users were using in 2012. The most common was simply 123456. Over 700,000 users had used this password. Because yes, LinkedIn's minimum password length was six characters at the time. The next most popular password was LinkedIn, then the password password, then 123456789, then 12345678, then 1111111. People use bad passwords. And you're telling me some of these users are using the same passwords on multiple sites? And on top of that, they're using the same password at work? Ah, it's outrageous. A few months after that, in October 2016,

the FBI got the break they were waiting for. Yevgeny Nikulin was spotted in Prague, in the Czech Republic. With the indictment all processed and things ready to go, Yevgeny Nikulin was spotted in Prague, in the Czech Republic. With the indictment all processed and things ready to go, the Czech police tracked him to a restaurant where he was eating with his girlfriend. There's body cam footage of the police arresting him. Here, have a listen. There's actually not much to listen to, so I'll describe what's going on. Yevgeny and his girlfriend are sitting at a restaurant.

From the video, I see about three police officers coming in, and they tell him to stay calm and put his hands where they can see them. Yevgeny puts his hands on the table, and they ask him to stand up and walk backwards towards the officer. Yevgeny does exactly that. Then they pat him down, take some things out of his pockets, and handcuff him. All while the other officer is making sure the girlfriend isn't getting up. It's all done very quietly, without any fuss. And Yevgeny was taken to a Prague jail.

For the next two years, Yevgeny's lawyers fought to keep him from getting extradited to the US. I wasn't able to get a second confirmation on this, but his lawyer said the FBI was trying to pin Yevgeny for hacking Hillary Clinton's emails at the time and was trying to get him to confess to that. Eventually, two years after his arrest in 2018, the Czech Republic did extradite him to the U.S. for a trial. And yeah, I went through the court records

and I never saw one reference to Hillary Clinton in there. The U.S. had nine charges on him. Computer intrusion, aggravated identity theft, conspiracy, and international transmission of information causing damage to a protected computer. And the victims of this case were listed as LinkedIn, Dropbox, and Formspring. But here's why I love this story so much. Yevgeny pleaded innocent on all these charges. He claimed he didn't do any of this. And why is that my favorite part? Because it means this case had to go to trial, which means witnesses, evidence, FBI testimony, and so much more becomes public record. To research the story, I got the pleasure of reading hundreds of pages of court transcripts. It was glorious to hear all the details from

victims and law enforcement. We rarely hear these things. Like there were three people from LinkedIn who all testified, explaining how the hacker got in and what their incident response plan was. There were three people from Dropbox giving testimony, and the CEO from Formspring explained everything he saw. On top of that, there were three FBI agents and a Secret Service agent who all gave testimony on how they were able to link all these pieces together and track them down. And it's only from all this that we know anything about this story, other than what you've seen in the headlines.

I mean, I reached out to LinkedIn multiple times and the FBI multiple times to get someone to tell me about the story, but nobody wanted to talk because I get it. What company wants to come on this show and tell me about the worst thing that's ever happened to their company? No one. So it's just really rare for us to see all the details of what happened written out so wonderfully. His trial started in early 2020, but then the pandemic hit and the trial was delayed like three months. And during that time, the Secret Service arrested a hacker suspected for breaking into the SEC, Oleksandr Irymenko. And when they arrested him, they got access to his laptop and on it was all kinds of evidence on Yevgeny. Pictures of him, videos of him,

chat messages with him, emails to him, tons more evidence. But it's weird though, because by 2020, Yevgeny had been in jail for four years. Which if you think about it, that's 13% of his life that he's been in prison without anyone deciding on whether he's guilty or not. This took a mental toll on him for sure. He barely spoke any English. He would sometimes shove guards or medical examiners or just try to walk out of the place sometimes. He made a mess in his cell by getting toilet paper wet and throwing it up on the ceiling. And he kept asking the judge for permission to have a Game Boy or a PSP to play. Yevgeny didn't testify himself. He had an interpreter in the courtroom saying everything to him.

The whole time he kept saying he had nothing to do with this. But the trial started up, and there was quite a lot of evidence connecting him to the incidents. Just to recap the trail here, IPs that access to victims' networks were registered to his name specifically. The IP used to steal data from LinkedIn led to a certain browser user agent, and that was associated to a LinkedIn account with a Gmail address. And that email was registered at Afraid.org, which had a username that was on Congregate.com, and that person bought things with a bank card. And that bank card matched other things that Yevgeny bought. On top of that, there was Kislitsin, who said that Evgeny did this, and Irymenko's

computer, which had more evidence. On July 10th, 2020, the trial concluded. The jury found him guilty on all nine counts. The judge then sentenced him to 88 months in prison, which is just over seven years. And they also ordered him to pay $1.7 million in restitution for the damage he caused to the companies he hacked. Oh, and after that, he has to do three years of supervised release, which I'm not quite sure how that works if you're not a citizen of the US. So what do we learn from this story? Well, it sounds like in 2012, these victim companies weren't doing user behavior anomaly detection like if a user vpns in from california and then an hour later that same user vpns in from russia

that should trigger an alert right yeah well it didn't the technology at the time didn't really do that kind of correlation now there's better tools for monitoring user behavior analytics and i think tools like that have a lot of potential. Next, it's crazy to me that some people use the same password for their LinkedIn account as their work accounts. Don't reuse passwords like that. Use a complex, unique password for every account you have. And the best way to do that is to use a password manager. They aren't hard to use, so go get one. I have an affiliate link to one in the show notes if you just want a good recommendation. It's also worth noting that these companies seem to have exceptional logging turned on. And when they learned about this breach, they were able to archive those logs and do system

snapshots right away to preserve any data that can be used forensically. I've seen a lot of companies just not log properly, and it just always really bothers me. Oh, and that LinkedIn engineer who was hosting those two websites on his home computer, he's moved those websites to host them on Linode now, which is one of our sponsors. I think one of the lessons he learned from this was that opening ports from the internet into your home network can be dangerous. It exposes your computer to a world full of chaos, which can ultimately result in someone getting access to your home network. And I do think about that a lot in this story. If he wasn't hosting those little websites at home, he probably wouldn't have been the way in for this hacker. And it's

also interesting to see that bad guys target employees at their house because that network is often not as strong as the corporate network. But here's the crazy part of all this. Remember that LinkedIn dump of 117 million user details that showed up in 2016? Later on that year, it just hit the public for anyone to see. So anyone can go look in the LinkedIn database to see what is in there. And there are still many people who did not change their passwords or change this as something and then just change it right back to what it was before that. And what about all those people who reused passwords on all the other sites?

Like, yeah, I changed my LinkedIn password because I was told to, but I didn't change all the other six things that use the same password. And that's where we pick up in the next episode. Someone finds a password in the LinkedIn database and has quite a story to tell about that. Hey, do you know about the Darknet Diaries shop? Listen, I love coming up with new shirt designs. Every month I throw a few more up in the shop. These shirts look great.

One is of Medusa, but she's got Ethernet cables coming out of her head instead of snakes. And there's one that looks like a bouquet of flowers, but the flowers are actually made of computer cables. Another one is of an archer who's shooting an arrow, but the arrow looks like a USB symbol. You've got to see these shorts for yourself to understand what I'm saying. So visit shop.darknetdiaries.com and buy some shirts. I'm an independent creator who loves bringing this show to you

free of charge every two weeks. But what really helps me keep on that schedule are my Patreon supporters. These are people who donate money to the show every month to help keep it going. If you want to show your support for this show, please visit patreon.com slash darknetdiaries and consider donating. Thank you. to show your support for this show, please visit patreon.com slash darknetdiaries

and consider donating. Thank you. This show is made by me, the chief biscuit dunker, Jack Recyder. Sound design by the dream alchemist, Andrew Merriweather. Editing help this episode

by the wizard of light bulb moments, Damien. And our theme music is by the phonic magician, Breakmaster Cylinder. And even though when you asky, stupid question, you get a stupid antsy. This is Darknet Diaries.