Darknet Diaries - Ep 3: DigiNotar, You are the Weakest Link, Good Bye!

Episode Date: October 1, 2017

The 2011 DigiNotar breach changed the way browsers do security. In this episode, we learn what role a CA plays, how browsers work with CAs, and what happens when a CA is breached.

A guy in Iran goes to check his email. He types in gmail.com into his browser and hits enter. A strange warning pops up. It says invalid server certificate. He's unable to get to Gmail. He connects to a VPN and tries again. Through the VPN, he connects just fine. He thinks there may be some funny business going on.

He posts a question to the Google forums, asking if there's a possible man-in-the-middle attack going on. He also says he suspects his ISP or the Iranian government to be doing something fishy. Google responded, not only to the forum post, but they published a security warning to the world and released an emergency patch to their Chrome browser. Mozilla, Microsoft, and Apple followed quickly with similar security updates. There was, in fact, a man-in-the-middle attack against Gmail users. An attack which undermined the security in all browsers. An attack that had devastating consequences. This is Darknet Diaries.

True stories from the dark side of the internet. I'm Jack Recider. callers work. They'll use anything they can find about you online to try to get at your money. And our personal information is all over the place online. Phone numbers, addresses, family members, where you work, what kind of car you drive. It's endless and it's not a fair fight. But I realized I don't need to be fighting this alone anymore. Now I use the help of Delete Me. Delete Me is a subscription service that finds and removes personal information from hundreds of data brokers' websites and continuously works to keep it off. Data brokers hate them because Delete.me makes sure your personal profile is no longer theirs to sell. I tried it and they immediately got busy scouring the internet for my name and gave me reports on what they found. And then they got busy deleting things. It was great to have someone on my team when it comes to

my privacy. Take control of your data and keep your private life private by signing up for Delete Me. Now at a special discount for Darknet Diaries listeners. Today, get 20% off your Delete Me plan when you go to joindeleteme.com slash darknetdiaries and use promo code darknet at checkout. The only way to get 20% off is to go to joindeleteme.com slash darknetdiaries and enter code darknet at checkout. That's joindeleteme.com slash darknetdiaries and enter code darknet at checkout. That's joindeleteme.com slash darknetdiaries and use code darknet. Support for this show comes from

Black Hills Information Security. This is a company that does penetration testing, incident response, and active monitoring to help keep businesses secure. I know a few people who work over there, and I can vouch they do very good work. If you want to improve the security of your organization, give them a call. I'm sure they can help. But the founder of the company, John Strand, is a teacher. And he's made it a mission to make Black Hills Information Security world-class in security training.

You can learn things like penetration testing, securing the cloud, breaching the cloud, digital forensics, and so much more. But get this, the whole thing is pay what you can. Black Hills believes that great intro security classes do not need to be expensive, and they are trying to break down barriers to get more people into the security field. And if you decide to pay over $195, you get six months access to the MetaCTF Cyber Range, Thank you. That's BlackHillsInfosec.com. that issue certificates are called certificate authorities or CA. The concept of CAs and certificates is so complicated that I'll have someone else explain it. My name is Jervis Markham and for about the last 12 or so years I've been involved with the certificate authority route program at Mozilla. A certificate authority is basically somebody who checks identity and that you trust to check identity. The level of checking depends on the type of certificate. They might

just check that the person who says they own the domain foo.com owns the domain foo.com. They might also say, and by the way, it is Foo Corporation of 116 Acacia Avenue, Birmingham, UK. And, you know, this is their phone number and this is their company reference number and stuff like that. It may check all of those things, but it checks some level of information. And the primary purpose of a CA is so that when you type Foo.com into your web browser and that request goes out across the internet which could be populated by any number of hostile or nefarious people controlling various bits of the network the data you get back a does come from foo.com and b hasn't been tampered with or can't be viewed along the way the certificate system basically underlies the secure connection ability of the web.

Web browsers such as Firefox contain an internal list of trusted CAs and hold a list of root certificates to use for verification. When you visit a website, it will present a certificate which identifies the domain of the website. The certificate also says which CA checked and verified this information is true. The browser then checks to see if the CA is trustworthy. Your browser contains a list of all trustworthy CAs and root certificates. Firefox, for instance, has 64 organizations and 159 certificates. This list of trustworthy organizations is called a root store.

When a company decides they want to become a certificate authority, they need to work with the browsers to be added to the root store. If they aren't added, browsers won't trust those websites. So there are four major root stores. There's us, there's Microsoft,

there's Apple, and there's Google. The criteria that we have for CAs to be included will include particular audits, which try to demonstrate that the CA is acting in accordance with the relevant sort of security guidelines. And auditors will come in and check that. And they will be able to present those audits to lots of different root programs. Jervis and his team are in charge of deciding which CAs are trustworthy and which aren't. This is an extremely important job. He is like the security guard for the internet,

looking out after everyone who uses Firefox and making sure the organizations that are in the trusted list are safe for the world. Think about it this way. If you use Firefox, then you are trusting that this man, Jervis Markham, knows which organizations can be trusted. It's not just me. There are three of us who currently work on the CA program.

But Mozilla runs the only fully open and transparent root program. Jervis thinks this kind of decision-making should be open to the public so they can see the decision process and even give input in to help make the decision. That's what he means by saying his root program is open and transparent. Because as you can imagine, deciding which certificate authorities are trustworthy is a difficult task. Trust is an organic thing, right? Trust is not something that results from coming to the end of a checkbox. So if we read a news article that says, for example, the government of Kazakhstan is considering manning in the middle all of its citizens, and then we receive an application from the government of Kazakhstan to include their root certificate in the browser, even if all of the paperwork is in place,

you know, we might, you know, be somewhat reluctant to add that root certificate to our browser, because we have external information about what that government may be wanting to use that certificate for. So, you know, there is, you know, the question of the reputation of the organization who is applying as well. So it is not, you know, just a simple checklist, but we do try and have criteria that are at least vaguely objective so that CAs know what they have to do in order to be included. But there's a few problems with this whole approach to using root stores and certificate authorities. Security researchers are still trying to find better solutions to this problem. One issue is, is that the certificate system had

a weakest link problem. That is to say, if you trust 64 different organizations and one of them has sucky security, then you have a problem. It doesn't matter if your particular site uses one of the other 63 because the attacker can get a certificate from the dodgy one and then impersonate you. That is, if one of the 64 organizations were to be hacked, it ruins the trust for all other CAs. Basically, the hacker would then be on the trusted list. The security of CAs has to be top-notch and impenetrable. Komodo is one of the largest CAs in the world. They issue certificates for millions of websites around the world. And in early 2011, they were hacked. The hacker issued nine fake certificates, but Komodo immediately detected this and revoked

the certificates. A few days later, Komodo had fixed the problems and publicly announced that an intrusion took place. But a few days after that, a second intrusion took place. But this time, the hacker was unsuccessful. All attempts at doing anything failed. The hacker was able to get into the network, but couldn't take any steps beyond that. They were unable to issue any certificates or do anything significant.

Then we see a strange post show up on Pastebin. Pastebin is a website where anyone can post a message anonymously. This message was written by a person named Komodo Hacker, and it reads, quote, Hello, I'm writing this to all the world, so you know more about us. At first, I want to give some points, so you'll be sure I'm the hacker. I hacked Komodo from Instant SSL. Their Komodo username password was GT admin and global trust. I'm not a group. I'm a single hacker with the experience of 1000 hackers, end quote. The message goes on to explain more on how he got

in and what he did. At the very end, he writes in Persian, I will sacrifice my soul for my leader. Five days later, Komodo announces the second intrusion, but mentions a hacker was unable to do anything, and they fixed the holes in their network. Overall, Komodo handled this issue fairly well. They quickly detected and fixed the issue and notified the public.

Komodo isn't the only CA. Another popular one is DigiNotar. It's a Dutch-based company, and they started out in 1998 doing notarizations in the Netherlands. Eventually, they became a respectable CA. In fact, the Dutch government used DigiNotar as a CA for many of their websites. And in early 2011, Vasco bought DigiNotar for almost $13 million. And DigiNotar, I think, is a case that had really sort of invested heavily into security, as you have to if you're a certificate authority. That's Josephine Wolfe. And I'm an assistant professor in the public policy and computing security departments

at Rochester Institute of Technology. She recently published an article in Slate regarding DigiNotar. The network was set up not only with all of these segments, with the public-facing, the external net, and the DMZ internal, and then the sort of several layers beyond that, but they also had physical controls in place. So once you're into the most secure portion of the network, if you then want to actually access the production servers

that are used to issue certificates, you had to get to these computers that were stored in a room that's something out of like a James Bond movie, right? There are two sets of doors. There's a hand recognition device and a PIN code, and you have to insert an electronic key card in order to actually begin the certificate issuing process. So there are several levels of security that ostensibly you have to get past in order to issue a certificate in this setup. Did you know Tara knew security was vital to the reputation of the company

and invested heavily in its own security? I like to sometimes think of securing a network similar to securing a castle that has 10,000 doors and windows. Even if you spend the time to go check every door and window to make sure it's locked, you may have missed one, or may not be aware of one, and over time you're bound to make a mistake and leave a door unlocked. Maybe because you were lazy or distracted, but humans make mistakes. In the summer of 2011, DigiNotar made such a mistake, and a hacker entered their network. The breach begins by the perpetrator actually connecting to the public-facing web servers that DigiNotar has up. And it's a little bit out of date. There are some patches that they haven't updated in the content management software. And so the perpetrator connects to their web servers, takes advantage

of some of those out-of-date vulnerabilities, and uses those vulnerabilities to tunnel through this incredibly extensive set of firewall rules into what's supposed to be sort of the most secure silo of their network. The hacker eventually made his way to the server that issues certificates. But DigiNotar had a security check in place, where physical keycard had to be present in the computer before a certificate could be issued. And it turns out that there's a keycard we think that's actually being left in permanently.

Not just out of laziness, but because DigiNotar wants to be able to automatically generate what are called certificate revocation lists, right? Every time a certificate becomes untrusted or outdated or is being revoked for some reason, DigiNotar wants to be issuing automatic lists of those certificates so that all the browsers that trust DigiNotar certificates will stop trusting those particular certificates. And in order to issue those lists, you need to have one of these cards inserted into the secure servers in this room. And so because it's just being left in there, it turns out that all of these layers of security, which seem sort of like overkill, are actually able to be bypassed.

This intruder is able to issue a bunch of rogue certificates in the names of a whole variety of different domains, right? The big one that comes up are the google.com certificates, but I believe there are also cia.gov certificates being issued and many, many others. With these certificates, the hacker can now become Google. They can trick the browser into believing they are Google.com. That is because DigiNotar was one of the trusted CAs within the browser. This breach took place on July 10, 2011, and he ended up issuing 531 rogue certificates.

Nine days later, DigiNotar detected the breach, but they didn't announce it publicly. Over a month later is when the Google forum post showed up about the man in Iran who couldn't get to Gmail.com. The people in Iran who are trying to connect to their Gmail accounts are being redirected. That directs them to the wrong website that probably looks exactly like the real Gmail. And because they've got these rogue certificates issued by DigiNotar, the people who've created this fake Gmail or Google website are able to actually sign it and look like it's really authentically a Google site. But because they've got this rogue certificate, they're able to do that. People are going to what they believe are Google sites, entering their credentials.

We suspect those credentials are then being used to spy on their Google accounts in various ways. The hacker then took these certificates and proceeded to create a man-in-the-middle attack. This is where a hacker intercepts the traffic that's supposed to go somewhere else. The rogue certificate is only half of what's needed to do a man-in-the-middle attack. The hacker needs to redirect people to his server instead of the real Google servers. We don't know exactly how he did this, but the best theory, with the most supporting facts, is he did a DNS poisoning attack. A DNS server translates a domain like google.com to an IP address so routers can

find where they need to go. He tricked the DNS servers in Iran so that anyone looking for google.com would be redirected to his IP instead. There's no definitive proof that that's how the redirect happens. There's sort of circumstantial evidence that you can use to try and make that case, right? So it's possible that there's an ISP in Iran that's actually either complicit or has been compromised and is therefore redirecting traffic to these fraudulent sites. Another possibility is that it's a very high level DNS server that has been compromised and is sort of propagating those fake records down to the other DNS servers that rely on it in the hierarchy. And again, there's a sense that the investigators have just based on the evidence that it's

probably not that because when they look at how many people are being redirected and when they're being redirected, it's very bursty. That is, you see sort of a big spike in the number of people being sent to the fake sites and then it goes down and there's a spike again, which makes them think that it's probably not poisoning happening at a high level. It's probably sort of some local DNS servers being flooded with messages that look like they come from high level trusted DNS servers, but instead are actually coming from the attackers saying, here's the correct updated DNS record for google.com. And that will

only last a certain period of time because then that DNS server will get the correct record from the higher level DNS server. So then it will start sending people to the right side again, and the attackers will have to come back and poison it again. Over 300,000 people from Iran visited the rogue server. This attack seemed to be targeting Iranian civilians. This attack would have went undetected for some time, but Google had a clever way of detecting it. It's finally noticed because they're actually doing this within the Chrome browser, which is manufactured by Google.

And because Google owns the Chrome browser, which is checking these certificates, and owns the websites that they're being used to imitate, the browser actually notices. This is a certificate that comes from a trusted certificate authority, right? Chrome trusts certificates issued by DigiNotar, but we know it's not the right certificate because we know what our Google certificates are, and this is not one of them. This is why the guy in the Google forums had a server certificate error. Jervis over at Firefox was on the front line. Google notified us that they had detected a misissued certificate for star.google.com,

which was being used in active attacks on users in Iran. And so we started investigating this and I basically took on incident response. So I was very busy. In the case of DigiNatar, their network was thoroughly penetrated and had been for months. Their logs were a mess. Their infrastructure was a mess. There was no way of telling the scope of the compromise and therefore no way of containing it to a particular

route or a group of routes or an intermediate or group of intermediates. Because of the, you know, catastrophic failures of security, it was impossible to continue any form of trust in the DigiNotar systems and organizations. When Mozilla decided DigiNotar was no longer trustworthy, they removed them from the root store. But users would have to update their browser in order to receive the version of Firefox that didn't trust DigiNotar. All the other root stores also removed DigiNotar from the trusted list. Almost two months after the breach took place,

and well after Iran had been target of a massive man-in-the-middle attack, DigiNotar finally publicly admitted they were breached. Once this actually becomes a public compromise, then the Dutch government kind of steps in and takes control of DigiNotar, which is sort of unprecedented in the history of breaches of private companies. And once that happens, the company is sort of, to a large extent, out of the picture. Their leadership is no longer making the decisions about hiring investigators and everything else. The Dutch government was using DigiNotar as their primary CA for numerous government sites and applications. And when browsers began removing DigiNotar from the trusted root store,

it broke a lot of systems for the Dutch government. So they reached out to the root stores and asked to reinstate DigiNotar back into the root store. The browsers did add DigiNotar back as a trusted CA, but to solve the problem of the rogue certificates, the root stores would block any certificates that were issued by DigiNotar after July 2011. This allowed Dutch government to continue and work towards finding a new CA. Eventually, the Dutch government moved to a new CA, and within three months after the breach,

DigiNotar was shut down permanently. This episode is sponsored by Vanta. Trust isn't just earned, it's demanded. Whether you're a startup founder navigating your first audit or a seasoned security professional scaling your GRC program, proving your commitment to security has never been more critical or more complex. And that's where Vanta comes in. Businesses use Vanta to establish trust by automating compliance needs

across over 35 frameworks like SOC 2 and ISO 27001, centralized security workflows, complete questionnaires up to five times faster, and proactively manage vendor risk. Vanta helps you start or scale your security program by connecting you with auditors and experts to conduct your audit and set up your security program quickly. Plus, with automation and AI throughout the platform, Vanta gives you time back so you can focus on building your company. Join over 9,000 global companies like Atlassian, Quora, and Factory who use Vanta to manage risk and prove security in real time. For a limited time, listeners get $1,000 off Vanta at vanta.com slash darknet. That's spelled V-A-N-T-A, vanta.com slash Darknet for $1,000 off.

DigiNotar hired a security firm called Fox IT to conduct an investigation as to what happened. They found numerous problems in the DigiNotar network. They found the Windows Domain Administrator account had a simple password and was easy to brute force. Then all the certificate-issuing servers were on the single domain. This means a single admin account was able to access all eight of their certificate servers. Numerous systems didn't have antivirus present, which would have stopped some

of these attacks. There was no central logging and no separation of critical systems. A combination of all these failures is how the hacker was able to bypass all of the security checks. Fox IT also looked through the evidence to try to find who did this attack. We don't know who did this and nobody's been caught or prosecuted for this breach. It's true, we haven't been able to determine who did this. But when Fox IT investigated the breach, they did find some interesting clues. First, they looked at all the IPs the hacker used in the network, and they were able to trace each IP back to a proxy, except for one. This IP connected to the DigiNotar network from Iran,

and it was not from a proxy. It only connected for a few seconds, and then disconnected, and a new connection showed up from a proxy. This may have been a few seconds and then disconnected and a new connection showed up from a proxy. This may have been a mistake by the hacker who then corrected himself very quickly. That IP was also seen visiting DigiNotar previously which may have been for reconnaissance. Fox IT also found the hacker left a message on the server that was hacked. It read in part, quote, there is not any hardware or software in this world exists which could stop my heavy attacks, my brain or my skills or my will or my expertise. With a message at the end in Persian saying, I will sacrifice my soul for my leader.

We also see a new Pacebin message show up from the person who claimed to have hacked the Komodo CA. This Komodo hacker now takes credit for also hacking into DigiNotar and also signs his paceman with the same message in Persian. He also goes on to say he's 21 years old and works alone. Certainly DigiNotar and the Dutch government gets gets very caught up in this because they're sort of the vehicle by which all of this happens. But the real target was espionage directed at Iranian citizens. But who would want to read the emails of Iranian citizens? The U.S. has had conflicts with Iran, so it could be suspect. And Bruce Schneier, a prominent security expert,

says it may be the work of NSA or an exploit of NSA. But he says this mainly because of a leaked NSA document showing the NSA had access to Didi Notar. This theory isn't very strong and has almost no other evidence. So who else would be targeting the general people of Iran? The Iranian government itself. To understand why, we need to dial back two years before the hack. In 2009, there was a presidential election in Iran. Mahmoud Ahmadinejad won the election by 63% vote. But there was a strong opposition to this. Many Iranians believed the votes had been tampered and the election was rigged. Protests began immediately. This created

a divide among the people of Iran. Some people became extremely distrustful of the government, while other people became extremely loyal. Police began arresting protesters, and when protesters didn't leave, they were pepper sprayed, hit with batons, and sometimes shot at. Within three months of the election, 72 protesters died. Corruption was so bad, the police forced families to sign papers saying their dead relatives died of a heart attack and not by police brutality. As you can imagine, this only incited even more emotion among the people of Iran. For years after, the Iranian government worked hard to eliminate any government opposition. This continued all the way to when this Dijonotar attack took place.

So it's a strong theory that this hack was done by the Iranian government itself, or someone trying to help the Iranian government. They were possibly looking through emails trying to find dissidents, and those who were unhappy with the Iranian president. And if they were found, it may have resulted in people being arrested, tortured, or killed. Certificate authorities and browser developers have learned some serious lessons from DigiNotar. Audits have become more strict for CAs to pass in order for them to be accepted into root stores. Public key pinning has seen more use. This is what Google did with their Chrome browser to detect this breach. They forced

Chrome to only accept certificates issued from Google's CA, essentially pinning the certificate to a specific CA. More websites have done this since DigiNotar, but it has its shortcomings. For instance, imagine the problems a website would have if they pinned their certificate to a CA that went out of business. Or imagine if a hacker were to pin a certificate to a rogue CA. Unpinning the certificate is currently a complicated task. Since DigiNotar, Firefox has added a new feature to help block rogue certificates. Here's Jervis to tell us about it.

So we have a system called OneCRL, which is, if you like, an emergency revocation system. And Firefoxes all check for certificates on this kind of blacklist every 24 hours. And so if we need to do an emergency revocation of either an individual certificate or, in fact, an entire sort of tree of certificates based off one intermediate, then we can put it into one CRL. And within 24 hours, every Firefox which has the system, and we've had it for quite a while now, will no longer trust those certificates. So it's not required to install an update

in order for us to distrust something. When a hack takes place at this worldwide scale, it changes the way we do security. In a way, hackers are like the immune system of the internet. They infect us, we get sick, we get better, and we become even stronger afterwards. And even today, six years later, when a major breach happens to a company, someone always reminds us of the fate of DigiNotar. Thank you to Josephine Wolfe

for telling us about Did You Know Tar? And a great big, over-the-top thank you to Jervis Markham for coming on the podcast. Because about a year after this episode originally aired, Jervis passed away. At 22, he was diagnosed with a malignant salivary gland cancer. And after battling it for 18 years, he passed away at the age of 40. Jervis made significant contributions to securing Firefox and the Bugzilla tool, and we have him to thank for keeping us safe in this unsafe world.

We're going to miss you, Jervis. Music is provided by Ian Alex Mack and Kevin McLeod.