CyberWire Daily - Middleboxes may be meddling with TLS connections. [Research Saturday]

Episode Date: June 22, 2019

Researchers at Cloudflare have been examining HTTPS interception, a technique that weakens security, and have developed tools to help detect it.  Nick Sullivan is head of cryptography at Cloudflare, and he joins to us share their findings. The research can be found here: https://blog.cloudflare.com/monsters-in-the-middleboxes/ Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. of you, I was concerned about my data being sold by data brokers. So I decided to try Delete.me. I have to say, Delete.me is a game changer. Within days of signing up, they started removing my personal information from hundreds of data brokers. I finally have peace of mind knowing my data privacy is protected. Delete.me's team does all the work for you with detailed reports so you know exactly what's been done. Take control of your data and keep your private life Thank you. JoinDeleteMe.com slash N2K and use promo code N2K at checkout. The only way to get 20% off is to go to JoinDeleteMe.com slash N2K and enter code N2K at checkout. That's JoinDeleteMe.com slash N2K, code N2K. Hello, everyone, and welcome to the CyberWire's Research Saturday.

I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities and solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. And now, a message from our sponsor, Zscaler, the leader in cloud security. Enterprises have spent billions of dollars on firewalls and VPNs, yet breaches continue to rise by an 18% year-over-year increase in ransomware attacks and a $75 million record payout in 2024. These traditional security tools expand your attack surface

with public-facing IPs that are exploited by bad actors more easily than ever with AI tools. It's time to rethink your security. Zscaler Zero Trust plus AI stops attackers by hiding your attack surface, making apps and IPs invisible, eliminating lateral movement, connecting users only to specific apps, not the entire network, continuously verifying every request based on identity and context, simplifying security Thank you. your organization with Zscaler Zero Trust and AI. Learn more at zscaler.com slash security. Several years ago, I worked with some researchers from the University of Michigan and a bunch of other institutions to help study what's going on on the internet. That's Nick Sullivan. He's the head of cryptography at Cloudflare.

The research we're discussing today is titled Monsters in the Middle Boxes, introducing two new tools for detecting HTTPS interception. Cloudflare is in a great position to be able to see a large percentage of what traffic is flowing over the internet and help study what clients are actually making these requests and what traffic on the internet actually looks like. And during that period, we were developing new technologies for encrypting the internet. So Cloudflare was one of the first proponents of a protocol called TLS 1.3, which is the latest web encryption standard.

Or anytime you see that nice little lock icon in your browser in HTTPS, TLS is the protocol that encrypts it. And so when we were developing TLS 1.3, we're really changing what the protocol looks like on the wire. So we were bumping the version number, we were changing a couple fields around in there. And when actually deploying this protocol, there were some unexpected failures and unexpected issues. And it turned out that these were caused by the way that

the architecture of the internet is built for some users. So it turned out that some users in corporate environments or in managed infrastructures, the encryption between the browser and the website was not actually fully end-to-end. In fact, this is something that we knew, but how widespread it was wasn't really well known to us. So it turns out that there's several companies and organizations who provide services to allow corporation or whatever the managed service it is that's providing internet or for people in a corporate environment or a high school or university or things like this to be able to inspect encrypted traffic and make editorial decisions or block malware. There's many different reasons that you'd want to look

inside of encrypted connections. And so it turned out that these middle boxes, which these were often actually pieces of hardware that were shipped to corporations to help inspect this traffic, were causing some issues with TLS 1.3. They were not necessarily compatible with some of the new changes that were happening. And so in this study, we tried to figure out what it was that was going on and in what way is traffic between what a user sees with the nice lock icon and typing in the actual HTTPS URL on their site and how that traffic transmits from the browser to the server and back, whether or not this is end-to-end encrypted. And if not, what is actually going on? This is the kernel of the research project that we did. And this was published in NDSS, which is a computer security journal. And this kind of led us down the trail of seeing how we could actually take this kind of one-time study that was done with these researchers

and turn it into a continuous dashboard that people could see what's happening on the internet is evolving. Let's walk through together some of the basics here for folks who may not be up on it. Can you walk us through how TLS works? If I'm not in a corporate environment, it's just me and my computer and I'm trying to connect to someone else and we have that secure connection. What's going on there? If you're typing in, say, www.securesite.com into your browser, there's several steps. The first is DNS, which is the domain name system lookup. This is what your computer uses to translate a name to an IP

address. So your computer does the request to a DNS server, which then tells you an IP address, whether IPv4 or IPv6. And then your machine has the address of the server that's connecting to, and then it begins what's called the TLS handshake. And the TLS handshake is a back and forth that the browser and the web server do in order to establish shared cryptographic keys that can be used to encrypt and authenticate any subsequent request and response that comes between your browser and the website. And part of this handshake is used for the server to prove ownership over that hostname. So server provides a certificate and the certificate was issued by a trusted certificate authority.

And it says, you know, my secure site.com. And this certificate is associated with the key that is used to sign the handshake. So the way the handshake goes is the client sends a first message called the client hello, which indicates which cryptographic algorithms it supports. And the server will respond with a server hello that says, great, of the algorithms that you've advertised support for, here's the ones that I choose. And also here's my certificate. And here's the proof that I actually control the key associated with the certificate. And then from that point onwards, the communication between the client and the server is encrypted and authenticated. So anybody seeing this traffic along the internet really only sees encrypted packets. So they can't

really tell what you're requesting, what the content of the website is, or anything you're sending from that point on. I see. And then so in these corporate environments that you describe, where someone has the opportunity to inject themselves into this stream, what's happening there? In environments in which your machine is managed as a user, you may have a phone that has a mobile device management profile. This is a system in which the corporation installs some additional software to help onboard you onto the system.

What happens is, rather than the server using its certificate to complete this connection, when the client connects outward from the network, it actually goes to this piece of hardware called a TLS intercepting middle box. And this TLS intercepting middle box has the ability to basically print out any certificate it likes or any website that it likes at all. So the certificate itself is not by a publicly trusted certificate authority, but it's by a specific certificate authority that's specific to that corporation only. So your device, you get enrolled into the system.

And typically, there's a set of trusted global certificate authorities that your browser trusts and you only trust websites that have certificates from those certificate authorities. When you're enrolled into one of these systems, they install a corporate certificate authority. And this corporate certificate authority is something that the middle box is able to issue certificates from. And so when you think that you're connecting to the actual website itself, in fact, you're getting a certificate that is created by this middle box. And so your connection is end-to-end encrypted between your browser and this middle box.

And then the middle box decrypts the traffic and then creates its own TLS connection from the middle box out to the website. And so effectively, you are getting an encrypted connection to the middle box, and then the middle box has an encrypted connection to the website itself. And the main reasons that this goes on, the things that folks want to look at are what? Well, oftentimes, there are restrictions with regard to what type of content that people want a lot of people to view at work. There's also restrictions with respect to what type of data can be transmitted outside of the enterprise. So DLP or data loss protection is a requirement in several compliance regimes

to make sure that the folks that are inside the company aren't sharing private corporate information outside of their corporate bounds. And so these middle boxes can sometimes detect this and record it and you can catch people leaking private information. Oftentimes it's also used to see if you have... If there's malware, if there's some sort of virus or something on the user's computer that is attempting to reach out to some kind of command

and control server or some orchestrator that lives outside the corporate balance. So you're looking for people leaking data, you're looking to restrict the types of websites and content that people can view at work, and you're looking to try to prevent malware or detect malware inside a corporate network. And so your research was looking at different types of interception of HTTPS traffic. What were you looking at here? We knew that these connections for a large percentage of the Internet were not end-to-end. And so we knew there were corporate middle boxes. We knew that there are services like antiviruses that effectively do the same thing,

where the antivirus software is installed on your machine and acts as a software middle box on your machine to look for different malware signatures. And we didn't know exactly how widespread these were or which specific technologies were used and which technologies were used to what extent. And so this is really what we were looking for, is what is going on for the percentage of the internet that is not actually the browser connecting to the server directly. From the server's perspective, we can tell that it's not actually the browser. And in TLS, there's certain fingerprintable characteristics that are

innate to the type of TLS client software you're using. So I mentioned in the client hello, you list the cryptographic parameters that you support. Well, some of these middle boxes don't necessarily support the same ciphers as a browser. So you can tell whether or not the request to the server is actually being sent from the middle box or from the antivirus provider or directly from the browser because it's just a unique fingerprint for each of these pieces of software. And so we were trying to figure out what these are. And from that, whether or not these connections are secure, the browsers invest a lot of time trying to use the best possible encryption algorithms and use different techniques

like certificate transparency and various ways to make sure that these connections are actually safe and secure. And browsers invest a lot of time and have big security teams to do so. And we were trying to see if these middle losses or these antivirus services, whatever these men in the middle, as you might call it, these middleware pieces of software, whether or not they were secure, essentially gave a letter grade to whether or not they're following the best practices with respect to what you would expect a browser to do to have a secure

TLS connection. And what did you discover? How did they rate? Well, some of them were quite good, but some of them failed quite miserably. In fact, some antivirus services that we discovered did things like not even validating the certificate at all. So if you imagine you're going to a website and your browser connects to the local antivirus middle box, and then that antivirus middle box would

connect out to the website, some attacker in between could potentially hijack that connection because the antivirus itself was not checking that certificate. Other things we found were using weak and outdated ciphers. Some of them were using ciphers like RC4, which have had known vulnerabilities over the last five years or so, but they were still using these old technologies. So the letter grades rated from... Some of them had a really good rating of an A that we sort of marked as equivalent to how a browser would do their TLS encryption to some that were a straight F, which actually destroyed all semblance of security that you would want to have from an intent crypto connection.

So the lock icon itself is supposed to indicate trust. And in some cases, these middle boxes, they had bugs and they were violating this trust. Would there be any way that the user would know that they had a problem? In fact, not really, which is one of the disturbing pieces. In corporate mode, if you will, where your browser is connecting to a site and it's trusting a non-publicly trusted CA, some corporate CA or some antivirus CA, there's no real indication to the client that you're using one of these other CAs or that the certificate provided by the website is not a real web publicly trusted certificate. And furthermore, there's no indication to the user

what type of TLS the second hop of the connection is using. So users really had no way to know. And so what are the ramifications of that? There's several ramifications, first of which is user security for a lot of these situations is decreased. And this is actually a surprising situation because oftentimes these middle losses and antivirus programs are implemented in order to increase user security, protect them from malware, protect them from other things that are malicious on the network. But in the end, it turns out to be weakening their encryption. So this is something that we did report in the original paper to a lot of these providers, the ones that we could identify. And they managed to,

in many cases, update their software, do things more safely. At the very least, it exposed this as a potential problem. And it exposed the fact that if you are going to be doing this type of inspection for compliance reasons or for whatever reason that you'd want to be doing it, that you really should take very, very good care of how you do it, because building a secure TLS implementations is challenging to do. And browser companies invest a lot of money and time and expertise in this. And sometimes these middle box vendors don't necessarily have the resources to do so. So it became something that was kind of a call to action

to the industry to improve their best practices. And were you able to observe any instances where these were actually being exploited out in the wild? It's not something that we particularly looked for in this study. We were more looking for the signatures of the connections and trying to figure out what actually was happening on the internet. This is more of a measurement exploration study itself. We didn't necessarily look for specific instances of exploitation, but you can't necessarily rule it out.

So based on what you've learned here, what are your recommendations? How do people best protect themselves? Well, if you're a corporation, ask your vendor how strong the TLS connection is with respect to connecting from the middle box up to the server. So we created a service called MITM Engine, which is something that can, on the server side, you can detect which of your clients are actually using one of these inspection services. And you can kind of provide recommendations. So as somebody who's operating a server, you can inform users that potentially whatever the corporation is using for middle boxes is not doing the right thing. This is something that our open source library is able to do to provide to people. And overall, it's also indicating that there's room for

philosophy change with respect to how corporations deal with internal security. So the numbers that we see, we have this dashboard. It's called malcolm.cloudflare.com, where we take the daily statistics and aggregate them and look at the fingerprints and say, is this really coming from a browser? Do we really think that this is coming from the browser that is sending the request? And our rates of interception that we're seeing are somewhere around 30%. Around a third of the internet users are potentially vulnerable to someone doing a bad job on one of these middle boxes. And so it becomes very important for these providers to do a really good job and put them through thorough testing.

What I was mentioning in terms of philosophy, it does bring you to this opinion that or this possibility that doing something that is pro-security, in this case, having an inspection looking for malware, trying to protect users, is actually putting them at risk. And it might be doing less of a service than what you really think it is. Is there any sense that this is kind of a set it and forget it kind of thing where people set up these systems in their organizations, as long as it keeps running and it's working, it's not something that they go back and check in on to make sure that it's doing what it should be doing?

Yeah, exactly. Browsers have been at the forefront of computer security with respect to auto-update. And so whenever a browser has a vulnerability, they're on a pretty frequent schedule of auto-updating. There's something of a six-week cadence for something like Chrome. So they're constantly scouting the world, looking for vulnerabilities, deciding to change which ciphers they're using, implementing new things like TLS 1.3, which is a much more secure protocol, and updating. This is not the norm for all security products. I would wish it would be, and I think most of us in the security community think auto-update is a great idea. But browsers are really at the forefront. And some of these other technologies are, as you said, kind of set it and forget it. And often they don't get patched automatically. Or if they do get patched, it's something that's manual and costs downtime. more frequently updated is likely going to result in better security than having it be

something that's some appliance in the backroom of the data center that you don't necessarily update. Some of these middle boxes, they do attempt to replicate the signature of the browser underneath it. And those can be detected as well. So we built several different techniques here to detect this. And it's based on just the variety of things you can do in TLS. There's different extensions you can support. There's different ciphers you can support. And then there's different quirks that involve which order different extensions are or which order different ciphers are in. And so if you have the time to dig into the Malcolm dashboard,

you can take a look at what these different appliances and pieces of software do. It's interesting to explore which types of devices are more likely man-in-the-middle than others. And this can give anybody who's generally interested some insight into what types of corporate environments are more likely to do interception, which ones are not. Our thanks to Nick Sullivan from Cloudflare for joining us. The research is titled Monsters in the Middle Boxes, introducing two new tools for detecting HTTPS interception.

We'll have a link in the show notes. Thank you. been breached. Protect your executives and their families 24-7, 365 with Black Cloak. Learn more at blackcloak.io. The Cyber Wire Research Saturday is proudly produced in Maryland out of the startup studios of Data Tribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing Cyber Wire team is Elliot Peltzman, Puru Prakash, Stefan Vaziri, Kelsey Bond, Tim Nodar, Joe Kerrigan, Carol Terrio, Ben Yellen, Nick Valecki, Gina Johnson, Bennett Moe, Chris Russell, John Petrick, Jennifer Iben, Rick Howard, Peter Kilpie, and I'm Dave Bittner. Thanks for listening. Thank you.