CyberWire Daily - It's still possible to find ways to break out. [Research Saturday]

Episode Date: October 10, 2020

Containers offer speed, performance, and portability, but do they actually contain? While they try their best, the shared kernel is a disturbing attack surface: a mere kernel vulnerability may allow containerized processes to escape and compromise the host. This issue prompted a new wave of sandboxing tools that use either unikernels, lightweight VMs or userspace-kernels to separate the host OS from the container's OS. One of these solutions is Kata Containers, a container runtime that spawns each container inside a lightweight VM, and can function as the underlying runtime in Docker and Kubernetes. Kata's virtualized containers provide two layers of isolation: even if an attacker breaks out of the container, he is still confined to the microVM. Joining us in this week's Research Saturday to discuss the research is Yuval Avrahami from Palo Alto Networks Unit 42. The research presented at Black Hat USA 2020 can be found here:  Escaping Virtualized Containers Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. data products platform comes in. With Domo, you can channel AI and data into innovative uses that deliver measurable impact. Secure AI agents connect, prepare, and automate your data workflows, helping you gain insights, receive alerts, and act with ease through guided apps tailored to your role. Data is hard. Domo is easy. Learn more at ai.domo.com. That's ai.domo.com. Hello, everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

We are talking about the container sandboxes, which is actually a technology for sandboxing and isolating containers. That's Yuval Avrahamy. He's a principal security researcher at Palo Alto Networks with Unit 42. The research we're discussing today is titled Escaping Virtualized Containers. It's part of a presentation that he gave at Black Hat 2020. Cat 2020. And now, a message from our sponsor, Zscaler, the leader in cloud security. Enterprises have spent billions of dollars on firewalls and VPNs, yet breaches continue to rise by an 18% year-over-year increase in ransomware attacks and a $75 million record payout in 2024, these traditional security tools expand your attack surface with public-facing IPs that are exploited by bad actors more easily than ever with AI tools.

It's time to rethink your security. Zscaler Zero Trust Plus AI stops attackers by hiding your attack surface, making apps and IPs invisible, eliminating lateral movement, connecting users only to specific apps, not the entire network, continuously verifying every request based on identity and context, simplifying security management with AI-powered automation, and detecting threats using AI to analyze over 500 billion daily transactions. Hackers can't attack what they can't see. Protect your organization with Zscaler Zero Trust and AI. Learn more at zscaler.com slash security. security. So, you know, even though containers have a tremendous upside in portability and in performance,

ironically, they're not that good at containing. The attack surface they exposed really isn't suitable for hostile workloads which are actively trying to escape and actively trying to break out and compromise the underlying host. So a couple of companies came up with an idea

or a couple of tools to sandbox containers and apply additional layer of isolation. And the research I presented at Black Hat is really how can you, even though there is an additional layer of isolation, how can you break out and still compromise the underlying host? So what is the additional layer of containment that you're working with here? So the sandboxing solution I looked at was Kata containers. Other solutions you might have heard of is Gvisor by Google,

for example. Kata containers was created by Intel. And how it sandboxed the container is it runs each container or Kubernetes pod inside the lightweight virtual machine, and then you have two layers of isolation. So if a malicious attacker tries to break out of the container, of the sandbox, you will have to first break out of the container and then escape the encapsulating virtual machine. So it's really a difficult challenge. And the idea here is, of course, if you run a lightweight virtual machine for every container, there is going to be a performance penalty. But the idea is you don't have to do that for every container, there is going to be a performance penalty, right? But the idea is you don't have to do that for every container in your cluster.

A really good use case for deploying that kind of solution is looking at the weakest link in your cluster of containers. So, for example, a Kubernetes cluster and saying, okay, that container is exposed to the network, is doing a lot of parsing of input coming from the network. He might be vulnerable. It might be compromised. And, you know, once an attacker compromises a container,

he might try to crypto mine. He might try to steal secrets. But one attack vector which is very concerning, it's a compromised container, a rogue container, might try to escape and affect the rest of your cluster, and that's obviously very dangerous. So when you're looking at your cluster and you say, okay, this is the weakest link container,

I can choose to only sandbox that container using a container sandbox like other containers and then apply an additional layer of isolation. So the reasoning behind container sandbox like Kata Containers, and then apply an additional layer of isolation. So the reasoning behind container sandboxes is really, it's really good and it's really an appropriate solution. But what really motivated me to get into this research is I started hearing folks mention container sandboxes

as a solution for what I felt like too many problems, right? They would say if you have this misconfiguration, just use a container sandbox instead of fixing the root cause of the issue. For me, what I

felt was missing in the container security field is really a research that shows even you know, even though container sandboxes are great for isolating your containers, you should be aware that it's still possible. It's much harder, but it's still possible to find ways to break out. And you should really be prepared for that kind of event, right? The main takeaway I want people to take from this research is that a container sandbox really isn't a substitute for a secure cloud-native architecture, right?

It's one layer, but you really shouldn't get, it's not a license to neglect the rest of your security posture, if that makes sense. Yeah, it does. And so you set out to escape from this sandboxed container here. Let's go through it together. I mean, what was your approach?

The approach was to take it one step at a time. We have two layers of isolation to break out. First the container and then the virtual machine. So I started off with the container and actually had more confidence that I could find an issue in the container because Kata containers, they created like a small container runtime for running the container inside the virtual machine.

And I was fairly confident that it may have some problems not because a knock of the Kata containers maintainer, it's just a really difficult job to create a flawless container runtime from scratch. And you can look at the five years of past vulnerabilities in Docker as an example. It's really a difficult job. And when I started to look into that, I started to investigate which isolation properties were applying to the containers, and I found one that was missing. And I used that to escape to the virtual machine.

So that really was the first part. And so where did you go from there? So the next stage was to break out of the virtual machine. And here there was a lot of attack surface to look at. The virtual machine monitor itself, the communication between the virtual machine and the host, a lot of stuff to look at. But where I eventually found issues there were in the share directory between the virtual machine and the host. So, Kata containers, the sandbox, they use a shared directory to transfer some files to the virtual machine. But because it is shared by nature, the virtual machine can control

the files there just as much as the host, right? So, I find a way for the virtual machine to plant malicious symbolic links in the share directory and then when the host try to place some files in the share directory or do some operations with files in the share directory, it will actually

inadvertently follow those sim links and do operation on the host instead of the VM. And that way, I was able to find a couple of attacks that you can launch on the host. And one of them was a denial of service attack where you could unmount the entire file system of the host,

which of course that really is bad if you're running multiple containers on one host and then one container is able to take down the file system of the entire host. That host really becomes unusable at that point. And the second attack, I was actually able to redirect the container image, the container file system, to every path on the host that I wanted to. So, for example, you can imagine that if I redirect the container file system over something like a slash bin or slash lib, it's really easy to gain code execution on the host.

Because now every binary that the host tries to execute from slash bin, I have control over that. So that was really the entire attack put down together. And how sophisticated were your efforts? I mean, how difficult would it be for an attacker to take the same sort of path that you did? It took me a couple of months. So you really have to have an agenda in place, a target that you're using, the sandbox that you really want to compromise. It's not that easy.

It's a chain of multiple vulnerabilities. So that's really never easy. But it could have had a lot of impact if it was found by a malicious actor. It's because some cloud service providers are actually using Kata containers to separate between different customers. So you can imagine that if you were running a containers, if you're someone like Azure, for example,

and you need to run containers for multiple customers on the same host, you might want something like a container sandbox to separate, you know, to really segregate between those containers so that one malicious customer can impact the rest of the containers of the other customers. So some cloud service providers have chosen Kata containers as their container sandbox of choice. And you could actually have a scenario if a malicious customer was aware of those vulnerabilities or if he hunted for those,

he could actually break out of his container and try to affect the rest of the platform, which could have like, you know, hundreds and thousands of containers from different customers. So that's really where I think that the vulnerabilities I found could really have a large impact. So what are your recommendations based on what you learned here in your research? What do you say to folks who are using these containers and these sandboxes? So my first recommendation is even if you chose to deploy a container sandbox, because of the whole of the security guaranteed with it, a lot of people are just deploying using the default setting,

which are often not the most secure one. So first of all, if you went the way to deploy a container sandbox, put in a little bit of more extra effort to really configure it using the most secure configuration. And in the example of Kata containers, those configurations would have stopped my attack. So there is real evidence that those configurations really work. For Kata containers, for example,

using Firecracker as the underlying virtual machine really improves their security and it hinders, for example, it mitigates the vulnerabilities that I found. A firecracker is an open-source project from Amazon which is really taking off. Another thing you can do is really be prepared for the scenario of a workload, a container, breaking out to the underlying hosts

and trying to affect the rest of the cluster. In Kubernetes, for example, there is a lot of ways where you can restrict the privileges of a single node, so that even if a container breaks out and compromises a Kubernetes node, that node wouldn't do too much damage to the rest of the cluster. The best recommendation I could give is using the node restriction

admission controller. You could look it up in Kubernetes documentation, and it really lowers the privileges of the nodes in your cluster so that even if one is compromised, it couldn't really do too much damage to the rest of the cluster. And of course, if you really have the time, put in some network policies and network firewall between Kubernetes nodes so that only meaningful and approved and logical traffic can go between nodes really also helps to deny a couple of attacks.

So in general, I mean, is it safe to say that putting your containers within these sandboxes is a good idea? Is it generally, is the upside, the security upside better than the, I guess, the potential performance hits that you could sustain? That's a great question. And I think it really depends on what you're running in your containers. If you are running your own code that you wrote and you are 100% certain

of what's running in the container, I think there are other best practices that you can follow that can improve the security of your containers without taking the performance penalty. But if you are running code which you didn't wrote, or if you are a cloud service provider

that you are running code of customers which you have no idea what they're running, then I think Container Sandbox really is required because the technology of containers simply isn't enough to secure hostile ones, which are actively trying to escape. Is this a situation where people who are working with cloud service providers, should they be asking them how they're doing this sort of sandboxing?

Is this a question that they should be asking to see what the level of security is? Absolutely. I do think you need to ask that. You know, there is a lot of trust when it comes to cloud service providers. And that's really a meaningful portion of the isolation that they provide to your workloads.

I think you probably would need to consult some security professional to evaluate the solution that they use. But generally, I think it's something that people should be more aware of. Definitely, when I'm doing penetration testing of cloud service providers, this is one of the things I look into. So I'm sure that malicious attackers are also looking into that. That's Yuval Avrahamy from Palo Alto Networks. The research is escaping virtualized containers. We'll have a link in the show notes.

have a link in the show notes. Cyber threats are evolving every second and staying ahead is more than just a challenge. It's a necessity. That's why we're thrilled to partner with ThreatLocker, a cybersecurity solution trusted by businesses worldwide. ThreatLocker is a full suite of solutions designed to give you total control, stopping unauthorized applications, securing sensitive data, and ensuring your organization runs smoothly and securely. Visit ThreatLocker.com today to see how a default deny approach can keep your company safe and compliant. approach can keep your company safe and compliant. CyberWire team is Elliot Peltzman, Puru Prakash, Stefan Vaziri, Kelsey Bond, Tim Nodar, Joe Kerrigan, Carol Terrio, Ben Yellen, Nick Valecki, Gina Johnson,

Bennett Moe, Chris Russell, John Petrick, Jennifer Iben, Rick Howard, Peter Kilpie, and I'm Dave Bittner. Thanks for listening.