CyberWire Daily - Vulnerabilities in the public cloud. [Research Saturday]

Episode Date: September 25, 2021

Guest Ariel Zelivansky, Senior Manager of Security Research at Palo Alto Networks, joins Dave to discuss Unit 42's work on the first cross-account container takeover in the public cloud. The Unit 42 Threat Intelligence team has identified the first known vulnerability that could enable one user of a public cloud service to break out of their environment and execute code on environments belonging to other users in the same public cloud service. This unprecedented cross-account takeover affected Microsoft's Azure Container-as-a-Service (CaaS) platform. Researchers named the finding Azurescape because the attack started from a container escape – a technique that enables privilege escalation out of container environments. The research can be found here: What You Need to Know About Azurescape Finding Azurescape – Cross-Account Container Takeover in Azure Container Instances Note: Microsoft is a sponsor of the CyberWire, however, we cover them as we would any other company. Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. of you, I was concerned about my data being sold by data brokers. So I decided to try Delete.me. I have to say, Delete.me is a game changer. Within days of signing up, they started removing my personal information from hundreds of data brokers. I finally have peace of mind knowing my data privacy is protected. Delete.me's team does all the work for you with detailed reports so you know exactly what's been done. Take control of your data and keep your private life Thank you. Hello everyone and welcome to the CyberWire's Research Saturday. I'm Dave Bittner and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace.

Thanks for joining us. We've been focusing for a long time on researching the public cloud. We've also decided to look into the actual public cloud that is offered by vendors. We decided to look into the actual public cloud that is offered by Vandors. And Azure ACI was one of the main platforms that we were interested in. That's Ariel Zelovansky. He's Senior Manager of Security Research at Palo Alto Networks. The research we're discussing today is titled,

What You Need to Know About AzureScape. And now, a message from our sponsor, Zscaler, the leader in cloud security. Enterprises have spent billions of dollars on firewalls and VPNs, yet breaches continue to rise by an 18% year-over-year increase in ransomware attacks and a $75 million record payout in 2024. These traditional security tools expand your attack surface with public-facing IPs that are exploited by bad actors more easily than ever with AI tools. It's time to rethink your security.

Zscaler Zero Trust Plus AI stops attackers by hiding your attack surface, making apps and IPs invisible, eliminating lateral movement, connecting users only to specific apps, not the entire network, continuously verifying every request based on identity and context, simplifying security management with AI-powered automation.

And detecting threats using AI to analyze over 500 billion daily transactions. Hackers can't attack what they can't see. Protect your organization with Zscaler Zero Trust and AI. Learn more at zscaler.com slash security. It allows users to run containers without having to set up Kubernetes and all the containers configuration that they usually have to do to set up containers. That is making ACI really attractive to use by companies and organizations that want to make a transition to the cloud. And for us as the research team, realize that is a good point to start looking for vulnerabilities.

So can you give us a little bit of the background here, just for folks who may not be completely up to speed on this stuff, some of the benefits of having these container instances and the general security that they provide? Not only containers, but also the public cloud in general is more more secure than running on-prem in most cases. The fact that there is a vendor that is taking care of security and making sure everything is up-to-date, patching any security issues, and in general monitoring what is going on on their instances,

that adds a lot to security. At the same time, we as a research team invest in finding flaws and other issues that may have been mislooked. So we try to find something that would allow us to do a cross-account attack. The way these containers are running on the public cloud is that they're multi-tenant. So one container is actually running on the same Kubernetes instance as another user's organization's cluster.

For us, this means if we could actually break out of a container and somehow access another user's data and instances and potentially execute code, this would be a really impactful vulnerability. And that's what Azerscape is about. We've been able to break out of a container and access containers of other organizations. For us, this was the holy grail of finding a vulnerability in the public cloud. And we're really happy to have found this vulnerability so we can talk about this as a threat, potential threat. You know, containers by themselves can be broken out of. And we need to understand how to keep the instances secure, even if one is able to break out of a container.

to keep the instances secure, even if one is able to break out of container. Well, let's walk through it together, because it really is a fascinating story, the way that you all went about this. You spin up your own instances here within Azure, and where did you begin? Where did you start looking? Right. So Yuval, one of the researchers in the team, was able to identify that there was an old vulnerability in RunC, which is the container runtime that runs the containers in Azure. And this was actually a vulnerability that we've looked at in the past, and we already had an exploit available for that vulnerability. So all we had to do is just execute that and we were actually out of the container.

It was a really surprising moment for us to find that we could just exploit that. Yeah, this was because it was an older version running on Azure's servers? Correct. So Azure was running an old Run-C instance. In fact, this was not a vulnerability just by itself. The model of ACI, the architecture that they build ACI upon,

is actually supposed to tolerate breakouts. Not only this vulnerability could have existed, but kernel vulnerabilities and kernel zero days could have been around. And Azure is actually supposed to expect attackers to break out in some cases where they do have zero-day vulnerabilities in the kernel. By itself, although it was surprising, it was not enough to contact Microsoft and tell them, hey, there is a potential RCE vulnerability on your instances. RCE vulnerability on your instances. But we did contact Microsoft at this point just to get the conversation started

while we were actually trying to see what else we can do. And really, the research begins at this point. When we have code execution on the host, that is, the Kubernetes node that is running the container, we want to see what else is around. How can we actually navigate and do lateral movement to other containers? And the way it is actually contained,

the container is contained inside ACI, is that it's pulled per node. That means the container itself has its own VM, its own node. So there are no other containers by other users on that same cluster, on the same node. So we'll actually have to find a way to get privileges on Kubernetes itself to access these other nodes.

And, well, I think that was the most interesting part of the research for Yuval to be able to find a JWT token that actually included credentials that allowed him to escalate privileges on that Kubernetes cluster. From that point, we actually had Cluster Admin, which is the administrator equivalent for Kubernetes. And Yuval could see containers by other users, see other nodes, list all the organizations that are running on that cluster, and potentially execute code

on them. So not only we could steal data, leak the credentials for other organizations, but we could actually execute code on their containers. That was really scary to see, and we started the process of writing an advisory and disclosing this to Microsoft at this point. Well, I mean, walk us through the discovery of that token. That's an interesting story in itself. Correct. So, there is another component that is called the bridge. It is something

specific to ACI. Kubernetes doesn't have this component. So you will actually try to find what is going on in the network level so you could understand how commands are getting executed on the container. So the JWT token is actually coming when the researcher, the attacker, is doing an external command to their container. Rather than doing that through the kubelet, the main Kubernetes brain,

than doing that through the kubelet, the main Kubernetes brain, there is this bridge component that is sending this request to the container. And in that request, we just were able to find that token. So once you get that token, then what do you do with that? What's the process by which you exploit the fact that you have that to do the things that you want to do? The token is actually already high-privileged, and it allows executing code inside containers.

From there, what we've done is execute code on the API server, which is what is actually the component that is responsible in Kubernetes for making decisions regarding authentication. So from there, the way to cluster admin was really quick. Yeah, that's fascinating. And as you say, I mean, a bit, I don't know, surprising, scary perhaps is a word because it seems like keeping these nodes separate from each other

is one of the basic value propositions of working in this sort of environment. Right. So multi-tenant platforms in general, not only Azure, have this as their main threat. And I think it's been recognized already, although we're the first to find and disclose such a vulnerability, it is something that's been discussed and considered for a long time. This is something that is the most difficult to defend from because really you have these zero-day vulnerabilities

and even existing vulnerabilities that could be exploited that allow attackers to navigate within your platform. After we were able to break out of the container and run code on the node, we were not expected to be exploring that area. And this is where the main responsibility of the cloud provider itself and not the user is to secure that area.

Because the user really has no visibility of that place. The user can do anything that they can to protect and detect activity on their containers. But once it's on the node, really this is the domain of the cloud provider. So there has to be a lot of mechanisms in place. And really, most of the attacks that we've tried and the ways we try to escalate privileges

before we found Azure Scape were actually mitigated by Microsoft. So although we were able to find this specific escape and privilege escalation, we actually tried a lot of techniques and things before that that were not possible because Microsoft has invested in mitigating them. So I think this is a community effort. We as researchers, companies working in this cloud security space,

and cloud providers themselves have to all invest in securing multi-tenant platforms. I think it is possible. And the fact that we've been able to find that one vulnerability doesn't make the cloud less safe. As in legacy platforms, vulnerabilities were really discovered and fixed and patched so users can protect themselves. In this case, ACI, Azure, can actually apply the fix directly

to all their users without having to ask users to take KB or patches and apply them. So we found this vulnerability, disclosed it to Microsoft, they took the fixes, applied them directly to the platform, and now everybody is safe. This is a process I believe will continue to be ongoing for all cloud providers, not only ACI. And I'm happy to be part of this community researching these platforms.

Well, there's a little more to the story as well. As you point out in the research that you published here, after you reported the token issue to Microsoft and they released a patch for it, you all continued to do a little more digging and you found another way in. Right. This was another way to escalate privileges. This was an SSRF vulnerability. As I mentioned, we were speaking to Microsoft about fixing these issues and we continue to look around and try to find any way we can do things we're not supposed to, like stealing credentials of other users, executing code on their containers, and potentially

just spinning our own containers and other clusters. This was just another way to escalate privileges in essence. Can you give us some insights on the types of tools that you all are using to be able to analyze these container environments? What sorts of things are you using to be able to see what's going on behind the scenes there? Great question. So I did want to mention, we have developed a tool that is called WhoSee or WhoContains. And it allows us to extract the container runtime from the cloud provider to actually see how our containers are being run.

The way containers are now run in these platforms, where you don't have the Kubernetes cluster and you don't control anything, it's all on the vendor's side. So before using that tool, we had no visibility on how containers are actually getting executed and what is the container runtime.

And Woosie, it's able to copy the binary that is executing the container in its bootstrapping process and send it back to us. And that's how we initially found this old run C instance and understood we could exploit the potential old vulnerability to escape out of the container. We encourage researchers, and we're also doing that, to run HUSI on all cloud platforms to understand and explore how cloud platforms are being built.

It's not open source, so we need to do some research and exploring to understand what is the architecture behind this platform. And they're also changing all the time. So there is really a lot of benefit for us to use tools like HUSI to understand what's going on behind the scenes. I'm curious, you know, while you were all in the midst of this research, was there any detection on Microsoft's part that you all were bouncing around from place to place,

you know, places that you shouldn't have been? Right. So we've been really careful not to run anything that won't be successful. And we didn't get actively detected as we were researching the platform. But as I mentioned, some techniques and things we tried just didn't work.

There were other cases where we researched cloud platforms where we've actually been detected actively and were approached by engineers from the vendors to understand what we're trying to achieve and how we've escalated privileges. But in this case of Azure Escape, it just worked. As soon as we tried the container escape we were on the node executing commands

and we were not detected at that point. So an attacker could have potentially exploited this to just do some crypto mining and earn some quick cash from executing code on the platform directly. Right. So as you mentioned, I mean, Microsoft has been very responsive here, and they've patched the issues that you've presented to them.

Given all of that, what are your recommendations here? Are there any take-homes for folks who are using these environments to better protect themselves? So I've mentioned earlier that there is a lot on the vendor side to detect attacks and mitigate vulnerabilities in their architecture, but there is a lot that can be done by the users, by the organization themselves on their containers. I personally work on Prisma Cloud, so I can talk about that as a potential security mechanism to detect attacks. But there are really a lot of solutions

that allow you to detect attackers as they're running code, you know, after the post-exploitation, after they're actually able to get into your container and try to do anything malicious. For example, crypto mining. It's just crypto mining is the most common attack for cloud and container attacks.

This can be detected. And even if you don't stop the attacker on their way in, there are many ways where you can catch them at runtime level, at network level, after something malicious is getting executed. Our thanks to Ariel Zelovansky from Palo Alto Networks for joining us. The research is titled What You Need to Know About AzureScape. We'll have a link in the show notes. in the show notes. Cyber threats are evolving every second and staying ahead is more than just a challenge. It's a necessity. That's why we're thrilled to partner with ThreatLocker, a cybersecurity solution trusted by businesses worldwide. ThreatLocker is a full suite of

solutions designed to give you total control, stopping unauthorized applications, securing Thank you. The Cyber Wire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing Cyber Wire team is Prakash, Justin Sabey, Tim Nodar, Joe Kerrigan, Carol Terrio, Ben Yellen, Nick Valecki, Gina Johnson, Bennett Moe, Chris Russell, John Petrick, Jennifer Iben, Rick Howard, Peter Kilpie, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.