CyberWire Daily - Keeping an eye on RDS vulnerabilities. [Research Saturday]

Episode Date: September 24, 2022

Gafnit Amiga, Director of Security Research from Lightspin joins Dave to discuss her team's research "AWS RDS Vulnerability Leads to AWS Internal Service Credentials." The research describes how the vulnerability was caught and right after it was reported the AWS Security team applied an initial patch limited only to the recent Amazon Relational Database Service (RDS) and Aurora PostgreSQL engines, excluding older versions. They followed by personally reaching out to the customers affected by the vulnerability and helped them through the update process. The research states "Lightspin's Research Team obtained credentials to an internal AWS service by exploiting a local file read vulnerability on the RDS EC2 instance using the log_fdw extension." The research can be found here: AWS RDS Vulnerability Leads to AWS Internal Service Credentials Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. of you, I was concerned about my data being sold by data brokers. So I decided to try Delete.me. I have to say, Delete.me is a game changer. Within days of signing up, they started removing my personal information from hundreds of data brokers. I finally have peace of mind knowing my data privacy is protected. Delete.me's team does all the work for you with detailed reports so you know exactly what's been done. Take control of your data and keep your private life Thank you. JoinDeleteMe.com slash N2K and use promo code N2K at checkout. The only way to get 20% off is to go to JoinDeleteMe.com slash N2K and enter code N2K at checkout. That's JoinDeleteMe.com slash N2K, code N2K. Hello, everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts

tracking down the threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. It actually was for fun, the exploitation part, but the main purpose of the research was to deeply understand how things are working to better detect it in the platform. That's Gafneet Amiga, Director of Security Research at Lightspin. The research we're discussing today is titled AWS RDS Vulnerability Leads to AWS Internal Service Credentials. And now, a message from our sponsor, Zscaler, the leader in cloud security.

Enterprises have spent billions of dollars on firewalls and VPNs, yet breaches continue to rise by an 18% year-over-year increase in ransomware attacks and a $75 million record payout in 2024. These traditional security tools expand your attack surface with public-facing IPs that are exploited by bad actors

more easily than ever with AI tools. It's time to rethink your security. Zscaler Zero Trust plus AI stops attackers by hiding your attack surface, making apps and IPs invisible, eliminating lateral movement, connecting users only to specific apps,

not the entire network, continuously verifying every request based on identity and context, simplifying security management with AI-powered automation, and detecting threats using AI to analyze over 500 billion daily transactions. Hackers can't attack what they can't see. Protect your organization with Zscaler Zero Trust and AI. Learn more at zscaler.com slash security.

Well, for folks who may not be familiar with Amazon Elastic Kubernetes Service and all of the stuff that you're working with here, can you give us a little bit of the background, a little description of how all this stuff works? Yes. So Kubernetes is the containers orchestration platform that was created inside Google. EKS is a managed Kubernetes service. It's really complicated and hard to maintain a Kubernetes cluster, even just to create it. It can be difficult sometimes. So what Amazon are doing is to give us like a black box, all the control plan area. They create the cluster for us. They maintain the control plan. And you can have access to the data plan and you can just start and connect your client to the cluster and start deploying deployments on Kubernetes.

It should be easier to work with that and maintain it. And EKS is that managed service. Well, that all sounds great. Walk us through your research here and how you discovered some vulnerabilities. In EKS, the research question was first, like how EKS performs the authentication because they are able to identify you as an AWS IAM identity. In AWS, you have users, you have roles,

you have groups, and in EKS, you can stay logged in to the terminal as an AWS identity, let's say an IAM user or role, and you can perform actions against your managed Kubernetes cluster in EKS. So the question was how it's done and to deeply understand it. And in Noga's research, you can see exactly how they're doing that. She specified that there is a config map

inside the cube system namespace. And this config map, the AWS auth config map, there is a mapping between AWS identities and Kubernetes identities. And the code that is running in the control plan by AWS is the code of the AWS authenticator for Kubernetes. This is an open source code that was developed first by Heptio and then moved to AWS EKS themselves.

And they're using this code and deploy this service inside the managed Kubernetes control plan. You cannot see it. You cannot access it, but it's there or like some kind of a version of it. version of it and the logic there is is written in that um in that code and by like researching the code and doing a code review you can see what is the exact logic of how things are being translated between the aws identity and the kubernetes identity And when I did the review and the debug, I locally installed it just to be able to play with it. I saw that there are multiple parameters

in the request that they are looking in order to decide what is going to be the Kubernetes identity they are going to map to. They want to figure out eventually what is going to be the Kubernetes identity they're going to map to. They want to figure out eventually what is going to be the Kubernetes username and Kubernetes group. And in order to do that, they're using AWS service and endpoint,

which is STS get caller identity. And AWS STS get caller identity returns back the logged in user identity. So they use this endpoint in order to get back the user identity. And then they take this identity and map it using the config map to a Kubernetes username and group. So in the way they are using the STS GetCallerIdentity endpoint is that they're requesting the user that accesses the cluster to send a signed request to STS GetCallerIdentity endpoint. And the signed request is signed using the AWS credentials, the access key and the secret of the IM user in AWS.

When you go over the code, there is a section when they take the signed request parts. So there are multiple parameters that are being passed as part of this request. But one of the parameters is, for example, what is the access key ID of the user that signed the request? So there are multiple parameters being parsed in the request, and they are taking these parameters and using a to lower function in order to save it. So there is a kind of a trick that you can pass the same parameter name twice and then you sign the request with both parameters but one is like with uppercase

and one is with lowercase parameter name. And what happens is that the EKS cluster will take one of the two because it depends on the ordering, but it will take one of the two. So we want the cluster to take the malicious one, the one that we're going to manipulate the value of the access key ID.

And the STS in AWS will take only the one that he is familiar with. It will just ignore the STS service itself, will ignore the manipulated parameter as long as it's signed as part of the request. So this is exactly what I did. long as it's signed as part of the request. So this is exactly what I did. I passed twice the same parameter name in the request and signed it. And then what happened is that EKS took the malicious access key ID and translated it into like a different user. So you can trick it to think you're another identity in the cluster.

This can lead to a privilege escalation inside the cluster. It depends on how the mapping in the AWS auth config map is configured. But yeah, this is the technical issue. So it's led to a privilege escalation inside the cluster. And has the EKS team addressed this? So they, what they did is to check that you are not passing the same parameter name twice. They allow you to pass it only once and then it prevents the attack. I see. So, So is this fixed now? Has this been patched?

Yes, it was patched and it was fixed. Actually, it was patched very quickly. And now all the EKS managed clusters are fixed. But if you're using the open source of the IAM Authenticator or that you're using EKS Anywhere, which is the on-premise version of EKS, then you should take an action to update this service by yourself. What do you make of this vulnerability in terms of how this code was written and the functionality that they were intending to have happen and the resulting bug?

I mean, do you have thoughts on how this sort of thing comes to pass? Well, I think that the service is written really good. You can see in the issues, you can see that they're in the pull request. You can see that they're taking into account all the security aspect. They have blocked many other leads that I thought that I will be able to exploit in the code there. They heavily thought about security while they created this extension. It's tricky. It's always tricky to enforce the connection between two services

and in this case is Kubernetes authentication and identity and AWS identity. It's always tricky, but I get the need to ease the usage of the end user

at the end, but I think that here also when when you look at the way they treat it, so they took it very fast from the moment you report to the moment it's deployed in production, the fix. So I think it's a good balance here. You know, you mentioned when we were first speaking that part of your reason for digging into this was fun. Was it fun? I mean, it sounds to me like this is the sort of thing that you and your colleagues really have a good time with. Yeah, yeah.

It's always interesting because you start just by researching to understand how it works. But then things start to come up to your mind like and what if and if I will change that into that how it will react and you always save it to a later time and when you have the time you try to play with it so it is for me it's fun. Our thanks to Gafnit Amiga from Lightspin for joining us. The research is titled AWS RDS Vulnerability Leads to AWS Internal Service Credentials. We'll have a link in the show notes. And now, a message from Black Cloak.

Did you know the easiest way for cyber criminals to bypass your company's defenses is by targeting your executives and their families at home? Black Cloak's award-winning digital executive protection platform secures their personal devices, home networks, and connected lives. Because when executives are compromised at home, your company is at risk. In fact, over one-third of new members discover they've already been breached. Protect your executives and their families 24-7, 365 with Black Cloak. Learn more at blackcloak.io. amazing Cyber Wire team is Rachel Gelfand, Liz Ervin, Elliot Peltzman, Trey Hester, Brandon Karp,

Eliana White, Puru Prakash, Justin Sabey, Tim Nodar, Joe Kerrigan, Carol Terrio, Ben Yellen, Nick Vilecki, Gina Johnson, Bennett Moe, Chris Russell, John Petrick, Jennifer Iben, Rick Howard, Peter Kilpie, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.