CyberWire Daily - Breaking barriers, one byte at a time. [Research Saturday]

Episode Date: March 29, 2025

This week, we are joined by Jon Williams, Vulnerability Researcher from Bishop Fox, discussing "Tearing Down (Sonic)Walls: Decrypting SonicOSX Firmware." Bishop Fox researchers reverse-engineered the encryption protecting SonicWall SonicOSX firmware, enabling them to access its underlying file system for security research. They presented their process and findings at DistrictCon Year 0 and released a tool called Sonicrack to extract keys from VMware virtual machine bundles, facilitating the decryption of VMware NSv firmware images. This research builds upon previous work, including techniques to decrypt static NSv images and reverse-engineer other encryption formats used by SonicWall. The research can be found here: Tearing Down (Sonic)Walls: Decrypting SonicOSX Firmware Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the CyberWire Network powered by N2K. Are you frustrated with cyber risk scores backed by mysterious data, zero context and cloudy reasoning? Typical cyber ratings are ineffective and the true risk story is begging to be told. It's time to cut the BS. BlackKite believes in seeing the full picture with more than a score, one where companies have complete clarity in their third-party cyber risk using reliable quantitative data.

Make better decisions. Reduce your uncertainty. Trust BlackKite. Hello everyone and welcome to the CyberWires Research Saturday. I'm Dave Bittner and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Part of what we do on our team is we regularly research end-day vulnerabilities in major software vendors. A lot of our customers use firewall appliances like the SonicWall devices, and so we stay on top of new vulnerabilities that are announced in those types of devices. That's John Williams, vulnerability researcher at Bishop Fox. The research we're discussing today is titled, Tearing Down Sonic Walls, Decrypting Sonic OS X Firmware.

And with Sonic Wall in particular, we had done some research on them in the past. We kept up with a lot of what was going on with Sonic Wall. But when they upgraded their platform to Sonic OS X, suddenly we didn't have access to the underlying file system anymore. They rolled out a new version of encryption to protect those images.

And so we took it upon ourselves to do the research to try and reverse engineer that encryption so that we could continue being able to research new vulnerabilities when they are announced on that platform. Well, let's go through that together. I mean, you mentioned that they enabled encryption here on their firmware. Why would they do such a thing?

And how does it work? Sure. Well, it's pretty common for vendors of appliances like these to use some type of encryption on their firmware images. The reasons why, it can be hard to speculate, but I'm sure that the general thought is it just slows down people who are trying to gain access to your files. It slows down people who are trying to gain access to your files. You know, it slows down reverse engineers. And that much is true, but as we've seen, it rarely actually stops them.

And SonicWall had used encryption on their earlier version of Sonic OS, but other researchers had succeeded in reversing that, and there were decryption tools available, which that's what we had used until they released Sonico OS X, at which point we had to do our own research to figure out how to gain access again. Well, let's go through it together. What were some of the challenges that you all faced as you made your way through decrypting this firmware?

Sure. Well, it started off as pretty much as you would expect if you're familiar with, you know, the way encryption often works on these. And what that means is we started with a virtual machine image, right? Once they released a VM, it makes it much easier to actually access the static assets. And when you unpack that virtual machine, you basically end up with several volumes that get mounted into the virtual machine, you basically end up with several volumes that get mounted into the virtual machine at boot time.

In our case, for Sonic, while there were only two that mattered, one of them contained the firmware itself as just an encrypted package, and the other one contained a bootloader. In a scenario like that, we would expect that, well, because it's a virtual machine,

the keys to decrypt the firmware must be somewhere. If all we have is the encrypted package and bootloader, then the keys are most likely going to be somewhere within that bootloader. And that initial assumption did turn out to be correct. However, it took a lot more, it was a much more complex process finding them and retrieving them than we first expected. In fact, I like to joke that it's one of the most CTF-like challenges

that I've ever actually seen in real life. That's interesting. Are there details you can share with us about that process? Yeah. We lay it all out on our blog, so you can go through it on there if there's anything that's not clear from this conversation. But looking at it from a high level, the idea is that we had to unpack the bootloader.

We had to extract an initial RAM disk from that. Within the init RAM volume, we found an encrypted installer package. But fortunately, because it was a virtual machine, the key for that installer package was also within the init RAM volume. So we decrypted the installer volume, and in there we found a pair of key encrypting keys,

along with a lot of bash scripts that actually handled the process of decrypting the firmware and installing the initial file system when you first set up the firewall. So once we had recovered those through the process of reverse engineering, we figured out how the decryption worked. And we were able to use the key encrypting keys to extract key material from the firmware image, decrypt the actual keys that were used for the encryption, and then use them to decrypt the package that was contained within the firmware bundle itself. So it was multiple layers of encryption all kind of piled on top of each other.

Yeah. But once we got through all the bash scripts that explained it and had a good clear sense of how it worked, we were able to automate the process of doing all of that in reverse. So as someone who admittedly has only a passing knowledge of how encryption works, when you're looking at something like this, is it fairly easy to establish, I guess I should ask it a different way, how hard is it to establish exactly what kind of encryption

you're dealing with? Sure, well, it depends a lot on what your target is and how you are determining the type of encryption used. In this case, because it was mostly bash scripts, once we had gotten to the point where we had access to those scripts, the rest of it was fairly easy to understand. It was a long process

because there were so many scripts working together. But in the end, they were using OpenSSL with AES keys, and those are pretty common encryption techniques. So once we found those commands, then it was easy to understand what was going on. It was just a matter of digging them out of some really deeply nested functions. And so how would you rate the sophistication of what Sonic Wall was doing here? That's an interesting question.

I'm not sure sophistication is necessarily the right word. Okay. Because it was really just, it was more complex than it was sophisticated, right? Persistence. Yeah, if you just pile more layers on top of it, does that make it more sophisticated or just more annoying to reverse engineer?

You know? Yeah, that's interesting. So you get in and you're successful at decrypting this. What happens next for you all? Well, so we released a tool called Sonicrack, which automates the process of extracting the keys from the virtual machine image, and then decrypting the actual firmware, so you can access the root file system.

We wrote this tool to simplify the process so that in the future when we do want to research new vulnerabilities, especially if there's a new end day that is dropped, we can very quickly pull out the binaries that matter from the root file system and be able to, for example, run them through a patch diff process and understand what code has changed between the two latest versions, which makes it a lot easier to find and write an exploit for a new vulnerability that's come out. We'll be right back.

Looking for a career where innovation meets impact? Vanguard's technology team is shaping the future of financial services by solving complex challenges with cutting-edge solutions. Whether you're passionate about AI, cybersecurity, or cloud computing, Vanguard offers a dynamic and collaborative environment where your ideas drive change. With career growth opportunities and a focus on work-life balance, you'll have the flexibility to thrive both professionally and personally. Explore open cybersecurity and technology roles today at vanguardjobs.com.

Cyber threats are evolving every second and staying ahead is more than just a challenge, it's a necessity. That's why we're thrilled to partner with ThreatLocker, a cybersecurity solution trusted by businesses worldwide. ThreatLocker, a cybersecurity solution trusted by businesses worldwide. ThreatLocker is a full suite of solutions designed to give you total control, stopping unauthorized applications, securing sensitive data, and ensuring your organization runs smoothly and securely.

Visit ThreatLocker.com today to see how a default deny approach can keep your company safe and securely. Visit threatlocker.com today to see how a default deny approach can keep your company safe and compliant. And what sort of vulnerabilities or security concerns did you all uncover here? Well, the last major one that came out, Sonicwell had announced an off-bypass affecting their SSL VPN. And because we had put in the work to do this decryption, we were very quickly able to run a patch diff report

when they released a patch for this issue, find the vulnerability, and write an exploit for it. And as a result, that allows us to first, you know, warn our customers of our Cosmos platform that where they're vulnerable, so we know exactly which of their servers are exposed and can be affected by this vulnerability. But we also are able to give this proof of concept exploit that we've written to our operators so they can demonstrate the impact, you, hey, not only do we think you're vulnerable based on what we see, but we know you are because we ran this script and we can prove it.

And this just gives the security teams for our customers a lot more evidence to help them get things patched quickly. So for us, it's all about trying to make sure that we are finding the vulnerabilities and exploiting them so that we can demonstrate impact and get things fixed before malicious attackers do the same thing. Well, help me understand some of the conversations that go on behind the scenes with you and your colleagues at Bishop Fox, because there's several layers of your process here as well. It's one thing to internally decrypt this for your own explorations in use.

It's another thing to publish a tool so that anybody can do that. Are there any discussions in house about, you know, how, hey, Ganger, you know, are we sure we want to do this and what's the justification? Oh, absolutely. Yeah. I mean, we take this very seriously. You know, we do have responsible disclosure policies that we follow. And when something like this comes up, we will talk about it and make sure that we're making the right choice. And in this case with Sonic

Wall, we felt like it was justified to release the tool. And the reason for that is that SonicWall is a regular target of attacks. It does regularly publish advisories about new vulnerabilities that have come out. And what we will often see is that as soon as the vulnerabilities are announced, they've already been exploited in the wild.

And so it's often malicious attackers that are driving this ahead. And so when you have a barrier like encryption that gets in the way of doing good faith security research, we find that it can be helpful to break down that barrier and release the tool so that small independent researchers have the same access to the root file system as do well-funded nation-state actors, which are typically the ones responsible for this type of malicious behavior, right? From our point of view, releasing a tool like this helps to level the playing field and facilitate more good faith research alongside of, you know, some of the

attackers who already have the capability to do this. I see. Have you been in communication with anyone at Sonic Wall about this? Not back and forth. We did give them a heads up to let them know that we were going to be publishing this. We didn't get a response from them on it. I see. Well, how do you suppose your research here informs folks who are out there using SonicWall's products. Well, I think it can help them, if they're technically savvy enough, it can help them to gain a better understanding of how their tools work and what SonicWall is doing to protect them. By unpacking the root file system of a firmware package, it's not like you're getting access to source code. You know, it doesn't go that far. Most of what you'll find in the file system are compiled binaries. So you still need to be technically savvy to be able to reverse engineer them and understand what's happening.

So it's really only assisting people who are already well versed in security research to do this. But by having more researchers looking at this code and reporting bugs and vulnerabilities directly to SonicWall, it gives them the opportunity to fix things before attackers get their hands on them. And so from my point of view, this is something that should inspire more confidence on the part of people who use SonicWall products. Do you understand SonicWall's motivation for encrypting the firmware? Yes and no.

I mean, I think that from a certain point of view, it makes sense to slow down research efforts. But on the other hand, I think that when you're the developer of a software, you often fail to take into account that those barriers typically only give malicious actors an advantage, you know, and that, but that is what we see from our side of things. I see. Looking at sort of broader trends when it comes to firmware security, how does this research tie into that? I mean, are these the kinds of things that you're seeing that are typical for organizations to try to secure their

firmware? Yeah, it's pretty typical. I mean, not all companies will go to the same length of piling on, you know, multiple layers of encryption. But it is pretty standard. You know, when we look at other vendors in the firewall space, they will usually have some kind of basic protection around their firmware images. Not all of them, but we would usually expect it. Is firmware an area of security that you feel people have, that it gets the attention it deserves?

I think it does, yeah. Particularly for these larger vendors, the ones that really dominate the market with hardware appliances that are used to protect corporate networks. From where I sit, I see a pretty even spread of attention across them, both from good faith researchers and from malicious attackers. But we regularly hear from other vendors in this space. So it's definitely not just SonicWall that we're picking on here. Sure. Well, based on the research here, what are your recommendations for the folks out

there who are using this kind of hardware? Any words of wisdom for them? Well, the main advice for consumers of this technology, it's the same as always. You know, if you have a firewall device deployed on the edge of your technology, it's the same as always. If you have a firewall device deployed on the edge of your network, make sure the management interface is not exposed because anytime there's a vulnerability affecting that side of the device or those components, having it exposed to the public internet just leaves you vulnerable and there's no need for it to be. Even the vendor advises, make sure your management interfaces are only facing your internal networks.

We still see way too many devices misconfigured that way on the internet. But when it comes to other vulnerabilities that also affect like the SSL VPN component, that gets a bit more tricky because that interface is designed to be public facing. And so really then, the trick there is just making sure that you are staying aware of the news, that you're on top of security advisories when they come out, and you patch your devices as quickly as you can. Our thanks to John Williams from Bishop Fox for joining us. The research is titled, Tearing Down Sonic Walls, Decrypting Sonic OS X Firmware. We'll have a link in the show notes.

And that's Research Saturday brought to you by N2K Cyberwire. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity. If you like our show, please share a rating and review in your favorite podcast app. Please also fill out the survey in the show notes or send an email to cyberwire at n2k.com. This episode is produced by Liz Stokes. We're mixed by Elliot Peltzman and Trey Hester.

Our executive producer is Jennifer Iben. Peter Kylpi is our publisher and I'm Dave Bittner. Thanks for listening. We'll see you back here next time. you