Risky Business - Wide World of Cyber: How state adversaries attack security vendors

Episode Date: May 9, 2025

In this edition of the Wide World of Cyber podcast Patrick Gray talks to SentinelOne’s Steve Stone and Alex Stamos about how foreign adversaries are targeting security vendors, including them. From North Korean IT workers to Chinese supply chain attacks, SentinelOne and its competitors are constantly fending off sophisticated hacking campaigns. This edition of the Wide World of Cyber was recorded in front of a live audience in San Francisco, with Patrick attending via Zoom. The Wide World of Cyber podcast series is a wholly sponsored co-production between SentinelOne and Risky Business Media. This episode is also available on Youtube. Show notes

Hey everyone and welcome to another edition of Wide World of Cyber, the podcast we do here at Risky Biz, which is produced in conjunction with Sentinel One, which is the company that also sponsors this series. Now this is our first one that we're doing of one of these where Chris Krebs is not a guest because the original idea behind this was that Alex Stamos, who is Sentinel One's CISO and Chris Krebs, who was their then Director of Policy and Intelligence

would join me to chop it up. But yeah, unfortunately, Chris has resigned from Sentinel One after the US President Donald Trump signed a memo ordering the DOJ to investigate Chris and whatnot. It's all a big mess. So yeah we're actually just gonna push through without Chris in this episode but I just wanted to let everyone know don't worry Chris will be back on Risky Business at some point soon he's just a little bit busy right now. So I recorded this podcast remotely. It was a live event

being hosted in San Francisco and I was around the RSA conference. I was supposed to be there, but unfortunately I chose to cancel my trip to the United States with everything that's happening there right now. So we just recorded it remotely. Now the focus of the conversation is actually very interesting, right? So Sentinel-1 released a bunch of research looking at various APT crews and various foreign threat actors and mostly the threat actors that they've observed targeting them, right? So there's the North Koreans trying to get their people hired into Sentinel-1. There are Chinese APT crews and even Ransomware crews all trying to sort of attack Sentinel-1 and do bad things.

So really this conversation looks at what these threat actors are doing generally, in a few cases what they're doing to Sentinel-1 specifically and it's just all around, it's a really interesting conversation. So joining me for the conversation of course was, was Alex Stamos, who as you'll hear, is also the CIO of Sentinel One now, as well as being the CISO. And also Steve Stone, who is the, and I've got his title in front of me,

the SVP of threat discovery and response at Sentinel One. So yeah, just a great conversation with Alex and Steve talking through Sentinel One1's research, which you might have seen pop up in your news feeds. A lot of people covered this research. So I'll drop you into the conversation now where Steve Stone is describing what Sentinel-1 did with all of these North Koreans who were trying to apply for jobs at Sentinel-1. And instead of just throwing their resumes in the bin, they decided to string them along

for a little bit and see what they could learn. Here's Steve Stone, enjoy. Yeah, so it actually started, we were going through resumes for a position on Sentinel Labs, which is our threat research team. And we just saw some things that jumped out and we started poking and it got really interesting. And we had a couple of late night phone calls

actually with Alex. And part of this was, it was a risk decision. We had to make a decision on where we're going to turn it off right away or we're going to play this thing out and really try to understand the scope and scale. So we've spent probably five months actively working against the North Koreans applying. We've seen almost I think three hundred and seventy distinct personas. I think 370 distinct personas. They've sent in more than 1,000 applications. And this is not a one-day thing. It is day after day after day.

So we just kind of stepped on it and just kept after it. And we got all the way to the point, we actually tried to interview them. And that's where they would peel off. They would refuse to go on camera. So we could get them on email. We could get them in everything except for on camera. I mean, that's not usually the case, though,, is it? I mean I've heard of plenty of these

applicants at other places actually being interviewed on camera and doing quite well in the interviews. Yeah, I think it's them learning just like we do. I think they've seen a couple of videos at conferences like this one and they saw their faces on camera and I think they probably adjusted to that. Was there any point where they were doing stuff like trying to send over or trick people on your end into looking at, hey, look at my latest software project, just run this binary and did it get that far or was it really just too early stage?

That's a great question. What they were actually doing was that before they sent resumes in. So they did exactly what you just described, but they did that as the very first step. And what I think is really interesting on that, if we take a step back and look at North Korean intrusions, that's not new. They've always done that. They did that on LinkedIn. They did that with emails before they sent spear fishes.

So it's interesting to watch these old habits carry forward in just their new set of activity. So yeah, we we have multiple people at Sentinel one receive effectively cold email saying, Hey, I'm really interested. Would you mind helping me? Would you look at some of this material? And that normally leads to fishing and this didn't they were just trying to get more eyes on their application to get a job. Yeah, I mean, it, I mean it's crazy the breadth that they've been able to spin up. You know, this has been a long-standing

conversation between a friend of mine and myself, Dimitri Alperovic, who's one of the founders of your competitor, one of your major competitors. You know, he and I have often talked about North Korea and he's always said, oh, they're very creative, very, very good. So, you know, he takes issue, I guess, with some of the coverage of them because they're really in the limelight now. But my argument is that what the North Koreans have now, which they didn't have previously, is scale, right? So you would find that they were able to do some very creative things, but now they're doing these creative things at massive scale. Does that vibe with your understanding of the evolution of North Korea's TTPs?

Yeah, it absolutely does. And I think there's really two interesting elements. The first is we always in cybersecurity treat North Korea as if they're the anomaly. But if you look at North Korean cyber activity in the context of North Korea, it makes perfect sense. The US had to redesign the $100 bill because North Korea was the number one currency counterfeiter in the world. The EU had to redesign the euro multiple times.

So they have no problem doing illegal activities to make money. They're just now doing that in cyber. And it's not that different than what we see them do with cryptocurrency as a good example. So that's takeaway number one. This is all caps North Korea. This is very them. Second part is if you look at any of the North Korean cybersecurity efforts, once they figure out how to make something successful, they will just stay with it.

So I think that's a good indication that someone's hiring North Koreans because they're still doing this and they just don't waste their time in cybersecurity. In this space, we see the, I mean, again, I'll go back to the cryptocurrency. They are really effective at stealing crypto. They're really effective at understanding crypto exchanges. So if we compare that to the North Korean IT worker piece, we have to make a reasonable

assumption that they're being really successful getting employment. Well, I think we have to assume the reason they're coming after us and they're coming after a company that has an Intel team is not because we are the easiest way for them to make money. It's probably specifically because we have significant penetration in the crypto industry. I think you're right. I think it's about we're a piece of that larger puzzle they want. Yes, which I would expect that we're getting better applicants, that we're dealing with

a better, you know, anybody who does anything at scale, you end up with a diversity of skill sets. You know, if the lesser skill sets going to be going after the remote jobs at, you know, the business process outsourcers and the folks who aren't going to be doing after the remote jobs at the business process outsourcers and the folks who aren't going to be doing as much work here. I expect we're getting the better applicants and the people who are better. You're not getting the C team, right? Right.

I'm not going to say it's the A team, but if we're getting the people whose job it is to eventually do a supply chain attack because we're running in kernel mode on a bunch of machines that are handling crypto, it's not going to be the C team like you said. Yeah so I just one thing you said there Steve about you know the lengths to which North Korea will go to to bring in money. A fun little Wikipedia rabbit hole for anyone watching this or listening to this might be to look up the Pong Su, so P-O-N-G space S-U. This was a boat that popped up off the coast of southern Australia, very close to the beach, and it was a North Korean fishing boat that was smuggling heroin into Australia, and the swell happened to be very

big when they were trying to do the drop off, and a couple of them drowned and it turned into this whole thing. But just fascinating insight into the way, I mean this was a long time ago, but it's just fascinating when you've got fishing boats smuggling heroin into Australia as part of a state sanctioned activity. So yeah, crazy. It's like what would you do if you had crime, but you did crime at state scale? Yeah. Well, I think one of the things that's interesting, again, in cyber, we tend to treat North Korea almost

as lesser, because they're not the Russians. They're not the Chinese. But they're really good at areas where lots of other essence companies and groups struggle. So like North Korean IT workers are a good example. They've got front companies. And we did. We started calling the numbers on their resume.

We started tracking down their employment. There are people that answer the phone. There's actual effort to give this structure. And we can even go back to, I spent most of my career working Chinese espionage. We did attribution on them for years because they were really bad at registering domains as a business. North Koreans are doing that all the time. and they're really, really good at it. So I think we just have to recognize what it is they're doing, why they're doing it,

and what works and what doesn't. And you know it works because sometimes when you check their resume, they really had those jobs at other companies. Yeah. So you're like, we're not going to name any of those companies. But yes, you go and you notify those companies and they're like, oh, yeah, I'm sure it's fake. And then you don't hear back from them. And you're like, oh, okay, I guess it wasn't so fake. Now, one thing that gives me hope with these, you know, North Korean

IT worker, you know, situations is that it's very rare that law enforcement gets an opportunity to asymmetrically disrupt, you know, cybercrime. But there is a weak point with these operations, which is they are always, it seems, relying on some sort of laptop farm in a basement where these workers can remotely control computers that are based in the United States. So once you collect a little bit of, you know, a few bits of information about what the originating IP is

for these workers when they're when they're doing their thing You know you detect one you can shut down a hundred kind of thing, you know Do you think that law enforcement is going to be able to? Keep a lid on this. I mean, you know, you can never completely eliminate a Crime syndicate like this, but do you think there'll be some effective suppression? I Do I'm hopeful on that and I started my career in federal law enforcement. So that space is very familiar to me. And the reason I have hope in that is two reasons.

One, just as a bunch of cybersecurity companies, we are all working together on this. And there's not a ton of time as we've done that. Like the Ukraine War, SolarWinds, there's like a handful. We are working with other companies and governments literally every day on this. So there's just a lot of really, really good collaboration. And the second part is, to your point, law enforcement's great at pulling threads to

get to what they're looking for. And they have lots and lots of threads to pull. And there's physically places they can go action. That's not very common in this industry. So I think we'll have some good success there, I really do. Seems like the other choke point's payroll, right? Because these people are using mules in the United States to get payroll.

It feels like ADP could just solve this, right? Like if the law wasn't in the way and they had a team who cared about it. And so I'm a little bit shocked, actually, we've gone this far. Well, I mean, like that's a good example I think too. Like law enforcement, they've been doing that for 40 years. They know how to deal with that.

So I think in a lot of ways, they're actually happy to get away from all the hacking and go after like, oh yeah, money mules, that we can do. Like we got people for that. Yeah, I mean, I think the best hack I've seen in years was the Bybit hack recently, which was attributed to the North Koreans and the way they did that was amazing. They targeted the upstream like safe wallet supplier

to Bybit and the way they did that is they got someone at safe to run something. I think it was they pretended to be recruiting for a role and said there was a coding challenge, just run this project. And someone did. That's what got them their access. Then they were able to fiddle with the JavaScript that was being served only to one customer, which was Bybit. And they got away with $1.5 billion by subtly changing something so that the multiple parties who had to sign that transaction didn't notice anything was amiss. I mean, it was an incredible bit of work and my co-host Adam Boileau and I joked that, you know, one strong argument for Korean unification or reunification is that we'll get to sit down and have some drinks with these guys and actually talk to them about their capers.

But look, let's move on to another thing that you covered in the report, which was what ransomware operators are getting up to these days. And there's something very telling in the research, which is that one of the first things that they're trying to do is actually get access to EDR consoles and shut them down, which makes so much sense because EDR, just as a category, correctly configured EDR, you can pick it from four or five different vendors where if you're running it properly, it's going to defeat ransomware. But it can only work so long as it's running and correctly configured. And if you, as an attacker, can get access to a console, you can effectively shut it down.

So Steve, talk us through what they're doing in terms of trying to get access to those consoles. And I think the other bit that wasn't so much present in the document that I looked at anyway is once they've got console access, what is it that they're actually doing? Are they allow listing specific malware? Are they completely disabling stuff?

What do they actually do once they get in there? Yeah, so I think the first part of that is they effectively are acting like a prospect or a customer. They use all this really convenient technology. All of us EDR companies build. So you can test it. You can play with it. You can see if it works.

That's all they're doing. They're doing that. And they're trying to steal credentials and log in to consoles. And that's not unique to us. That's all they're doing. They're doing that, and they're trying to steal credentials and log in to consoles. And that's not unique to us, that's every single EDR. And I think what's interesting on that is we actually see them try a new technique or a new tool against all of the EDR technology in one fell swoop.

They're just being practical. They want to be able to ransom networks and steal data, and they know they have to deal with X amount of EDR companies. So I guess that's the second part of the answer there, too. That's what they're doing. They're just testing their gear, and they're seeing what fires, and they're doing one of two things. They're figuring out how to do something a little bit different

to either hit a more generic alert or just avoid it altogether, which actually really rarely happens. And then the second part is they just are testing what fires and what doesn't, and how do they change the settings, and then they go from there. So there's two parts here, right? So there's the part where they're trying to get access to environments with EDR in them so they can test, right? So that's what you were just talking about there. But there's the other part where you were mentioning stolen credentials and whatnot, even authentication tokens, I'd imagine, and that's going to be a feature of this conversation

a bit later on. I guess, you know, the first thing I wanted to look at is what are they doing once they've logged in with the credentials for, you know, someone who's administering the EDR for one of those companies? Are they just straight up turning it off? Or are they, you know, trying to allow lists certain bits of malware or disable certain features and alerts? What do they do once they actually get access to the console in a targeted environment? So this is separate to the testing side. Yeah, so that's changed a lot over time. In the early days of ransomware, which wasn't that long ago, they would just turn everything

off, which is obviously pretty easy to find. So what they do now is they effectively try to change configurations and hide in the settings noise for lack of a more technical term. So they really just try to make it so it looks like it's running as normal, but they flipped a couple of bits that allow them to run. And then I think if we look at a ransomware intrusion, like a typical ransomware intrusion lasts about five days. Lots of variation there, but typically they're inusion lasts about five days lots of variation there

But typically they're an environment for five days before they get to a ransomware deployment. So they're not Interested in being there forever. They're not interested in learning everything there is about it They want to know just enough to be able to do what they're trying to do for effectively a week And then they move on to the next thing and so they're not It's very different like the Chinese model the Chinese espionage groups have to hit a certain target. And so they go after that very, very differently. Ransomware groups just need to make money. And if that's at a target and it doesn't work,

they'll go on to the next one. And so their level of effort is really around how do they just make the console not work enough to let them do what they wanna do? And is this is this something that has happened you know and I'm not just talking specifically about Sentinel One customers but is this something that ransomware operators have been able to successfully do quite a lot like how common is it that someone who's doing

the right thing has EDR software rolled out across their enterprise is then getting owned sideways because someone was able to get access to the console. Well, it's super common if you have single sign-on, right? And there's a certain competitor of ours where their EDR is free if you buy your email from them, for example. And you can't turn off single sign-on. And so if you become tenant admin, right, then you can just turn off everything.

This is where you have to... This is the challenge of buying the product that's free when you get the bundle, is you have to be careful because almost by definition, if a ransomware actor is being successful, they're getting some level of administrative access in a hybrid mode Active Directory. These guys know what they're doing. They can pull the PowerShell. They've got their standard PowerShell script. They can turn the settings that they need.

And if you don't have the alerting in place, you won't know that happened until it's too late and you're doing the forensics. And so it's really common in those environments, because single sign-on's always turned on, which is one of the things I'm a big fan of single sign-on in lots of situations. This is where though having different administrative

domains for your normal IT administrative domain and your security protections, I think is actually important. And I often for high security environments, recommend that people are running a separate administrative domain for their EDR product and or their security logging than for their actual IAM and for tenant admin or whatever the equivalent is for whatever cloud service they're using their security logging than for their actual IAM and for tenant admin or whatever the equivalent is for whatever cloud service they're using

for this specific example. So that if somebody is able to escalate within their IT environment, they're not able to also turn off all of their logging and turn off all their security protections at the same time. Yeah, I was just gonna say, once you've installed all of your EDR,

can you easily migrate it to another administrative tenant though? Like I'm just, I don you know, once you've, you know, installed all of your EDR, can you easily migrate it to another administrative tenant though? Like I'm just, I don't know, but. Well, for us, you don't have to, you just don't have to turn on single sign on, right? But I'm just saying like, if you're, if you're running, if you're running M365 E5 and you're using Defender. Hey, look at that. You said the name.

You said the name. But earlier, a moment ago, it was our competitor who also. Everybody knows I'm talking about, so I've just decided to say it. So, I mean, there are ways you can do it, right? But like, it's tough, right? So it's like, that's where I've seen it the most from a forensics perspective,

just because bad guy becomes tenant admin, and they just know, great, I turn off Microsoft Sentinel, I turn off Defender, bam, bam, bam, I have one PowerShell script that does everything I need to do, right? And so that's where, and there are ways to try to prevent that, but most people have not put the steps in place to try to prevent that kind of stuff. And so my suggestion is, if you're running Azure Hybrid Mode, do not put your Sentinel-1

console plugged into that, because the whole point is if you if if Microsoft has failed you were there to keep you safe right and so don't make it that you've also lost control of both unified that authentication in somebody's FIDO token yeah Steve you wanted to jump in there yeah Alex's point on on MFA is a really great example, I mean, I think it, EDR killer, EDR bypass, we really want these to be like these super sexy things. I mean, here at RSA, there's spray paint on the mailbox about EDR bypass, EDR killer. Pentests love talking about them. Red teams, cyber criminals.

It's this big beefy topic. And we spend an inordinate amount of time as people would expect researching this. We research the claims, we research the tech, we go through all of this. Here's what has not happened. An attacker came in from an external avenue and found a way to fundamentally subvert the technology and used it for their own purposes. That has not happened.

What happened is exactly what Alex says. You have legacy agents, you have features that are turned off, you have one account, and it's just a password that's been sitting in a forum for two years. That is how, that is one of those things I just mentioned is in 100% of EDR killer EDR bypass. So you have an unpatched parallel to a box, you pop it, you pull creds out of memory,

you have a token that lets you become tenant admin, and you can just turn, you're a tenant admin, you can go to all the consoles, you just turn everything off. That takes no research, right? You don't have to spend 170 hours coming up with a kernel bypass for us, or CrowdStrike, or any other EDR product

that then might get patched out any time. It's the single sign-on MFA. And then it just turns out to be the exact same credentials that allow you to then push your ransomware everywhere with one RM or PowerShell script. So yeah. That's much more likely. The EDR killer is almost always talking to a SaaS interface.

It's almost always poor hygiene. It's poor, yeah. Yeah, so now let's talk about the other aspect to this research, right? Which is, and this was fascinating, and you were touching on it earlier, which is the fact that there's this entire underground economy where people are sort of spinning up

like, you know, virus total, but for EDR and for criminals, right? So that they can actually test their stuff against various EDRs. These guys have figured out that virus total belongs to Google. They finally figured out you can't upload your bad stuff to Google and get away with it.

Yeah, because someone's gonna notice, right? So I guess the interesting thing is here, like it's not terribly hard to get yourself a license for an EDR You know product right there is some basic KYC around that that most of the you know Most of the organizations do well all of them do But you know, that's a compliance checkbox, right? Like KYC no company really wants it to be an impediment to sales So stuff is always gonna to slip through the cracks.

And your research argues that that's sort of what's happening here is that these companies exist to sort of bypass KYC, you know, bypassing KYC as a service and essentially setting up test labs for people to throw their ransomware into. You know, pretty interesting. Yeah, we talk about in the research, there's a particular ransomware group that's actually spun up a whole front company just to acquire EDR technology in demos and POCs to then run at it and then sell it on the dark web in the criminal forums.

So I think what's so interesting about this, and this is part of why we released the research the way that we did, we don't want to just say, here's some bad stuff, it's all bad and we're out, like have a great day everybody. Like we wanna talk about how do you really solve these things? And if we look at the North Korean IT piece,

we didn't solve that in a traditional cyber way, work with Alex's team and we're manning the firewalls. That's not what happened. We worked with the people team and we worked with the people team to figure out how this was all going. And now we fast forward to your question here, we had to work with the procurement team, we had to work with the people team to figure out how this was all going and now we fast forward to your question here We had to work with the procurement team. We had to work with the sales teams. We had to work with SDR These are teams when you talk about the list of your cyber security professionals

They're usually not on that list, but that's exactly how we got to a lot of this and that if you don't know your customer Well, you've just accepted a ton of risk. And we're using the customer word when in a lot of ways it's actually even prospect. If you don't know your prospect, if you don't know your pipeline, you're just running blind to an entire surface area in your organization. And that's not unique to us as a cybersecurity company.

That is every single company who's using technology, period, hard stop. Yeah, truth is though, I mean, every security company, including us, has a though, I mean, every security company, including us, has a significant, I mean, we have MSSPs, we have system integrators. It's tough. We don't know, lots of our customers

we don't have relationships with. I mean, this was literally gonna be my next question, which is like, once you're selling software at the sort of scale that you do and your competitors do, like, I can't imagine that it's possible to have, you know know foolproof KYC right that's gonna prevent this. I mean you can minimize it but you know you're gonna have resellers and as you say like MSSP's and whatnot. We sell our

product in the Amazon Web Store and the Amazon marketplace. Right. Well that's why I go back to the procurement teams as a big part of the solution. Here's what we are not going to be able to effectively do. We are not gonna be able to look at every agent across the fleet and say, this one is attached to this MSSP, they've resold it and now we can technically find our way to see who's using that agent. That is not technically possible. What is possible is saying, this doesn't look right,

let me pick up the phone and talk to the reseller and they can tell you usually within minutes, because this is business, it is not hard finding a business person in a company. That is an effective and very quick way to find a thread to pull and now you can bring in your technical experts and let them do what they do really, really well. But what we have done is we've found multiple situations in which either their front companies or a company has been, the identity of a company has been stolen and there's a free trial has been provisioned but it's not the real company and such.

The great thing there is once we identify it, we don't just shut it down. We then toss it over to Steve's team and we go pay attention and now we have all those logs that are in our cloud so we can see everything they're doing. We can see the malware they're testing and then we can burn it. Right. So there is a risk to them of doing that and that we are a cloud. You know, they're not getting up on prem product. They're getting a cloud connected product. So, you know, if you're listening to this podcast and you're doing this, right.

Like it's not like virus total where that stuff is running, you know, in Google's environment. Um, we still have all that telemetry and so we have the ability to roll it back once we figure out that somebody was, you know, was using a fake identity and then have the ability to burn all of that malware later. Yeah, that was going to be something I was going to, I was going to touch on as well, which is, you know,

one of the opportunities there because we saw a fantastic bit of research, I think that was what, last year from the Sophos people. You familiar? Yeah. I mean, that was incredible what they were able to do, which is they realised that people, I think it was a Chinese contractor actually, providing tooling to like APT crews there. They realised that they were downloading like virtual machines, like VMs of Sophos products and using them to do exploit development.

And I think they were trying to turn Sophos boxes into orbs basically, right? And they started dropping like some pretty nasty stuff on these exploit developers and were able to burn their exploits before they could even use them. And these people had no idea what was going on, which was fantastic. I mean, have you actually done that yet, Steve? Or is it something that you're thinking about?

Yeah, I mean, we definitely do that. And I wanna give SoFo a lot of credit. That report they put out last year was great. And I think it really did expose how tough of a challenge this is and how pervasive it can be if you're not looking at it. I will tell you on our end, and Alex said it right out loud, and I don't think it's a secret, we look at all that. We're able to go back and say, here is what's working, here is what's not.

Oh, by the way, they just gave us all of these files. Let's rip all this out. Let's go alert on this here's what's not. Oh, by the way, they just gave us all of these files. Let's rip all this out. Let's go alert on this and just keep going here. I keep using that risk word in this podcast, but that's a big part of that decision. We make a very intentional risk calculus to let some things play out because we can move faster than the attacker can. We always talk about the attacker has the advantage. That's not always true.

And this is a great example. We get to see what they're doing and this is all people business. When the day is done, cyber is people and people get lazy really quick. And if that box doesn't turn off on day one, they're pretty sharp on day two and by day 10, they are just winging it.

And we're gonna pull all that, we're gonna learn all that just like we are, just like they are. And at the point we're gonna turn it off is when we think there's nothing else to gain. Yeah, I mean, once we know that they're doing this, it's like, it's way better. Oh, you're gonna run our agent on your systems?

Yeah, cool. That's great. No, you can have it for free now. Yeah. We won't charge. That's great, yeah. The idea of doing this used to be a little bit sort of Hollywood movie,

right? Like it just wasn't something that happened. I feel like that's really changed in the last few years where vendors are saying, okay, in order to develop the exploits that are going to be needed to target our technology, or in this case, in order to develop malware that will be able to bypass this technology, you know, we need to be playing with this technology. And there are just some terrific opportunities. I mean, you know, I agree with this technology. And there are just some terrific opportunities. I mean, I agree with you, Steve, that that was fantastic research from SoftBoss. And I like that they did that publicly, because it sent a

signal to all vendors that this is something that they should be doing. Yeah, I think that's important. And I think what they did, I think they deserve a lot of accolades and kudos. They took a risk. And back to that R word again. But whenever a security company goes out loud and says, hey,

we're getting messed with, and this can be really, really nasty, that is a really risky decision. And I'll use the research we put out today. Here's what also did not happen. Some of our researchers did not write a blog, and it went out. And we're all like, OK, great. We had lots of meetings and discussions.

We're meeting with our C-suite. We're of meetings and discussions. We're meeting with our C-suite. We're meeting with our lawyers. We're meeting with our tech experts to say, if we talk about this, what are we gonna lose here? And it's worth it. We think it's important to show other organizations, one, this is real, this is happening. This is not us making up a marketing story

to sell technology. And also, here's how this gets solved. And in many cases that's solved. You don't have to be a gazillion dollar Fortune 100 company. It's actually like really accessible for most companies once you kind of do things just a hair differently and I think that's really really important here. Alright so look one more thing we're going to talk about from your report, from your research is, you know, a look at China, which is obviously the, you know, the most at scale adversary, I would say.

I mean, you've got North Korea doing things at scale, but a lot of that's involving like social engineering at scale and you know, that's the way they roll. China, a little bit different, more sort of, you know, classic hacking, if you will. You had a look at two APT crews one you call Purple Haze, one you call Shadow Pad. I believe with the Purple Haze example they were actually trying to target your customer environments and then swim upstream into your systems. Why don't you walk us through what you found there? Yeah so old-school China watcher, China cyber is my absolute most favorite part of this.

And I think the hardest thing I've ever had as an intelligence professional is accurately conveying just how bad the Chinese cyber problem is. Because you sound crazy. You sound crazy discussing how big and how pervasive and you lose people. And so I think we look at the two different Chinese events that we had, it's a good example of just a tiny slice of that scale. We had one Chinese espionage group that did subvert an organization that happened to be one of our clients.

In the course of that, they made a really hard run at our console. They wanted to understand our technology for all the reasons we've already talked about. They had also fully subverted the VM it sat on. They fully subverted the server that the VM was on, and they fully subverted the account of everyone who had logged into that VM. So we talk about like pervasiveness. That is just one tiny little piece.

So we saw them, we dealt with that incident response, we worked with that organization, and what we were able to do because of that, we really dove into that, we could then pivot back and say, oh wait, they came at us. They really tried to take what they learned at that victim and immediately pivot and see if they could compromise Sentinel-1 direct. So we have that going on in one case with a group of malware that is not unique to that Chinese group, which is not unusual. Then we have a separate Chinese espionage group that went after at least two of the IT providers that we use. And I want to be careful here. I can't tell you they compromised these organizations because they wanted to get to Sentinel-1.

I just cannot tell you that. One of them definitely did because it is a small company. The odds that they went after them the same month And this is it one of them is like a very important company that services a lot of organizations One of them is a tiny company that does not and for them to go after them the same month that we're like engaged in I fight with them. Yep, that is not a coin. Yeah highly likely. Sorry highly likely I I don't believe in coincidences of that level. No, no.

Alex, as a lifelong CSO, more or less, can feel it in his waters. It's been interesting a couple of months, Patrick. No, it's just, I mean, this is what you're dealing with when you're dealing with the Chinese adversary. I mean, what I tell my team is there are at least 20, maybe 50 people whose job it is just to break into Sentinel-1 working in the Chinese government.

Wouldn't know that at all. Yeah, absolutely. Yeah, right. Because it's like that's the size, you know, the Chinese have what? Probably 100 to 150,000 offensive operators. So we're not as big as Microsoft, right? They have at least a thousand people breaking into Microsoft. But if you had 150,000 operators, you would take the entire security industry, then you'd

divvy up. And so that means for my team, I tell them, like, when you stop them, they don't go away. They go home. They come back the next morning. They have their local culturally appropriate caffeinated beverage. They get a little talk from their bosses, and then they start again. That's all they do all the time, is they try to break into us. And if they can't get in, then they're going to look at our supply chain, and somehow they found out that these two IT vendors, neither of which were obvious, right? So neither of which these vendors were advertised, we're not the government.

So like Beyond Trust and Treasury, you can look that up in the database, right? You can just look up that treasury had to be on trust. There's no way to publicly determine that these two vendors worked with us, which is an interesting thing to try to figure out how they knew to go after them. So yes, it definitely raises one's level of paranoia when you're playing at this level. Well, I think when you're being targeted that pervasively as well, you have to operate under the assumption if you can think of a way to get you owned, they can too. And I think that's changed a little, right? Because we've always said security through obscurity is no good, but you have to operate under the assumption now that your advers your adversaries particularly Chinese APT crews

Have a very detailed understanding of your environment your tooling how it all works I mean, they're very good and our people right we're now at the level where they'll be using human intelligence Because like this is also what I tell my team is like we're at least at their level when it comes to cyber That doesn't mean we're perfect. They could beat us, but we're at least playing the same game. When it comes to human intelligence, we're children. When you're talking about the Ministry of State Security, they are the descendants of intelligence agencies that have been doing this since Confucius.

It's the same when you deal with the SVR, came from the KGB, which came from the NKVD, which came from the Czechists. It's like, they're like, oh, we're communist now? That sounds good. Right? Like change the hat. Right? Like you're talking about like a tradition of human intelligence that goes back hundreds of years. And they're good at it. And they're very good at it. And it's the same thing in the People's Republic of China. They're like, oh, Mao, okay, sure. I'll be on that team now, right? Almost by definition, it is the people who are the best at changing their stripes are the ones who survived

the Cultural Revolution and the Great Leap Forward. And they're like, oh, sounds good, sure boss, right? And we're at that level. That's what Steve's research shows. And so we have to be hardened up against the human intelligence level too, which is I played at that level at Facebook and it's not fun to play at that level. It's not fun to have your employees targeted at that level. I'm just going to cut in now because we are already, I can see that we're going to get

a little bit crunched for time and my apologies to the event organizers, but we are going to go over time. But just quickly Steve if you could fill us in a little bit too on the Shadowpad research that made it into the report. Yeah so Shadowpad that group of activity was one of the the two events that I mentioned and I think what's really interesting there and it keeps going back to the scope and scale Shadowpad's not unique to a single group you can't do attribution off of Shadow Pad. You have to understand lots of other things around it. And when we were looking at just this chunk of Shadow Pad

activity that we were involved in, we found 70-plus victims. So this is a narrow slice of one particular approach that they use, that multiple groups use. And in just that blink of an eye we see 70 plus victims Then why do they like we haven't talked about this? When they know that then we can key off of something like this We find that we can find and notify these 70 victims Why do is it just that valuable for them to have that economy of scale? Yes, that way they do it

I think it's just that I think it's that straightforward and I think it's an issue over it's non-negotiable. There's been so much research that says there is a list of companies and a list of organizations and they must get subverted. Right, so this is the flip side of them having such large capabilities and then having such a long list of operations is that you don't give people creativity.

You're like, here's your tools, here's what you're doing. Go do it. The best example I've ever seen in research is from our, and this is going to be weird now, but our own Dakota Carey over on Pinnacle One, he had this just incredible piece of research where it actually showed a Chinese company had to take and compare what they were tasked to do in China's five year plan and write out the technologies that they thought they would struggle with. That went to a specific government organization

that then looked across China and said, we have these technologies other places, or we don't. If it's a no, it went to a third organization, which then had the original company go and fill out an actual form. There are screenshots of this form that said, do you want us to buy it? Do you want an insider?

Are we going to send students? Is it going to be a partnership? And then the classic other at the very bottom. And then you could take that other and you could map that directly to lists of compromised Western companies. And it's just that when you actually even like speak with them, you talk like some of the original negotiators with China and US with cyber,

they will tell you, yeah, they're very, oh, they think we're the fools. They think we're the fools for not doing this. Yeah, because we don't put an other on the procurement form. And they'll be like, you don't hack other companies? Like why, you get so much great research. I mean, fine, you don't want to,

that's your choice, but shit, okay. Works pretty well, guys, come on. So I think it's just that. We recommend other. Yeah, other works great. Other works great. Okay but shit, okay. Works pretty well guys. Come on. So I think it's just that. We recommend Other. Yeah, Other works great. Other works great. Okay, okay, okay. So let's move on to the next part of this conversation, right? Which is about how the threat environment has changed and how the tech stack has changed in response to this. So one thing I would like to note here is we just spoke about attacks from North Korea, ransomware actors, and China, right?

One thing we weren't talking about is client-side exploitation of like, browsers. This is not how attackers do it anymore. We weren't talking about people just dropping malware on, you know, via email into, onto someone's machine and having them execute. I mean, in the case of North Korea kind of but that's developers weird edge case whatever my point is attacks have changed right a big part of the reason for that is because of the ascendance of EDR right appropriate endpoint controls and not just from you I mean it's an established category it's a mature category there's a bunch of companies that make good EDR. The point

is that the way that you get on target these days is not the way it used to be. Where it was like a watering hole attack and you throw a browser exploit at someone and you get a shell and then off you go. It's not like that these days. You've got, you know, we were talking about some supply chain stuff there, like the Bybit one is a great example of that. You were talking about some supply chain stuff there, like the Bybit one is a great example of that. You were talking about some Chinese APT crews who were trying to target your suppliers to get into your organization. So supply chain, supply chain, supply chain is a very big part of all of this.

We're also seeing particularly the Chinese hitting those devices, vulnerable devices at network edges. And then from there, because often they're domain joined appliances, they get all of the material they need to then go and access, you know, Alex was talking about this earlier, access the EDR consoles, shut them down. We're also seeing fish kits these days, the sole purpose of which is like pass-through phishing, which is even effective against one-time codes, right? Not effective against FIDO, but that's a whole other conversation.

And their whole goal there is to grab a session token out of a browser. We're seeing people starting now to publish malicious Chrome extensions, for example, so that they can get those authentication tokens. Again, maybe from there they can pivot into some sort of administrative console, either, you know, Microsoft One or for an EDR platform. But the point is, things have really changed. It's great that we've racked up some wins with EDR, but I do feel like the tech stack that we're dealing with now isn't quite equipped for this

world. Things have moved on a lot but I don't feel like, I think we're sort of still focused on yesterday's problems a little bit. Alex, I want to start with you on this. You know, what do you think about that statement that we haven't appropriately moved forward without, you know, security tech stack. Yeah. So I think the thing that now needs a change, and it is changing, is the traditional way that security products are built is that one product does the instrumentation and detection itself, and that product is dedicated to a specific deployment scenario.

It does endpoint. It does cloud. It does email. A company might have three or four different products, but those are different products. Yes, you might platformize. I'm sorry, I owe Palo Alto Networks $5 now. Every time I say the word platform, Nikesh makes $0.25, and an angel gets its wings.

But they're actually different products, right? And the idea we think about it. And so you end up with an endpoint product that does the instrumentation of the endpoint, and does the detection of the endpoint. And it might pull that back into a single console, but you still have a Cloud product, endpoint product, an email product.

And I think where we need to go, and this is starting, we're doing it, and I know other people are doing it too, is you have to detach the instrumentation from the detection. Because as you talked about, we now have this very complex mix of intrusion sets, where an intrusion doesn't just happen in one place, it happens in a bunch of different places at once. And the behavior on all of those different domains is sub malicious in those different domains. And it is only malicious when you zoom out and you see the whole thing.

And so you'll have endpoint instrumentation, like an EDR that sees all the stuff that's going on, but the detection can't be there because you can't see all the stuff you need. And you'll have cloud instrumentation, which is mostly just talking to APIs. I don't know why you'd pay $20 billion something that just talks to APIs, but you know what? I don't have that much money, so I guess I'm not smart enough. You'll have email instrumentation that looks at all the email, but then what you need to do is you need to have the instrumentation that pulls that data into one place, and then you're looking at the stream of all of the events, all those places, and

you're delaying, you have delayed gratification here where you're delaying the decision of whether or not something's malicious until you see all that context. This is super hard. It is super hard because the mixture of all of those things at every single organization is totally different, right? It is much easier for us to come up with an alert that happens on a Windows endpoint, because a Windows endpoint at JP Morgan Chase versus ExxonMobil versus Delta, those Windows endpoints might be configured a little bit differently, but realistically, somebody is

running Word and they open up a Word file with a bad macro, that looks the same, right? But those three organizations have a very different mixture of the Windows endpoint and AWS GCP, Microsoft 365, how their email's configured, how all these things are configured. And so that is, we've made it this far, nobody's talked about AI, right? But this is where the kind of modern machine learning algorithms can start to let us be

a little fuzzier and all this stuff, and I think we can start to get there. And I think that's, I think, the next revolution here. Does this make sense? No, it does, it does. I mean, I guess one of my issues, though, with what you're saying is I feel like there's certain sort of detection and primitives and controls that we just don't have. And a good example of that is a malicious extension that a user installs on their browser that grabs a very important authentication token and from there the attacker just goes

immediately to great victory. So the idea that there's such a simple path at the moment and that's just one example, you know I just sort of think well what are we doing? That's an interesting one because we you point out a different issue which is we've created these blind spots that exist in these like incredibly complicated execution environments that are self-contained. And the browser is the best example of that. And that a Chrome is a world unto itself.

It is a operating system unto itself. It has multiple security boundaries unto itself that never leave the process boundary. And so like an EDR product, which operates in the kernel, that expects that you have to leave the process boundary for anything to work, that you have to at least call a DLL. If everything is happening in proc to a single Chrome process, then to us it is just a black box.

Unless you now, you have to do like injection into the process and Google updates Chrome like literally every 72 hours or something. So Godspeed doing that, not crashing Chrome all the time. Well, I mean, there are options there, right? So you can look on disk at extensions and things like that, but you know, try detecting that. There are options, but then that's why like, I'm just saying like that's, we've created those weird little places and they're like, but you know, try detecting that. There are options, but then that's why I'm just saying

we've created those weird little places and they're like, okay, great, now I have to have an incredibly specific spot solution because Google decided to create this entire pocket universe that collides, it's like looking inside of a neutrino and you're like, oh my God, there's an entire universe in here, right? Like, and it becomes like a dorm room pot session, right?

You're like, every neutrino's a universe, right? But like, that's effectively what Chrome is. And then you have to come up with a spot solution. And there's only like a couple of those. But browsers are like the best example of that. And you're right. The other one's visual is IDEs. And we're dealing with this at Sendle 1 right now.

For all of you CISOs, if you're bored, if you're like, man, my job's really boring, I'd like to ruin my next quarter, go look to see what plugins your developers have installed in Visual Studio Code. And there you go, you have ruined your life, yeah. Let me just jump in there, because I think another thing about all of this

is you were talking earlier about, oh, GCP and Azure and M365 and whatever. You know, from an authentication and an identity perspective, when you're thinking about what is cloud and what is SAS, is there even a difference talking about dorm room pot sessions, right? Is there a difference between those things from a, well it's just a token in your browser, isn't it? Right? So I feel like, you know, we're seeing some really sophisticated attackers these days just moving through clouds and SaaS, right? And intermingling between both of those things. And I just feel like, okay, what's your solution there? I feel like the IDPs and the SaaS providers are kind of letting us down by not giving us standard form logs that can even be,

you know, that you can even make sense of. And, you know, the IDPs might develop some great features that the SaaS providers then don't implement. And, you know, I just feel like there's gaps here that are actually quite surprising. I mean, you're hitting upon another problem, which is the vast majority of companies, internal cloud authentication is a complete disaster. And that the vast majority of companies I have worked with,

you're like, oh, we just threw into a VPC. And so anybody who could talk to another IP address, we're fine. Almost nobody who's not an actual hyperscaler themselves has actually implemented some kind of cryptographically secure dual authentication mechanism, because to actually do that at scale and for it not to be spectacularly brittle

is almost impossible. And the fact that the hyperscalers have not made that easy is honestly on them. And so that is a huge problem. We have not dealt with identity. You're totally right that identity is the biggest problem in a lot of these cases. And if you drop onto a single endpoint, if you're smart enough, you can grab cookies out of a browser for a bunch of DevOps engineers that then can be used in a bunch of different contexts.

And that is a problem in a lot of cases. And then people are not tying that to like a hardware root of trust, even though these are all $4,000 totally stacked MacBook Pros with hardware roots of trust and biometrics tied to them. But they're just in the end using cookies that are in the unencrypted Chrome store, right? That are actually being used, yeah.

I mean, I think the reason I keep coming back to this is occasionally I'll be talking to someone keys that are in the unencrypted Chrome store, right, that are actually being used. Yeah. I mean, I think the reason I keep coming back to this is occasionally I'll be talking to someone and they'll say, well, what do you think we should do about this? And I just say to them, I have no idea. Alex, before the hook comes out Warner Brothers style to pull me off stage, we've actually got to wrap it up there. But that was a fascinating conversation. Alex Stum or Steve Stone, thank you so much for all of that. That was great. Thank you.