CyberWire Daily - Software supply chain management: Lessons learned from SolarWinds. [CyberWire-X]

Episode Date: January 3, 2023

Between the emergence of sophisticated nation-state actors, the rise of ransomware-as-a-service, the increasing attack surface remote work presents, and much more, organizations today contend with more complex risk than ever. A “Secure-by-Design” approach can secure software environments, development processes and products. That approach includes increasing training for employees, adopting zero trust, leveraging Red Teams, and creating a unique triple-build software development process. SolarWinds calls its version of this process the "Next-Generation Build System," and offers it as a model for secure software development that will make supply chain attacks more difficult. On this episode of CyberWire-X, host Rick Howard, N2K’s CSO, and CyberWire’s Chief Analyst and Senior Fellow, discusses software supply chain lessons learned from the SolarWinds attack of 2020 with Hash Table members Rick Doten, the CISO for Healthcare Enterprises and Centene, Steve Winterfeld, Akamai's Advisory CISO, and Dawn Cappelli, Director of OT-CERT at Dragos, and in the second half of the show, Rick speaks with our episode sponsor, SolarWinds, CISO Tim Brown. Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. Hey, everyone. Welcome to Cyber Wire X, a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the Chief Security Officer of N2K and the Chief Analyst and Senior Fellow at the CyberWire. And today's episode is called Software Supply Chain Management, Lessons Learned from SolarWinds. At Program Note, each CyberWire X special features two segments. In the first part, we'll hear from an industry expert on the topic at hand. And in the second part, we'll hear from our show's sponsor for their point of view.

After a word from our sponsor, some of our regular subject matter experts will visit me at the CyberWire's hash table to tell us how they think about software supply chain risk. I'm right back. Here's a word from the leading IT management software company, SolarWinds. For more than 20 years, SolarWinds has focused on providing simple, powerful, and secure IT management software, built to accelerate your digital transformation. Everything the company does is guided by being Secure by Design. Secure by Design is a new gold-plated initiative designed to set a new standard in secure software development. With Secure by Design, SolarWinds' internal environments, software build processes, and ongoing lifecycle management all adhere to a

multi-layer security framework. The newest solution built using Secure by Design is SolarWinds Observability, the company's first fully integrated SaaS offering. SolarWinds Observability uses powerful machine learning and artificial intelligence to provide comprehensive visibility into today's modern distributed hybrid and multi-cloud IT environments. For more information about SolarWinds Observability or the company's Secure by Design initiative, visit SolarWinds.com. Rick Doughton is the CISO for Healthcare Enterprises and Centene, a regular guest at the CyberWire hash table and an old friend of mine.

And we were talking about the fact that software supply chain attacks aren't new. The first one that everybody remembers is the 2013 hacks against Target. But since then, in 2017, we've had not Petra, solar winds in 2020. We got the log4j stuff in 2021 and 2022 and still ongoing. And every time we get one of these big attacks, you hear pundits say, well, this is the wake-up call. Everything is going to change now. But that never happens. At least that's my sense. I asked Rick if he felt the same way. Yeah, I thought you were going to say when we see those things, we're going to be like, yep, not surprised. It's like, I don't know why everyone's upset because like, how did you not see this coming? But it should be an opportunity to be able to say, see,

let's not be that. But it's really short-lived. And we've struggled with third-party risk management for decades. I mean, I remember as a consultant in the early 2000s going out to companies and walking around data centers and call centers and print shops where banks mostly or other companies were sending information that was shared to go out to their customers. And so, but we keep asking the same questions, but we're not really digging down into the root of where these came from, which is not checking for just the presence and appropriateness of controls, but the effectiveness of them. Steve Winterfeld is the Akamai Advisory CISO, another regular member of the CyberWire hash table, and just happens to be my best friend. He said that in order to cover third-party risk management, you should focus on two areas. Here's Steve.

So Rick, I agree. You know, supply chain is a really weird threat. We don't systemically have an approach across the cybersecurity industry. And it's one kind of like insider threat. You know, there's a major incident with insider threat or supply chain every year or two. And we really put a lot of effort into it in the short term. But systemically, it's not a high enough risk that we're doing long-term and systemic investment. When we do think about how to solve it, for me, there's two approaches. There's vendor management, and then there's technical controls. On the vendor management, it comes down to the kind of contract you write. What are the technical controls they're required to operate? What audit rights do you have?

If they have a pen test or they have a third party come in and do an audit, are you allowed to see the results? And are you included in the remediation plan? And finally, what are your notification rights? If they start to do an investigation and determine that they're breached and then make an announcement, at which one of those stages are you notified? Ideally, you want to be notified before the public is notified and you want to be part of the investigation so you know if there's any chance that there's actually penetration of your network as well. Then on the technical side, for me, it's all about situational awareness, that visibility. A great example nowadays is, you know,

that micro-segmentation, a software agent-based thing that's allowing you to see data flows. And now I can say, why has this application got a steady communication outside my network? Or why is this application touching multiple other servers in my network? So I can see those anomalies. And this might be, you know, not only a tool like that, but there are teams. I want my threat intelligence team involved.

I want my threat hunting teams involved. I want my threat hunting teams involved, looking for those odd behaviors for both insider threats, supply chain, anything that's in my network and inherently trusted with odd or suspicious behaviors. And I say my environment and this multi-environment we have now, I've got stuff in a data center. I've got stuff in maybe two or three different cloud providers. I may have stuff in SaaS providers or third-party applications. And all of this, you know, how do you get a common risk portfolio across all of that, I think is a big challenge. I want to go back to the original question. Because of these big software supply chain hacks of the last decade, is anybody

managing this any differently these days? I don't think so, because it's like you said, to get to that level of granularity to find those things, where those examples come from, take real scrutiny to say, how does this process actually work? the SolarWinds is my favorite one to talk about because when we look at, oh, SolarWinds got bought by an investment firm, they decided to move some of the development to Eastern Europe, you know, a lot more Eastern Europeans

closer to Russia are doing the development. Of course there's going to be, you know, it's like, you know, even if you just said that to me, I'm like, oh yeah, then there's going to be some coercion and people are going to get some, trying to sink some code in because SolarWinds is everywhere. And it's also one of those non-dis, you know, like easy to ignore because security people really don't use it. The network people love it. It's SNMP based. It's like whatever, but it's always there. And it's just quietly like sitting there collecting

stuff. So it was the perfect compromise and the perfect way of doing it that you're you know like i said when i saw them like of course we didn't see that it's on us for not having somebody scrutinize it enough to notice that well i agree and then i and even let's talk about log4j for a second i read a news item this morning that said the a third of the instances that are deployed on the internet for Log4j are the old version that was compromised a year ago. A third. Yeah, I mean, configuration management is still very hard for a lot of organizations. I think I've said before that in the last few years of doing all these CISO roundtables, I'm really feeling this big gap between the haves and have-nots.

And you have a Fortune 500 companies that have people and budget and things, and you have 600,000 other companies who don't. Yeah. And so we have just, you know, this little sliver of companies who can do these things and others are like, I don't even know how to update it. You know, and I look at that as being responsible for my subsidiaries who are not connected and are smaller, that they don't have security people. They have IT people. They're focused on the business. And so even when we as the parent company say, hey, have you fixed all this stuff? They're like, where is it?

I don't even know what to do with that. Well, you're right about that. And on my own podcast here at the Cyber Wire, CSO Perspectives, I've been spending the last two years talking about first principles. And what I've come to the conclusion is that for most of them, most organizations don't have a way to do them. If you're talking about zero trust or intrusion kill chain prevention, that's mostly in the box for mid-sized to Fortune 500 companies. Because, you know, I work at the Cyber Wire, it's a startup. I don't have a staff to Fortune 500 companies. Because I work at the cyber, it's a startup. I don't have a staff to do those things.

So for the small companies, they have to do something different like resilience. Just survive it and keep delivering their service. So it's definitely different for how big you are and how you deal with these supply chain attacks. You talk to a lot of CISOs. You're on so many forums and things that I can't keep up with you.

What are they talking about? The things that, is zero trust coming up in those conversations? Yeah, I mean, it comes up just because it's a very popular term. And, you know, and we all roll our eyes when the first person who brings it up.

And, you know, because it's different for everybody. You know, it's not a thing. It's not a product. And there's no like, there's no finite way that you're finished with it. It's just an ever,

it's a journey, not a destination. And it's also, if you look at all the tenants to it, it's something that we should have been doing for 20 years. And it's like, what do you mean you're not doing that already? So, but I think that we come back to the basics,

you know, back to the CIS, critical security controls, and just like, let's go one, two, five, asset management, configuration management, vulnerability management, patch management, you know, I mean, just get the basic out of the way and you'll take care of a lot of these things. And then we lay on top of that application security, which is what this is about. So, you know, obviously I've been saying more about like third-party risk management in general, But I know the topic being application security supply chain, we take that gap of maturity and then apply it to software

and it gets even bigger. People who actually understand software, unless you're a small company, that's what you do is create software. And back in the day when I ran pen test teams 20 years ago, we would just do code reviews. Because usually it's libraries or segments or pieces, not like whole applications. We do pen tests on whole applications. So how does somebody put that in?

And I think that the cloud is actually giving us opportunity because in the CICD pipeline and its application development that is made up of so many different pieces, development that is made up of so many different pieces, most of it being third-party libraries and other borrowed thing or shared services and APIs, we're starting to get back to it. I've been really thrilled to see that the industry is now coming back to threat modeling, you know, that we've kind of ignored for 15 years and then now cloud is making it viable again. And so I think that the tools are there. It's just kind of getting the awareness back out to say, we need to look at all the pieces of it, not just the stuff that's in the checkboxes and some of the frameworks. So with your interactions with all these CISOs,

Zero Trust gets a hand wave. What about S-bombs? Anybody thinking, oh, we got to do S-bombs to fix this? Anybody jumping on that bandwagon? Yes. That one is because there's kind of a mandate for it. So yes. But fortunately, the tools are better now,

good enough now to where they can create it. It's not like some poor developer has to sit there and document everything. The tools now can look at it and create them and then even run vulnerability management against vulnerability assessments against them. So, that's been very good. Don Capelli is the director of the OT CERT at Dragos, a regular guest here at the CyberWire

Hash Table, and also author of the Cybersecurity Canon Hall of Fame book, The CERT Guide to Insider Threats, How to Prevent, Detect, and Respond to Information Technology Crimes. I asked her if she thought the InfoSec community was making progress with their SBOM deployments, and her answer completely surprised me. She said that because of a set of security standards developed by the industrial process sector years ago called IEC 62443 that mandated SBOMs, industrial organizations are way ahead of all the other sectors, which is unusual because typically those organizations lag behind the other sectors when it comes to defensive architecture. Here's Dawn. I'm a big proponent of SBOMs. I think that industrial organizations are way ahead of the rest of the community because in the industrial control system arena, IEC 62443 is the security certification that the OEMs have been marching to for many, many years. And it takes many, many years to get IEC 62443 certifications. One of the things that's required from 62443

is a secure development life cycle. And part of that secure development life cycle requires software bill of materials. Since companies like Rockwell Automation and Schneider and Siemens and Honeywell have had these certifications for years, this means that they have SBOMs for their products, at least their products that are certified. So like at Rockwell Automation, where I was CISO previously, part of our SDLC required an SBOM. So before you could release your product, you had to submit the software bill of materials for that version of the product. Now, years ago, that was basically Excel spreadsheets. So here's my source code, and here is my SBOM in this Excel spreadsheet. Then as time went on, technology improved so that now there are tools that will create

your SBOM for you. So when Log4J hit, for example, it was discovered and made public, disclosed in the middle of the night on Friday morning. By Sunday night, Rockwell Automation was the first vendor in the industrial control system arena that disclosed all of our log4j vulnerabilities in our products. So by Sunday night, we had the complete list of all of our products that had the log4j in it, and we never had to update that list. So it was 100% accurate when we put it out on Sunday night. And that was because we had those S-bombs going back many, many years. in IEC 62443, it is required that you also impose the SBOM requirement on your software suppliers. So again, in the industrial control system arena, this is one place where SBOMs are much more mature, not only for the OEMs themselves, but also for their software supply chain. The obvious strategy to me to reduce the risk of software supply chain risk is zero trust.

Unfortunately, because of the way security vendors glom on to popular ideas, many marketing claims insist that their product solves zero trust for their customers, even when at best, whatever their product does is a mere piece of the zero trust equation. So much so that CISOs and other security practitioners just roll their eyes whenever a vendor makes another claim. And as a community, we tend to throw the baby out with the bathwater. Here's Rick. And I'll go back to like mature CISOs I roll at Zero Trust. You know, CISOs who are, you know, have always have been successful in using that as getting budget, you know, because it's made the mainstream media people saying,

and their executives like, what's a Zero Trust thing? Are we doing that? You know, it's like, did we buy that yet? So they are getting traction traction with it. So, as much as I roll to it, it is effective. And if you're not doing it, then it's the right path to take. Well, I'm going to push back on that. We're just in the Gartner trough of disillusionment with the idea of zero trust. But the idea of it's right. Limit access. That's all it is. Right. To like five different things. To identity, to resources, to data, to systems, to, you know, blah, blah, blah. Yes, of course. That's how you're supposed to do it.

If you haven't been doing that and you're too mature, then, you know, you can label it and then get money because it's now got a label. So let's get back to what you said before. I want to bring you back to this because it seemed like that's the most practical thing to do is really a formalized third-party risk management process. This will be my third semester. I've had friends who teach master's courses and we do capstone projects on third-party risk management to have the students kind of develop what a program would be. And so it's interesting kind of what they come up with. But foundationally, it's kind of like you identify all of the people who you share data systems or rely on for systems from a resiliency standpoint. And then you prioritize them based on impact to the business. If something goes wrong, whatever those things may go wrong, you kind of identify those, whether it's a stolen of data or a lack of that resource being available.

And then you kind of define, what is it do I need to know about them to make me feel comfortable that they're protecting it to the level I need to be protected, whether it's, again, resilience standpoint, protection or privacy, etc. Then we come back to like the old school questionnaire thing, you know, or there might be, you know, yes, there's things like security scorecard and bit site, they're doing this real time passive scanning, you know, those are, you Those are kind of like a credit score thing, which doesn't really mean a lot, but some people use it because at least it's a piece of the puzzle. But then sometimes it might take being able to say,

all right, you're so important to me. I'm going to send somebody to come sit and talk with you. And I did that 20 years ago. And just'm like, just make sure you're cool. You know, do you have someone responsible? Do you have anything? But what's been interesting in this particular space is the insurance companies. Because the insurance companies are kind of stepping in since they've been losing their shirts

the last couple of years. And they're getting very, very granular with what they're asking for. I have peers of mine saying, oh my gosh, I had like 15 questions about how I manage my service accounts. Or another one is saying, they're really drilling into me about how my backups are and whether it's connected and offsite and how I'm managing the passwords, the backwards, you know. So you can tell which people are getting, you know, having payouts for

things like, okay, your service account's misused or backup's not blah, blah, blah, you know, from a ransomware standpoint. And I have a feeling that that, you know, and I don't know if it's a good thing or not, that that might be our standard is like, hey, did you go through the insurance process? And which one was it? And do they think you're cool? Because they're actually starting to get some real metrics of like, what's, you know, what's eating their lunch. So that's like a shorthand. So you either do the third-party investigation yourself or, hey, do you have cyber insurance? And if it's through Aetna, let's say, oh, that's good enough.

I don't have to do anything else. Is that what you're saying? You know, you might get, you know, you can, you know, depending on how big you are, because also like it depends on size, like I'm not going to be able to get Microsoft or Google or Amazon to like give me their, I mean, they'll have on their website, their SAS 70, I mean, I'm not going to be able to get Microsoft or Google or Amazon to give me their, I mean, they'll have on their website, their SAS 70. I mean, I'm sorry, what year is this? Anyway,

their SOC 2 and their ISO certification. That's Rick going through a senior moment, everybody. Yeah. And so that they have those and they say, listen, you're going to accept these because we don't, you know, we're bigger than all of you. So just trust us. We're doing more than you are. But if you're smaller, you can request those things. And, and going back to the insurance is like, we will often request insurance, but some of them can't afford it for the reasons that I said.

I mean, one thing that all my friends have said is like their insurance premiums have gone up like 300% and they're getting where two and a half years ago, it was a two page like, Hey, do you suck or not answer these five questions? And that's it to now, you know, you pretty much are going through, uh, you know, an IRS audit, um, you know, of like, how are you doing this thing? And I want evidence of that and et cetera, et cetera. So I don't think people are necessarily using them yet. So it might be required. Also, for instance, we're in healthcare. So if there's someone who's sharing PHI with,

then there is a business associate agreement which has all the contractual stuff and everything and requirements of what they're supposed to do that we will negotiate whether like, well, we don't really use five passwords and you're out, is that good enough? But we won't say you can't, we won't accept like they don't use MFA as example.

So is there a takeaway here, Rick? Is there a one-liner that says, if you're going to do anything to mitigate the risk of software supply chain risk, is there one thing you could focus people on and say, do this and you're in the right ballpark? Well, certainly examining any code you get from somebody. If you're getting libraries and other things, or if you're having third-party developed software for you in your

software development, then as we've said for 20 years, scrutinize that code that you get and validate that it's effective. If we're talking about the examples you said, which are, you know, full COTS products that are like, okay, well, does this thing have some backdoor into it? Then you might want to look at the providence and, you know, kind of see, is anything changing in the relationship, you know, such as the SolarWinds example, you know, or, you know, even in the, even going back, you know, 15 years now, has it been 15 years since Target? We just talk about that like it's yesterday. Because everybody's like, you know, the Target breach. And I'm like, what do you mean?

There's youngsters out there who don't even know what the Target breach is. So that it's kind of like, again, when I saw that, I'm like, well, of course the HVAC had access to it. Why wouldn't they? Because then no one would have thought to have not given them a service account with these access, you know,

the access controls as an entity. That's why I think that the Zero Trust for solar, for if you're a SolarWinds customer, Zero Trust is your best bet because you shouldn't be allowing the SolarWinds platform inside your network to have access to anything besides the things that it needs access to, right?

And that's what those bad guys did. They moved laterally once they were inside, right? And that's what those bad guys did. They moved laterally once they were inside, right? So that's why. So I would agree with that. And yeah, so I agree with that concept of, you know, isolate it, you know, to least privilege, to only those things that it needs to do, that should do, whether it's talking internally or externally.

You know, it should only be able to talk out to, if it needs to talk out to anything, then it only talks able to talk out to, if it needs to talk out to anything, then it only talks out to the thing that it needs to talk to and nothing else. And, you know, it's kind of a whole security architectural approach to it. So, yeah. So, I guess it's the thing, I guess I will say about zero trust is it's kind of like the expect that everything is compromised. Exactly. Which I don't think that, you know, unfortunately, a lot of my peer group just expect all the technology

to work. I was at an event one time and we were talking about ransomware and one of my CISO peers said, it's like, oh, don't worry about ransomware. I don't worry about ransomware. I have a good EDR.

Okay. And I'm like, wait, you're serious. Yeah. You know, so I think that there is a mentality of, hey, I bought all these tools and they work 100% of the time, so I'm okay. But if you're prudent, then you realize like, okay, what if it fails?

What's my way of catching it? I think it was one of your peers, it was one of my West Point friends, had said, I asked him 20 years ago, what did they teach you in War 101? And it was like, well, if you have something of value, put up a barrier and you monitor that barrier with power because at some point, minutes, hours, days, weeks, or years, it's going to be breached and you've got to be ready for it to happen. What I love about bringing subject matter experts to the Cyberwire hash table is that we definitely get a wide number of viewpoints about what is and what isn't working in the InfoSec community today. And speaking of the hash table, next up is my conversation with Tim Brown, the CISO for SolarWinds. Yes, that SolarWinds.

Tim Brown is currently the CISO for SolarWinds. He joined the company in 2017 as the VP of Security about three years before the SolarWinds attacks. After the incident, he got the CISO title. But he's had a long 30-year career wearing many hats. He's been an IT and security engineer, a strategy guy, a security architect, a cloud security CTO, a journalist.

He's even been a Dell Fellow. I started out by asking him, what was the thing that attracted him to InfoSec? Yeah, security is always changing, right? And there's always something new. And, you know, kind of I love the challenge of something new happening every day. And, you know, kind of I love the challenge of something new happening every day. And, you know, I sometimes get more than I ask for. But I really do like the challenge of everything new and changing. It's never boring, right?

We have plenty to do. I love that, too. It's the reason I love this field. But you know what I hate about it? It changes all the time. It is kind of a love-hate relationship. So on December 12, 2020, FireEye announced to the world that it had been compromised by what has become the poster child for supply chain attacks and the nightmare for all CISOs responding to a breach in our internal networks.

So, Tim, in broad terms, can you describe what the hackers behind the Nobelium attack campaign did? What was their process? Yeah, absolutely. You're one of the few people that had known the term Nobelium. So that was what Microsoft called the attackers, Nobelium. We also know them as the Russian SVR, also know them as APT29. So a number of different names for them. So, yeah, so FireEye ended up discovering that SolarWinds had shipped Hated Code

and contacted us on December 12th. December 13th, everything went public and, you know, went public and, you know, cumulated with us doing a, you know, issuing a notice, a 10Q, a 10K on early Monday morning. So attack was, you know, just as we said, right, we ended up the threat actor essentially compromised email first, did reconnaissance for a good period of time over a year doing reconnaissance. Then essentially did a test run in October where they inserted no code. Came back in February and inserted 2,500 lines of code, which was really sunburst code. That stayed in releases from March to June. And then they left so um attack against us was very specific very targeted very mission centric very quiet wasn't like they're in our environment making noise all the time their whole

model was how do we do it stealthily um and then the code that they dropped did things that made it hard to detect right so it wouldn't run inside of solarwinds domains wouldn't run inside of microsoft domain wouldn't run in a number of domain. Waited 14 days before it started. So very thoughtful, right, from an attack perspective. You know, I don't say it's innovative. I don't think it's like machine learning type stuff, but very, very thoughtful in the way they attacked us and others. So supply chain, the first time we called it a supply chain attack was that first day because they didn't attach our source code system where they would have been discovered. What they attacked was one of our transient virtual machines that's part of our build process.

That's where the insertion was. We couldn't find the source code anywhere. It wasn't in any of our systems. So it happened in our supply chain. It wasn't in any of our systems. So it happened in our supply chain. What came to be known very quickly, though, is where Stolens participated in the larger supply chains around the world and where we sat at our customers became very important. And that's what really sparked the big supply chain conversations now, simply because we were in the middle of supply chains and people didn't even know we were there. So having them discover that, you know, so as I say, you know, I had my Christmas

ruined, but a lot of IT folks around the world, you know, also had their, you know, Christmas ruined trying to figure out, hey, it's so low in tier, where is it? What's it doing? What version do we have? So quite a journey. Well, I agree with you. It wasn't like we've never heard of supply chain attacks before. They've been in the public sphere many times, but this seemed to be prominent notice because of how successful SolarWinds is, all the customers. Yeah. And it's a good thing, right? When you look at some good that comes out of it, it's the discussions around supply chains, the discussions of us, how do, get vendors more resilient, but also how do we figure out what's in the supply chain at our customers? And so those conversations starting is, you know, one of the kind of the good things that has come out of it.

Legislation is starting to come out of it. Executive order is coming out of it. So the progress is, you know, we hate to see something bad happen, but it's good that we see something good come out of it. So the progress is, you know, we hate to see something bad happen, but it's good that we see something good come out of it. So let's talk about that because you guys came up with some new strategy, all right, to handle security going forward. You call it secure by design, right? And you've come up with a number of tactics to implement it at SolarWinds. It's triple build, software development, zero trust, red teams, and security awareness training. Let's start with triple build software development process.

What is that? One of the most important things we had to do was reinstall confidence in our customer base. Part of doing that is saying, hey, we're not just as good as everybody. We're better. We took six months off from development of new features, and the full engineering staff focused on the secure-by-design efforts.

So part of those secure-by-design efforts were build processes. So with 12 products essentially making up Orion, we had multiple repos for information. We consolidated down to one repo. That was kind of stage one. Did a two-way build to start with. So we go from source code to product. We then take that product that's built, we install the product, we decompile it and get back to source code.

So we can do source code checks against the decompiled code C sharp. So by being able to do that, we get assurance that source code matched what was in the release. So second part of it was to take our build environment and move it to AWS and make it all ephemeral. So nothing exists, everything's in code, so nothing to attack. Third part was multiple build pipelines. And multiple build pipelines needed to be deterministic, right? You know that if you build something one time, you build it again a minute later, that you're not deterministic anymore because time changed. So we were able to make certain languages deterministic, Java, C Sharp. Having them deterministic means that I can do multiple builds and then compare them at the end and only ship

when those builds are compared. Why that's important is because my production build has two people having access to it. My staging build has about 30. My development build probably in the hundreds, right? So by comparing them, I end up saying, well, you would need collusion among multiple people. So those two people that are in charge of production don't have access to dev or staging. So in order to affect builds now, you'd have to have collusion among multiple people. Really taking an assumed breach process across the board and saying, okay, if Tim Brown has been breached, what would happen? Now, our incident didn't show insider, but very what could

be possible in the future, possible for somebody else. So it's important to think about that when you're doing builds and everywhere along your line, whether it's your SRE team, whether it's your build team, how do you take that assumed breach process and then add protections over the top? So the triple build software development process is really a protection against the insider threat to protect against outsiders inserting code like what happened to your incident. So it would protect against that

because they inserted into the build process. So GitHub did not match what the product was that was produced. So that insertion would get checked too. We plus did things on GitHub. So we always did peer reviews. Now we have peer reviews plus architect reviews. So additional reviews on anything that's checked in. Additional red teaming of the environments as well.

So the TripleBuild software development process protects against insiders and protects against outside breach infecting your source code because it's going to check the beginning stages of the code against what happens at the end and make sure they're the same. Correct. So much more resilience from any build process. And we wrote white papers on it and we've done things like that to try to bring it to the community. So it's been in place really since July, August of 21. So that just adds resilience. It adds assumed breach. It says a single individual won't be able to affect the builds, take the collusion.

Those types of things just really hardens that whole process. So let's talk about one of the other tactics. And it comes to mind immediately after you say all that, your red team tactic to go after this process, I'm assuming. You're sending your red teams to break that process. How does that work? We had a part-time red team before. Post-incident, we have a full-time red team. Now, the first object of my red team was both infrastructure testing, but then every component that is associated with the build system.

I trust certain things that they probably do well. Salesforce is fine. Palo Alto is fine, as you know. They're fine, but what about my configuration of them? Is my configuration where it needs to be? My red team's focused on, can I take advantage of the configuration that you have in place to circumvent security controls? So as opposed to always looking for domain admin, these guys are looking to say, okay, when I look at my firewall, all right, who has access? What's change control look like? How are my checks and balances there?

What is my configuration there? GitHub, same idea. How many admins do I have? What if I take over one admin? Can he affect something else? So do I have protections in place from that? So on the SaaS services, it's often looking at, yeah, okay, I've got a well-configured SaaS service

or a well-configured product. And is that configuration resilient to attack? So we've done that for the components of essentially the build system and always looking to tighten things down a little bit, right? So Red Team hasn't gotten through, but they've tightened stuff down. They said, okay, if this was here, we could have done this. So it's a double check. So the red teams are very important to us. Now we're starting on mission and business critical assets and those applications, and we'll spin back around and

redo essentially build system. So is the red team in this new strategy, Tim, is that purely for the software development process or do you turn them loose on the network configuration also? Yeah, I turn them loose on the network. I turn them loose on the infrastructure as well. Are they dipping their toe into adversary emulation? Are they trying to duplicate what Nobelium did? Are they trying to duplicate what Nobelium did? Yeah, in some ways, the Nobelium guys were coming into our environment, absolutely. Secondary attacks, not so much. Secondary attacks were very specific. Orion had to be public-facing. You had to talk to a command and control server.

You had to be able to be somebody that the folks cared about. So that's one of the reasons why we saw our number of truly affected customers. Now, everybody was affected by looking and saying, am I running one of the versions which is tainted? But when we look at the number of affected versions, it's actually not 18,000 which downloaded a piece of software. It's under 100 that went to a second stage used to say nine agencies gao report came out and said four agencies went to a secondary target so had to be very specific so we don't necessarily test that scenario but we do test the scenarios of them you know potentially entering our environment making sure that the environment itself is as resilient to attack from the outside.

The other benefit of the red team, it tests my SOC, tests my back-end systems and my recovery and my playbooks. All of those things get tested. So the third tactic you guys have launched in your Secure by Design strategy is zero trust, something on everybody's mind these days. So on a journey, right? I think zero trust is a journey, right? So part of our journey is to move the authentication authorization out to the edges, right? In a software-defined perimeter kind of idea?

You consider that we've got so many cloud properties, so many things that we've been moving towards from a cloud perspective. Office 365, Salesforce, you name it, are moving to essentially the cloud-based solution. So the more centralized we get authentication authorization, that's been a big push to use Azure SSO, conditional access. Now, YubiKey for authentication authorization for all admins, all dev folks, and moving YubiKey out to the world.

Because just because of the MFA overload that we've seen, Yeah, that's got people concerned, right? That, you know, are you going to get them? I'm worried about UbiKeys too, because we use them here at the CyberWire. But, you know, if you're like me, I'm going to lose that UbiKey. Right. So if you just keep it in your machine,

it's the little ones. We keep it in the machine. It works pretty well. Everybody I talk to on this show, Tim is talking about how they're doing Zero Trust. You're a long way down that journey. Is there any one lesson learned that you wish you knew before you started that would have helped you in this that you can part to all of our listeners here? Yeah, I think just definitely start.

Start your process. Get from one app to two apps to three apps to four apps i think we're up to um i can't remember what my numbers are but it's in the um above 50 right so but start right start down the process because the benefit of the benefits of having kind of central law the benefits of central auditing the benefits of having central control are worth their weight in gold. So start the process. Don't be afraid that you have to have it all right or all done. Because get to 50%, get to 20%.

It's better than you were yesterday. Yeah. Like you said at the top, it's a journey. We're not going to get there. We're not going to get to the end anytime soon. So the last tactic in your Secure by Design process is security awareness training. And it's a little different than most people think about it, I think, because of the software pipeline that we've been talking about. So what's involved in that part of that tactic?

Yeah, really, when you look at, you know, kind of culture of security, right? How do we keep imbibing culture of security across the company um so some of those are having additional places to communicate security topics to people so you get your annual training that's normal so we'll do newsletters we'll do topical things out to people um we'll also, we've been doing a fishing campaigns pretty consistently for, I think the last six, seven quarters. Right. And when somebody clicks on something, they get remedial training. So we're getting towards the end of this, Tim, any last words for the audience about your experience after the incident and what can you recommend for everybody? Yeah. So, you know so what worked for us is really making a focus on our customers. That was our key focus, was how do we understand

and help the customers get through it? And it took time. It was hard. It took a lot of effort. But the customers responded. We helped them move forward. We helped them get into the right place. And then be exemplary. So where can you be exemplary? Where can you do things that are not just okay, but really exemplary in process? We'd like to thank our interview guests, Tim Brown, the CISO for SolarWinds, and Rick Doughton, the CISO for Healthcare Enterprises in Centene. And special thanks to Don Capelli from Dragos and Steve Winterfield from Akamai for helping us get some clarity about where the InfoSec community stands with software supply chain management issues.

And finally, we'd like to thank SolarWinds for sponsoring the show. sponsoring the show. CyberWireX is a production of the CyberWire and is proudly produced in Maryland at the startup studios of DataTribe, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Iben, our executive editor is Peter Kilby, and I'm Rick Howard. Thanks for listening.