CyberWire Daily - The great overcorrection: shifting left probably left you vulnerable. Here’s how you can make it right. [CyberWire-X]

Episode Date: July 24, 2022

Shifting left has been a buzzword in the application security space for several years now, and with good reason – making security an integral part of development is the only practical approach for modern agile workflows. But in their drive to build security testing into development as early as possible, many organizations are neglecting application security in later phases and losing sight of the big picture. In this episode of CyberWire-X, the CyberWire's CSO, Chief Analyst, and Senior Fellow, Rick Howard, talks with two Hash Table members, Centene’s VP and CISO for Healthcare Enterprises, Rick Doten, and Akamai’s Advisory CISO, Steve Winterfeld. In the second half of the show, CyberWire podcast host Dave Bittner talks with our episode sponsor Invicti’s Chief Product Officer, Sonali Shah. They discuss the challenges and misunderstandings around shifting left, and provide tips on how organizations can implement web application security program without tradeoffs throughout the whole application security lifecycle. Learn more about your ad choices. Visit megaphone.fm/adchoices

You're listening to the Cyber Wire Network, powered by N2K. Hey, everyone. Welcome to Cyber Wire X, a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the Chief Security Officer, Chief Analyst, and Senior Fellow at the Cyber Wire. In today's episode, we are talking about shifting left. A program note, each Cyber Wire X special features two segments.

In the first part, we'll hear from an industry expert on the topic at hand. And in the second part, we'll hear from our show's sponsor from their point of view. And since I brought it up, here's a word from today's sponsor, Invicti. Your organization is building and updating business-critical web applications faster than ever. And with so much pressure to move fast, you may find yourself making trade-offs between innovation and security. Now you can build fast without sacrifice and security with Invicti, the application security platform that helps your dev, sec, and ops teams work together to secure every website, web app, and API.

With unparalleled accuracy, coverage, and automation, Invicti scales like no other app sec solution. Discover why many of the world's largest organizations innovate securely with Invicti. And we thank Invicti for sponsoring our show. I'm joined by Rick Doughton, the VP and CISO for Healthcare Enterprises and Centene. Rick, thanks for coming on the show. Thanks, Rick. Happy to be here.

Today, we're talking about application security testing. And what I mean by that is extending our zero-trust strategy beyond the traditional network defense for our data islands, like data centers and cloud deployments and mobile devices, and include reducing the attack surface for the software we use. You know, the code we write ourselves, the code that we get from open-source libraries, and the code we run from vendors like Microsoft, Google, and Amazon. Now, you and I have been doing this stuff for many years, and application security testing has always been around. But until recently, say the last five years, I wouldn't say that many CISOs have had this on the top of their priority queue. But you talk to a lot of security executives in your day-to-day job. Is that changing these days? Do CISOs own the responsibility? Or is it a shared responsibility with the CIO's DevOps teams or some combination of all that?

Yeah, it really depends on the organization. If your organization is a software development, I mean, you create software products, then the development team usually has more power and they manage their pipeline. If you are a large organization like ours who does a lot of development, then the IT security organization that's a very large organization kind of oversees the security aspects of it, many of which may be applied and used by the application development teams.

But then because most application development is cloud nowadays, the cloud team is also involved as well. So whereas before, like 20 years ago, when we were doing application pen testing, you know, we deal with the developers, security would oversee the testing and the QA process, but usually didn't have anything part of the process until the pen testing done at the end. Then we started evolving into training the developers and in the early 2000s started integrating security in the application lifecycle. And then today, as you kind of referenced,

like we have a CICD pipeline incorporating into DevOps that the development teams have been using and integrating security in a DevOps lifecycle to be DevSecOps is a relatively new thing. So this is kind of a matrix organization here, right? It doesn't feel like anybody's really in charge of it. I guess the CIO would typically be in charge of the CICD pipeline, but there's all kinds of groups in there, right,

influencing how that is built and maintained and things. Is that how it's done today? Yeah, and it really depends on the organization, like I said, and who has the most influence. And so some organizations, like the developers, may be at the bottom of the chain. And there's application architects who dictate to them what tools and approaches and architecture they're going to be using. There may be cloud security, the cloud team that dictates the cloud infrastructure they're going to be using. And then there'll be security teams that dictate the security requirements for that.

And so when we start adding security into the application lifecycle, it's a whole stack. We have threat modeling, we have static code analysis, we have dynamic analysis, we have testing of the infrastructure and the infrastructure as code. You had mentioned before third-party libraries,

we have dev and production environments. And so a lot of times you also have secrets management. You have data protection. You have APIs. There's lots of pieces that are involved. So yes, it definitely becomes a team sport. I was talking to my best friend, Steve Winterfeld, about this. He's the advisory CISO for Akamai.

He says that who owns the application security function is largely based on the existing business process. So it's interesting when you talk about testing CISO versus CIO, really ultimately, you know, you have two testing that I think about quality and security. And I think those testing should be done collaboratively and somewhat based on culture. Ultimately, the CISO has to provide the tools, the resources, be they hooks or tools that they use. Often the CISO is going to do some of the monitoring and alerting. You have to build a collaborative process. And whether it's security champions within that group or whether you're putting your staff in with the right skill sets to collaborate in there. I think really that's kind of where it is. Ultimately, it's a business process, not a standard answer.

Rick, you mentioned at the top of the show that there are two kinds of application testing, static and dynamic. So let's start with static. What is it? How do you do it? And how do you know it's working correctly? This has been around for 20 years or so, and we used to call it just code reviews. And we would have it where we talk about dynamic analysis is running a tool of an application that's already in place. And it could be somewhat automated where you are just crawling through the directory structure, you're testing certain commands to see if there's SQL injection cross-site scripting, tools running, and it comes back. That's also been around for 20 years. The next step above that is like active penetration testing, where you get some valid accounts and you

try to manipulate things and you work on things. And, you know, I spent the early part of my career doing that and managing people who did that. So the static code analysis, we used to kind of call like, that's the deep dive. We're like, I'm looking at what that code actually does. And there are tools that you just kind of dump a code sample in, and it goes through and looks for a lot of those standard OWASP top 10 type of things for releasing buffers, releasing cookies,

cross-site scripting, cross-site request forgery, looking for those kinds of things. As a human or even like a crawler doing an automated dynamic testing, we might miss something because we might be a part of the application we didn't touch because it was behind the scenes or something that we didn't actually interact with. And then a static code analysis may actually see that because we didn't interact with it. Nowadays, it's become more common. The static stuff in my mind, you tell me if I got this wrong, is things that we already know about.

Like you mentioned the OWASP top 10. And any kind of regression testing, mistakes we've made in the past, we don't want to remake them again in the new code. So that collection of things, and like you said, we can run tools to see if our existing code base has those mistakes. And so that would be static. It's not a live attack of the code. Am I saying that correctly? Yes, absolutely. That's a good way of putting it. It's like

looking at the code directly for things that we already know are vulnerabilities. Right. And so then you mentioned dynamic testing. That's the way you described it is it's like hackers going after running code. Is that correct? Yeah. And nowadays it's like some of the same tools that we would use for network testing, penetration testing, or vulnerability scanning. But they've kind of evolved to the application layer. Yeah, so it is more human-driven, and it's interactive.

There are dozens of stories from years ago, but for instance, one of my favorite ones was you go in as a user who doesn't have admin access or access to certain database to get certain data but there might be a widget that is used to be able to query something in another database because i'm in there i'm imitating that widget to go access the database and i can pour more information as that service account than what i have that has greater access than i do as a user with static code analysis is usually done against specific modules,

and so the combination of those things may not be seen as static, but it would be in a dynamic. The extension now to the static is infrastructure as code. And that's kind of the new thing in the last few years as we've done cloud application,

then we're spinning up temporary workloads and the code will say, okay, I'm going to spin up this container. It's going to install this application. It's going to run for five minutes and then it's going to shut down and I'm going to move on. And so now we have a whole new layer of looking at that, not just the things we know about, but the infrastructure of code, which is not unlike doing an on-site network scan of the operating system of the device that the server that's running on. So it's like you said, the static testing, it's just a list of things we're going to check, but

dynamic testing is really, like you said earlier, it's a red teaming exercise directed specifically at the software that we run, not necessarily against the entire network, but very focused on that. And so that makes it different. But there are tools that will help people that are worried about this do this themselves as opposed to having, you know, world-class op four going against your networks, right? Oh yeah, absolutely.

And the tools have gotten a lot better over the years. And some of them are still have been around for 15 years. But yeah, so they will do that in an automated way. And then there's also tools that help you, assist you if you want to do it in a more hacker way and get in the middle of the proxy and adjust things in the fly

and stuff like that. Well, I think the interesting thing that's changed in the last five years or so is our community becoming aware of third-party code being used in our environments. All the supply chain attacks that we've seen in the last couple of years has really made that issue height on our radar.

So that's a whole different thing, right? Because I don't think we were even talking about open source modules over five years ago, but now it's a thing that we have to worry about. Well, I mean, I guess it's at a broader scale. I mean, even 20 years ago, we were looking at, because most applications are 90% other people's code. And there's lots of open source libraries that are used. There's standard modules that are being used for encryption. There are libraries used for managing identities and maintaining session state.

There are libraries used for managing identities and maintaining session state. And so back in the day, when we were really doing application pin testing, we would look at all of those as well. And those would be things that we'd probably ask to do a code scan of, of these third parties. Now, because it's a lot more prevalent, and there are tools that are kind of like we're built out of doing this third-party code library analysis in the repository. So in GitLab or GitHub, you know, there's like, okay, when I'm maintaining these things, I'm kind of making sure that the library that you're going to call that's a third-party library is the one, A, the version of the one that's already been tested and it's cool. So, yeah. And then as we're now moving more towards, you know, the software bill of materials,

then that becomes a play. So, yes, it's happily been getting the attention that it's never really had unless you are a very mature application security team. And Steve says, let's not forget about the right side as we are all shifting left. So really, I think of this as kind of shield to the right, shift to the left. You know, and as we shield to the right, we're protecting APIs, JSON, XMLs. Obviously, you're talking about the right tools here. Static, dynamic, interactive. I think they're all useful. I think we need to expand beyond that. Some of the traditional things, pen testing probably have a part in this framework. I think you need to expand this into your JavaScript environment

and you need mobile testing as well. So I think these are the right questions to ask, but you need a holistic set of tools here. But ultimately, I think if you're not doing these, you're just pushing vulnerabilities forward. So Rick, you've been doing the static and dynamic analysis for a while now, your current job. Is there any tips that you can pass on to our listeners about any pitfalls they need to worry about to steer away from? I guess it's like, make sure that everyone knows their role, you know, because it's a team environment. And make sure that the developers are educated and have the tools, if you're coming from the CISO perspective. And also come up with a process, you know, because the real challenge is that crossover with infrastructure of code is like, if a flaw is

found in a piece of infrastructure, did it get there because of the code or was it there because of some immutable image that was used to pull from, you know, whose fault is it for lack of better word, because that will change the workflow to fix it. And so understanding these different scenarios where, okay, this was put in through infrastructure of code, this is a bug fix for the developers need to fix, or this is something that's infrastructure was already put up by the cloud team, they will use another process to fix that.

And then also go back to root cause. So the number one thing I would say is like, find the root cause of how it got there and fix the root cause instead of just kind of keep patching these containers. And then every time they get refreshed, the new vulnerability gets put in there again. And then figure out like who, and this, I say this a lot when I talk to cloud and application security is who has influence and, and, you know, power as another word. And so like, and sometimes we may need to start giving back to the developers some influence

because we don't want to just wait until the thing is running live, kind of like we did pen testing 20 years ago after it was about to go live, and then we find a problem and we go back and have to fix it and push out the release dates. But instead of finding it during a vulnerability scan or a dynamic scan of an environment of the infrastructure,

and then sending it back to developer, have it to where you're checking it earlier on. And so that you're checking the infrastructure as code in the code to see if it's pulling the right image, if it's configuring it the right way, and securely, instead of having to wait. And so that will kind of speed up that cycle. Well, all those problems you say to concern yourselves with, I think I just, from my perspective, I would like to remind the audience, just think about agile development

and think about DevOps and the problems that those methods try to solve. They try to break down those walls between developers and operators and security people. You know, instead of throwing code over the fence and say, okay, I'm done with it until it comes back to me. It's more of a system of systems. And if you can run your organization that way, where it's one big system of systems that everybody is paying attention to, as opposed to stovepiping different work efforts, I think that would move your organization forward more quickly and more efficiently. I think that's great advice.

move your organization forward more quickly and more efficiently. I think that's great advice. And I like to say that DevOps is really like Agile Scrum or XP, but automated. So it's the automation of it. And that's what we're trying to get to is more automation, more automation and radiation, more automation and release. Because that's what DevOps is and that's what DevSecOps is. And so if we don't have a good process in practice, automating is just going to mess it up. You don't automate bad things.

You automate good things. So make sure you have a good process and then go to automation. I've been doing it wrong all this time. This is all good stuff, Rick, but we're going to have to leave it there. That's Rick Doden, the VP and CISO for Healthcare Enterprises at Centene. Thanks for coming on the show, Rick. It was great talking to you. Thanks, Rick. Always a pleasure. Next up is my colleague Dave Bittner's interview with Sonali Shah, the Chief Product Officer at Invicti, our show's sponsor.

What's the background on when people in security started talking about shifting left at all? How did this notion take hold? The shift left movement became popular at the same time DevOps became popular and now DevSecOps. And the idea of shifting security left is that if you are able to find things earlier in the software development lifecycle, it is easier, faster, cheaper to fix. And as we release codes, if you think about DevOps, the idea there is efficiency and being able to increase the frequency of code releases and doing it kind of in batches and continuously improving your product. And if you wait and do security at the very end,

well then you're not going to be able to do those rapid iterations because you're going to have to take time out, well, then you're not going to be able to do those rapid iterations because you're going to have to take time out, scan the application, find and fix, and then release. So the idea is that if you do security early on, you'll be able to catch it and it doesn't slow down the release. And so when people generally come at this, when they decide, okay, we're going to adopt a policy

that involves shifting left? Typically, what would that look like? So one thing I just want to start out by saying is that shifting left doesn't mean that you don't shift right, right? So the idea of DevSecOps is that security is built in to the entire lifecycle of the application. So from, you know, you're testing in development before you release code, but then you're also continuously testing in production because we all know two things. One is applications that are secure today may not be secure tomorrow, right? You find a new zero vulnerabilities discovered, suddenly now something that's in production that was safe when you released it is no longer safe. The second thing is that it is really impossible for a company to say, all right, I'm going to find and fix everything before I release. There was a survey recently that said that the majority of developers are releasing code that they know is insecure.

And it's not because they want to, it's just that they don't have the time or the skills to fix it. So when we talk about shifting left, that doesn't mean abandoning the right. It means, really, it's doing both. When you think about dev second ops, it's about doing security early and often,

but also doing it continuously while applications are live. Well, and certainly that all sounds like a good thing, but what are some of the places where people find themselves being tripped up by this? Well, the way to shift left effectively is to automate security into the development lifecycle. So where you get tripped up is if security is a separate stage and you do security and then it goes back separately to the developer. The way to do this effectively is to automate security. And what I mean by that is automate the testing so that, you know, within the CICD pipeline, just as you're building the code, you're also testing and results should automatically go back to the developers in whatever tool it is they use, right? Whether it's Jira or

ServiceNow or whatever it may be. And where people get tripped up is if they don't do the automation, then it just becomes too cumbersome for developers to have to then go to a separate tool and run scans and get the results. So I would say that is one area. Another area is make sure that the tools you're using are accurate and actionable. Because even if the tool you're using is fully integrated and automated, if the results they're giving you are not accurate, it's really garbage

in, garbage out. And you're only going to frustrate the developers and end up wasting their time looking at a lot of false positives. So I think it's really important that you make sure that the tool you're using is producing accurate results that are actionable, as well as having that your security tool automated and embedded in the whole software development lifecycle. Yeah, I mean, that's a really interesting notion. And I guess it makes me wonder, how does an organization dial it in, you know, that balance between the security that they want, but also not introducing too much friction to the process? that they want, but also not introducing too much friction to the process.

Yeah, no, that's a great point. And look, every customer I talk to has friction. No one has solved that problem. But there are companies that are really struggling, but there's companies that have made a lot of great progress. I don't know if you were at RSA last week, but I spoke with Ian Myers, who's head of application security at Marriott Vacations. And he talked a bit about how he's been able to improve the relations between developers and security teams and reduce the friction. He talked a lot about having developers help with deciding what are the security tools you're going to use and how are you going to weave them in so you get their buy-in early on. You know, he also talked about security champions programs. I was actually recently visiting a large global bank in Paris and I heard from our customer there how they've been really successful with their application

security champions program and how that has really helped. And what was interesting, what he told me is that when he asked his development teams who would like to be in this application security champions program, he was really surprised at how many developers were interested. Because he's like, you know, they really do understand the importance of security. They take pride in their code. And by the way, you know, having something security related on your resume or your LinkedIn profile is actually, you know, quite a marketable thing for them,

for developers to have. So, you know, there, the story that I heard from him as well as from Ian and other customers is that that building a security champions program is just another way to help reduce that tension. How do you measure success with something like this? When you've implemented it, it seems as though everything's running well. What are some of the metrics by which you know if you're doing the right thing?

I think it's like anything else, that when you're going through a transformation, start small, get the buy-in, prove success, and then expand. So we actually help our customers roll out their application security programs. And what we always recommend is, if you have a thousand applications, let's start with one or two groups. Let's start with a couple hundred, and let's do the automation. Let's do the integration. Let's prove success there.

Show that you can build security into your tool chain, that you can automate it. And really, the key thing is about risk reduction. So some of the metrics you can look at is number of applications scanned, number of vulnerabilities found, but really the metric here is risk reduction. So the measure of success is how many vulnerabilities and of what severity are being released in production. And that is your risk reduction. And then of course, there's also the efficiency metrics at what cost did you do this? And those are the two things, but I would always suggest our customers

that they start with the subset of their organization, prove it out and then expand. But the ultimate metric is always risk reduction and then at what cost. What are some of the stories that you hear back from folks who have successfully implemented this sort of thing? What's their experience like on the other side of it? Yeah, it's a journey. No one gets there immediately. It takes time and there's steps forward and backwards. Another large global customer that we deal with, they actually had all their developers come together

for a two-day virtual off-site and had security training as part of that and we were sharing some of the stories. I think what good looks like is when security, it's like air. It's just something you breathe. You don't think about it. You know, when you develop code, right, you want to develop quality code that does what is expected, right? That does something that meets a customer's need. And what success looks like

is when security is built into that. So you can't possibly deliver a new feature that meets the customer's needs if it's not secure, because that has to be part of the criteria. And when security is just built into it and it becomes part of the developer's job, that's what success looks like. that's what success looks like. We'd like to thank Rick Doughton, the VP and CISO for Healthcare Enterprises at Centene, Steve Winterfeld, the Advisory CISO for Akamai, and Sonali Shah, the Chief Product Officer at Invicti, for providing some clarity to us around the idea of shifting lab. CyberWireX is a production of the CyberWire and is proudly produced in Maryland at the startup studios of DataTribe, where they are co-building the next generation

of cybersecurity startups and technologies. Our senior producer is Jennifer Eidman. Our executive editor is Peter Kilpie. And on behalf of my colleague, Dave Bittner, this is Rick Howard signing off. Thanks for listening.