Screaming in the Cloud - Viewing Security through an Operational Lens with Jess Dodson

Episode Date: April 13, 2023

Jess Dodson, Senior Cloud Solution Architect at Microsoft, joins Corey on Screaming in the Cloud to discuss all things security. Corey and Jess discuss the phenomenon of companies that only care about security when reacting to a breach, and Jess highlights how important it is to have both a reactive and a proactive approach to security. Jess also shares her thoughts on why it’s valuable to get security and operations working well together, and why getting the basics right in security is still a more pressing priority than solving for level 10 security threats. Jess and Corey also reveal best practices when it comes to monitoring and revoking admin rights and much more. About JessChances are if you’ve run into “GirlGerms” online, you’ve spoken to Jess! Based in Brisbane, Jess joined Microsoft in 2019 and is now a Senior Cloud Solution in Cyber Security, after working in a mixture of both government and higher education industries for over 15 years. Jess regards herself as a 'recovering systems administrator' and still wears her operations hat when looking at security - doing REAL SecOps!Outside of work, Jess is mum to a 5 year old daughter, a cat, 4 chickens and a hive of bees. In her downtime, she spends far too many hours building Lego, playing video games or doing random crafty projects.Links Referenced:Twitter: https://twitter.com/girlgermsMastodon:https://infosec.exchange/@girlgermsDevNxt: https://devnxt.nz/

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably?

With SIEM, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be set up in minutes. By automating the access request lifecycle, SIEM helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with SIEM. Go to SIEMOPS.com slash COREY to learn more. That's S-Y-M-O-P-S dot com slash Corey. Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is Jess Dodson, who's a senior cloud solution architect at Microsoft. Jess, thank you for joining me. We have been passing like ships in the night on

social media for years now. It is so good to finally talk to you. Lovely to talk to you in person. Thank you for inviting me on. Well, to be clear, we're talking remotely when we record this. You are presumably Australian, and I'm still operating from a somewhat American-centric viewpoint that more or less everything in Australia is deeply poisonous. That includes me, yes. So I am in Australia at the moment. I believe it is the 9th of March for you. It is the 10th of March for me. Ah, yes. Some of us are living in the future. The future! So yes, it's about seven o'clock in the morning for me, which is fabulous. I'm awake. I'm awake. So let's talk about security.

It seems to be top of mind and everyone's talking about it. Unfortunately, it seems that they're usually talking about it in the form of an email that starts with your security is extremely important to us and then transitions into here's how we drop the ball on it. I was once told by a analyst client of mine that I was the only analyst who ever told them that companies don't care about security. No one says that. Why is that? And my answer was, well, no one will say it out loud, but I ignore what people say. I pay attention to what they do and where they spend the money, and it is clearly not a priority. And I would argue that in some ways, that's okay, depending upon context and who you are and what you do.

And in other cases, it is the exact opposite of okay. It is an unmitigated disaster for everyone just waiting to happen. How do you feel about security? Very loaded question. Isn't it, though? I don't care. It's probably not going to be your answer on this one. I'm just going to assume.

But let's go. Well, considering my title, I would hope not, considering the work that I do. You have words in your title, but none of them are cares about. So we're good. That's true. That's true. I do care about security and that's my job. I do agree that organizations probably don't care about security until they do care about security.

And by that point, it's probably too late. And that's the issue that we're facing. It's that proactive versus reactive. Reactive is great. Reactive is wonderful. But unless you're doing the proactive and spending the money on the proactive, the reactive is just constantly going to be fighting fires. It's two classes of problems. My world of cost optimization absolutely is in this world as well. Security is. Buying fire insurance in case the building burns down. These are all things you should probably do, but you can spend infinite money and effort and time on all of those things, and it doesn't get you one iota closer to your business goals. Whereas speeding time to market, launching into new markets, and being able to ship faster, that is transformative.

That gets you to your next business milestone. So companies will overwhelmingly bias towards investing in that and not the other stuff until they're caught flat-footed. And I'm sure that's wonderful, but my issue when I come in as a security person to an organization is, what if something goes wrong? And it's no longer, and I hate using buzzwords, but when we're thinking about zero trust, one of the principles around zero trust is assuming breach, which is, it's not a matter of if, but when, and it's not a matter of when, but it's happening right now. Because as blue teamers, which is what I think of myself am, we have to plug all of the holes. We've got to try and patch all of the

defenses. We've got to be the ones who are blocking out every attempt. An attacker only has to succeed once. And so the way that we see it is there is going to be something in your environment. So what are you doing to make sure that that is contained as much as possible? And that's proactive work. And that's something I don't think you can skimp on. It is, and it's defense in depth. You can also turn it into business value if you're clever enough. For example, whenever a cloud provider releases a new service that I can't figure out how to configure, all I do is I wind up scoping a security role to adjust that service and then leaking credentials online. I wait, you know, 20 minutes for someone to exploit it and then configure the thing, presumably to mine Bitcoin.

Great. Then I turn off the Bitcoin stuff and I take the config that they've built and there we go. It's how we outsource intelligently on a budget. Professional advice, please don't do that. I was going to say, that's certainly a unique way of configuring services in the cloud. Not sure I would recommend it for everyone, but for you, I can totally see that working. Yeah, I learned it from my buddy. He works at a bank. Yeah. It doesn't go well. When it comes to all of the stuff, and I think that's probably one of the big issues that we have with the cloud. And I love the fact that the title of this is Screaming in the Cloud, because when we're looking at all of the stuff that keeps coming out, everything is changing

so quickly. How do you secure it when you don't know from one week to the next what new services are going to be included, what changes are going to be made to the services that you are already currently using? How do you keep up to date with that? And I think that is what leads to security being seen as the no people, which I hate. Security shouldn't be no. Security should be yes, but. It also leads to hopefully our operations teams being a little bit more on the ball when it comes to that security, because if they're the ones who are looking at putting these new services or new productivity features in place, they should be vetting them from a security perspective as well.

I say should, maybe not necessarily actually happening. And that's a bridge we kind of need to cross at the moment. A couple of years ago, I looked around the ecosystem trying to find a weekly AWS-centric security newsletter. And there are some great ones now, don't get me wrong. And some of them might have existed at the time, but I didn't trip over them for one reason or US centric security newsletter. And there are some great ones now, don't get me wrong. And they, some of them might've existed at the time, but I didn't trip over them for one reason or another. And they tended to all fall into two buckets. Either they were security people talking to security people with a bunch of acronyms that I wasn't tracking because my, I don't eat, sleep, and breathe security most days. And, or they were vendor captured and everything was, see how

terrible it is. That's why you should buy our widget. It almost doesn't matter who the vendor is. So I started a Thursday issue of the newsletter that I write just for the news that is security centric for people who don't have the word security in their job title. So all the DevOps people of the world, the folks who are building applications, the folks who do have to care about this, but they don't have time to filter through all of the noise that everyone's putting out. And what is the stuff I should actually be paying attention to this week? And that seems to have struck a nerve on some level. The thing I'm continually testing and being pleasantly slash unpleasantly surprised about is that I'm rarely the only person who has a particular problem. And you're definitely not in that regard. And I think when we look at security

and how security is marketed, it is very pushed towards SISOs and security analysts and security operations. I, most of my life, and I still regard myself as a recovering systems administrator. I'm a SIS admin at heart. So I come from an operations background. The work that I've done in operations is what feeds the work that I do in security because I knew how it worked. I helped build those systems and I don't see it purely through a security lens. I see it through an operations lens. Security is useless without some form of usability. And I constantly talk about the line that divides security and usability and where that line gets drawn. And for each organization, it's going to be different depending on their risk, depending on their business profile, what they're actually doing, how big their teams are.

And we've got to make sure that we get that line right. And that means getting your operations teams involved because they know how their stuff has to work. They know what they need to be able to make the business run. So security can't just be constantly saying, no, you can't do that. We have to be working with operations teams and in some cases, taking a lead from operations teams because at the end of the day, they know this stuff better than we do. So it's us providing advice and them providing their expertise for us to come together to a joint agreement on how to secure the thing.

And I don't think that's being done at the moment. I call it the operations sandpit. Everyone needs to play nicely in the sandpit. No one should be fighting over toys. Everyone should be playing with each other with no arguments that we're not there yet. When I read a lot of security researchers and the stuff that they're focusing on in stupendous depth, or I walk around expo halls at security conferences making fun of the various vendors there, it seems like so many of them are talking about these incredibly intricate, in-depth defenses against attacks that seem relatively esoteric. And then I fly home from

the security conference that I'm at, and I happened in the airport lounge to see their CEO, who's yelling into a phone, and I know that they're using the same password for work that they are for their personal stuff. And it's kiddy. And I know this because the Post-it note on their laptop says it. And it feels like, yeah, you're selling and solving things for problems at like level 10 for complexity,

but you need to effectively handle the problems at level two first and do the easy stuff and the basics before you start getting into the world of imagining that the Mossad is coming for you. And it's not even level two. I'd say we're almost, we're at level zero for a lot of this stuff. And you're 100% right. we'll be talking up all of these heavy duty, this is what we can do to protect you without even recognising the elephant in the room of you're not even using MFA. You're not even looking at securing your administrative accounts. You're not using separate administrative accounts

and user accounts. So they are the same person and log in to everything and can access the internet while still performing full administrative work using that account, which is just terrifying. I still get constant notifications of security alerts from various vendors when there's a new TLS policy that should be applied to the load balancer that I'm using with them. And it's, you really think that people decrypting TLS encrypted traffic somewhere between a user and my web server for a website that has the actual term shitposting in its domain is my number one priority on these things.

It just feels like it's such this, it's this incredible cacophony of noise. And then you completely miss the next email that comes in. Oh, by the way, you put your credentials up on GitHub. Maybe do something about that. It just becomes this entire walking modern version of a Nessus report with the 7,000 things you need to fix. And number 3,768 is, oh, by the way, the kitchen's on fire. It hides this stuff and it's awful. The prioritization of what can be done to

fix some of the security holes that we see, it's out of whack, I think is the polite term to use. It's absolutely awful because we are prioritizing things that, yes, they are considered high severity in terms of what could happen, but in probability and likelihood, really, really low. Whereas things like having your password stored on a post-it note that you have taped to your laptop that you then carry around and everyone can see it and they know who you are, maybe that's a little bit more concerning. So it's trying to, from a security basics perspective, which I've been, I want to say talking about, but it's trying to from a security basics perspective which i've been i want to say talking about but it's not it's ranting about and screaming about since about 2018 it's about

making sure that we get those basics right because without the basics all you're doing is putting band-aids over giant holes and giant leaks that anyone can slip through, and it's a waste of time. So making sure that we are doing those really small basic things that, let's be, we've been talking about them for years. I think I've been screaming about MFA for at least 10 years now. We've had this available to us as an option. Why are we not doing it? One of the statistics that I saw recently inside a cloud provider, only 33% of accounts are protected with MFA. 33%. What's happening with the other 66? Because that's just, that's a staggering number of accounts that are, I would consider, unsecure. On some level, you almost put this back on the companies themselves.

Our timing is apt on this. Earlier today, Jithub, which is, yes, how I pronounce it, announced that starting in a couple of weeks, they are going to require MFA for every contributor on Jithub, or GitHub, if you want to use the old school pronunciation. And I think that's great. Everyone is used to MFA on some level. Mandating it for accounts for which the blast radius is significant goes a long way. And yes, down the road, longer term, passkeys might make this a

lot easier or not as necessary, and or better methods of authenticating against an MFA because typing in six-digit codes is annoying and goes out of bounds on these things. But I don't think it's necessarily the end of the world to make the sensitive accounts just a little bit harder to access. I understand that for some folks, MFA is that it's that next step. It's that next barrier. It's another thing that it's that next step. It's that next barrier.

It's another thing that they then have to do. It sounds terrible finding what I call silver linings in some of the events that have been happening recently. But if anything, particularly here where I am in my very small, tiny neck of the woods in Australia, some of the recent breaches that we have seen over here have made not just operations and technical people, but end users, consumers, more aware of the fact that it only takes one slip up. It only takes one small thing for something bad to happen and anything that they can do to protect themselves is a good thing.

So we're starting to see a bit of a shift towards an acceptance for some of those trickier things like MFA. It's a pain in the bum. It's not something that is what I would consider user experience friendly. We're getting better at it, but it's still an issue when it comes to signing in and having to find your phone which most of the time you have no idea where it is you've got to use whatever automation you have to call your phone to find it to be able to find your code so you can actually log in it is a pain but anything that we can do to be able to protect even our personal accounts is something that I consider to be really important.

And it's not just for organisations. I've got to admit, I'm still working on my parents. They are of that generation where maybe it's still a little bit of a step too far. But just trying to get people that I know who aren't technical to start looking at things like this, because as we move forward, it's going to become the norm. And I'd much rather people become comfortable with it now than later when it is forced upon them. One of the problems that I've got is we just went through our annual security awareness training that we are contractually obligated to provide to everyone. And all the vendors that do this, I have problems with it in different ways and different degrees.

It's all terrible on some level. And it's not that these vendors themselves are bad, but it's the state of security for most folks. One thing that gets hammered home in all of these trainings is don't click on the wrong link, because if you do that, it might destroy the company. And I can't help but think if someone in the accounting department clicks the wrong link and it destroys the company, I can't really get myself to a point where I blame the accountant for that. I don't feel that's the accountant's job. That is definitely not the accountant's job. And there is no way that a single user or a single device should be able to take out an organization. If that is the case, something has gone very badly wrong. And I would say the blame is definitely not

on the person who clicked the link. There would be quite a range of people who would probably be hauled over the coals in regards to that one. Educating users on what to be aware of and what to look out for and what to not click on versus click on, how to spot scams, all of that is helpful and beneficial, not just for an organisation, but I also see it as very useful even in their own personal lives because we are seeing ransomware scams targeting individuals. We're seeing some of those awful scams talking about coming from tax departments or coming from anything that's talking about, you've got a fine, you need to pay this,

please go off and buy Target gift cards. Those are the kind of things that we want people to be aware of, even in their personal lives. But if an organization can be taken down by one of those emails, then there has been something that hasn't been put in place. I would say something. A lot of things that haven't been put in place. A lot of the work that we do from a security perspective is to limit what we call a blast radius, to try and reduce the impact an incident will have on an organization. And clicking on an email like that should produce an alert. It should say, hey, you probably shouldn't be accessing this website, even if they put their username and password in. It's the credentials of that account that

have been compromised. That can be reset. If necessary, the account can be rebuilt, but it certainly shouldn't be something that brings down an entire organization. And I think our messaging around that puts the burden on users when it should be on those of us who are technical to have the, not necessarily the accountability, but the responsibility for looking after that and ensuring that that is not the case. And again, that comes back to that number one basics and making sure we've got the basics right. Because if you're clicking on something like that and it's going to be installing malware, you shouldn't have admin rights on your machine. That's just no bad. But number two, it means that- And as part of that, the software we need to do our

job should not require admin rights to run. There are a couple of vendors that are hearing their ears burn because I'm thinking about them very hard right now. Oh, if I have one more piece of software, I need domain admin rights to be able to run. I need global admin rights. No, you don't. You really, really don't. You are being lazy. You're taking the piss.

You're making it easier for yourself and causing a massive security hole for your customer. Stop it. Back to my point. If our operations folks and our security folk can work well together, we can get around some of that burden that we put on our users to be able to work out what it is from a usability perspective they need while still giving them the security they require. And I think when we're looking at putting security in place, it's that putting security first. It's not an afterthought. It's built into whatever we are doing, whatever processes we have, whatever systems we're bringing in, whatever solutions we're looking at using. It's thought of at the beginning rather than,

oh, maybe we should talk to security about that. That involves a little bit of a culture shift. I realize that's going to take a bit of time. I can do technology. People are someone else's problem. That is definitely not me to try and fix. And each organization will have their own battle when it comes to that. There's also this idea that, you know, companies figure this out after the first time they get it hilariously wrong, that not every person should have access to everything. I appreciate the idea of transparent culture. Don't get me wrong. But if I'm not working with a particular customer, why should I have access to that customer's data just lying around?

It should be something that takes work that you have to affirmatively say, yes is it's a heck of a lot easier if you can constrain the regulated or sensitive information to as small a surface area as possible. Here's the PCI environment where all that stuff lives. And yeah, you turn on all the obnoxious, difficult security things involving that environment. But what that means then is you don't have customer data in staging and development environments to worry about. And you can relax a lot of the other controls just because you don't need to have that high friction process for people to do things that are completely unrelated to the sensitivity of that data. And that still seems like it's a revelation for some folks. For a lot of folks. So the idea of role-based access control and giving people the rights they need to be able to do their job, no more and no less. And again, it's a balancing act of trying to work out what

that is. And when it comes to the people who are doing that job, I often find when looking at putting role-based access control in place, it's never end users that are the problem, ever. It's always technical people because they need all of the access all of the time. You don't. role-based access control in place, it's never end users that are the problem, ever. It's always technical people because they need all of the access all of the time. You don't. So it's working out what they actually need versus what they want. And that's an even harder discussion to have. So I find it's not what access do you need, it's what are you trying to do and let's work backwards from that. And that's something that I still think a lot of organisations haven't managed to get

right. And also from an access perspective, that governance, again, we're getting into process and policy and people, but it's that access governance life cycle as well. Just because someone had access doesn't necessarily mean they need that access going forward and making sure we're reviewing those levels of access going forward and removing people who don't need that anymore. Normally, when I'm talking about that, people are thinking like IT folks who have lots of privilege or finance people who have lots of privilege, I'll be honest, the worst folks in any organisation for access privileges are executive assistants.

Those people collect access rights like it's going out of fashion because they never get revoked. And they will move from person to person, from organisation to internal organisation, collecting all of those rights. And they'll maintain them for the entire length of their stay with that company. It's an extraordinary amount of rights. They're like the human equivalent of the CICD server because it has access into every environment. It's generally configured by hand and evolves naturally and is bespoke.

Infrastructure is code for everything except the Jenkins box. We just take a disc snapshot of that and kind of hope for the best if we ever have to rebuild it somehow. Yeah. They are. And they're what, like, don't get me wrong. They are wonderful folks. If something happens to them, usually the organization is in a lot of trouble. But when they leave, looking at the sheer amount of access they have, the mail permissions they have to be able to send on behalf of so many folks in the organisation, and you're like, this is a lot of trust and a lot of responsibility to give to one person. So making sure that the rights that the folks have in your organisation are what they need. They are checked regularly. And I think that's the part that gets skipped a lot.

It's easy to give rights. It's harder to remove them. So it's making sure that those access reviews are being done to say, hey, you're no longer the EA for that person. Maybe you don't need senders permissions on their mailbox anymore. Maybe you don't need access into their personal files and their calendar to be able to see exactly what's going on and being able to look at that personal folder that contains all of their photos and files of what's going on. So making sure that you're reducing the risk, not just on the organization, but on those people as well, because that's a lot of responsibility to have should something go wrong.

That's part of the problem too, I think, is that the service area has gotten complex and the sheer number of services that we all use to go about our jobs, regardless of what those jobs might happen to be, is so overwhelmingly massive. I remember going through an acquisition at one point, and we had something like 40 people at the company, and we were well over 800 distinct SaaS products that we were using throughout the entire company for different things. And how many of these are important? How many things are going in other directions? And if we look across the entire surface area of these companies and the things that we're using no one knows what's going on in their environment and where okay you get access to this i don't know

this this ridiculous little thing i use to caption funny pictures okay it's technically a sass product but it's probably not critical path oh here's the system we use to do payroll yeah you probably want to double check that one and it all winds up lumped together in the big bucket of, we have no idea what's in this thing. Oh my God. I need people to understand that documentation is important. And this is exactly why, unless it is written down, unless someone has been able to take out of their head a decision that was made about, this is a product we are using. this is what it does, this is how critical it is to our organisation, unless that is somewhere, you end up in exactly the situation you're talking about. You've got all of these products and you don't know which ones are

business critical. You don't know which ones are considered your primary goals, particularly in the case of a BCP, so a business continuity plan or disaster recovery perspective, they're the ones that you want to protect. And that should be somewhere. But a lot of people seem to think that documentation is boring and documentation is, it's not necessary. It's in my head. Why would I need to write any of that down? I know it all. Please make sure you're getting it out of your head. I don't want to have to pilfer through your head to find the stuff. And I know that there's a couple of folks personally that I know who will feel very attacked by this. Just because it's in your head, just because you know, doesn't mean

everybody else knows. And I'm very much of the opinion that if someone asks you a question twice, that is a piece of information that needs to be on a piece of paper somewhere that other people can see. And we don't do that enough. We don't pass on information enough and share information enough. There is very much an information hoarding mentality for a lot of folk of, I need to protect this information to be able to keep my job. And that is just not the case, because in the case of a disaster, in the case of a merger where you're trying to work out what is actually needed versus what is not, that information is crucial and it can cost time and a lot of money if you don't know that information, if you don't have someone that has put that down and you can reference it in an easy

and quick way. I really want to thank you for taking the time to go through how you think about these things. If people want to learn more, where's the best place for them to find you? And please don't say Australia. Well, yes, Australia, but no one wants to come down here because, again, we have things that will kill you. I would say Twitter or Mastodon. You can find me as at GirlGerms if you haven't found me already. I am on infosec.exchange as Girl Germs.

I will also be speaking at a number of conferences coming up here in Australia again. If you want to come to Australia, please do, but I know that recordings of these will be available at some point. So I will be speaking. Luckily, I've been invited to go and speak in New Zealand, which is very exciting. So I'm going to be speaking at DevNext in Christchurch on the 21st of April. And I will also be at AusCert between the 9th and 12th of May this year.

Again, that's in the future, literally in the future. So I will be talking about all sorts of things related to operations and security operations. My talk at AusSearch is actually going to be really interesting. It's talking about using your operations folks and how to get the best out of them from a security perspective based on my history being an operations person and how I've moved into security and how that can be a real benefit and asset to an organization. And we will, of course, put links to that in the show notes.

Thank you so much for being so generous with your time. I appreciate it. No dramas. Thank you very much for having me. Jess Dodson, Senior Cloud Solution Architect at Microsoft. I'm cloud economist Corey Quinn,

and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that I'll log in as you and delete later because you use the same password for everything you log into. It's Poisonous Kitty. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS. We tailor recommendations to your

business and we get to the point. Visit duckbillgroup.com to get started.