Screaming in the Cloud - CloudDev for Retail Companies with John Mille

Episode Date: April 27, 2023

John Mille, Principal Cloud Engineer at Sainsbury's UK joins Corey on Screaming in the Cloud to discuss how retail companies are using cloud services. John describes the lessons he’s learned since joining the Sainsbury’s UK team, including why it’s important to share knowledge across your team if you don’t want to be on call 24/7,  as well as why he doesn’t subscribe to the idea that every developer needs access to production. Corey and John also discuss an open-source project John created called ECS Compose-X.About JohnJohn is an AWS Community Builder (devtools), Open Source enthusiast, SysAdmin born in the cloud, and has worked with AWS since his very first job. He enjoys writing code and creating projects. John likes to focus on automation & architecture that delivers business value, and has been dabbling with data & the wonderful world of Kafka for the past 3 years.Links Referenced:AWS Open-Source Roundup newsletter blog post about ECS Compose-X: https://aws.amazon.com/blogs/opensource/automating-your-ecs-container-architecture-deployments-with-ecs-composex/ECS Compose-X: https://docs.compose-x.io/LinkedIn: https://www.linkedin.com/in/john-mille/Twitter: https://twitter.com/JohnPre32286850

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. It's easy to f*** up on AWS, especially when you're managing your cloud environment on your own.

Mission Cloud unf***s your apps and servers. Whatever you need in AWS, they can do it. Head to missioncloud.com for the AWS expertise you need. Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably? With SIEM, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be set up in minutes. By automating the access request lifecycle, SIM helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with SIM.

Go to simops.com slash Corey to learn more. That's S-Y-M-O-P-S dot com slash Corey. Welcome to Screaming in the Cloud. I'm Corey Quinn. Today, my guest is a longtime listener, first-time caller. John Mill is a principal cloud engineer at Sainsbury's, which is UK speak for grocery store. John, thank you for joining me. Hi, Corey. Thanks for having me.

So I have to begin with, I guess, the big question that I used to run into people in San Francisco with all the time. They would work at Walmart Labs, and they would mention in conversation that they work at Walmart, and people who weren't aware that there was a Labs out here figured they were a greeter at the grocery store. Do you ever wind up with people making that sort of fundamental assumption around the fact, oh, you work at Sainsbury's as a checker or whatnot? No, but actually, it actually is one of the, if you look at one of the job descriptions from Sainsbury's, the first thing is, why would you join a retail company to do tech? And as it turns out, tech, I mean, I think retail companies, as any other companies in the world, rely on cloud more and more and more.

And I think that one of the things that is interesting today is, if you look at the landscape of retailers, I've heard many times people saying, we don't want to go for AWS because we're giving money to the competition. And actually, I think AWS does a fantastic job overall, giving you all the tools to actually bid them as your competition. And as it turns out, we've had really, really great success running a lot of our workloads on AWS for many, many years now. On some level, if you can't come to terms with the idea of Amazon as competition, you shouldn't be using AWS, regardless of what industry you're in. Because their entire company

strategy is yes. It's very hard to start to even come up with industries that they don't have some form of presence within. On some level, that's a problem. In fact, a lot of levels, that's something of a problem. Everyone tends to wind up viewing the world in a bunch of different ways. I like to divide companies into two groups. More or less, it's, is the AWS bill one of the top three line items at the company? And if the answer is no, on some level, that usually is an indicator that there's a sustainable business there

that both our grandparents and our grandchildren will be able to recognize in the fullness of time. You absolutely have a business that winds up falling into that category. Whereas, oh yeah, I fixed the AWS bill. Yeah, my parents would have no idea what I do, and my kids don't have much of a better one. It feels like it's very point-in-time type of problem, at least I hope.

Technology is not the core of what grocery stores tend to do, but I also don't get the sense that what you're doing is sitting there doing the back office corporate IT style of work either. How do you use technology in the overall context of the business? Well, so we use it in a very wide variety of sense. So you obviously have everything that has to do with online shopping, orders, and all of those sort of things, which obviously, especially with the drive of COVID and being everybody from home, has been a huge driver to improve our ability to deliver to customers.

But certainly, I think that Sainsbury's sees AWS as a key partner to be able to go and say, we want to deliver more value. And so there's been a number of transformations over the years, and one of the reasons I was hired is actually to be part of one of those transformation where we're going to take existing infrastructure servers that literally i usually say to people oh are we doing an upgrade this month has somebody got in their little brush to go and brush under the hard drives to make sure that nothing is going to die and actually do that transformation and move over to the cloud in order to never have to really worry about whether or not they have to manage

hardware and infrastructure. It's strange in that I never got very deep into containers until I was no longer hands-on hardware managing things. I was more or less doing advisory work and then messing around with them. And you'd think, given my proclivities historically of being very unlucky when it comes to data, you would think that this would be great because, oh, yeah, you blow away an ephemeral container. Well, that's kind of the point. We'll all laugh and it'll reinstantiate itself and life goes on. But no, making fun of them was more or less how I tended to approach them for the longest time until I started to see them a little bit, well, I guess,

less as a culture, less as a religion, and more as an incredibly versatile packaging format, which is probably going to annoy the people I know who are the packaging pedants for Linux distributions. How do you tend to view them? And how did you start using them? Right. So that's a great question. So historically, I was a student at, I think, the school where one of the original creators of Docker were. And one of the things that you learn when you do development at the school is that, you know, give everybody a great framework, a great API to drive the deployment of containers in the world and bundle them and ship them around the world on your laptop and somebody else's and help a little bit with it, you know, solving the problem of it works on my laptop, but not just on the laptop. Probably, maybe. It's obviously gone viral over

the years and I really enjoy containers. I quite like containers. What I find interesting is what people are going to do with it. And I think that over the last few years, we've seen a number of technologies such as Kubernetes and others come into the scene and say, and trying to solve people's problem, but everybody seems to be doing sort of things in their own way and historically i started off using ecs when it was terrible and you didn't have security groups for containers and all of this but over the years you know you learn and edws has improved the service quite significantly with more and more features and i think we are today in a place where there is this landscape

i think where a lot of workloads are going to be extremely ephemeral and you can go for whatever you want. And if you have a platform or a workload that you need to have working in different places, maybe Kubernetes could be an easy way to have a different sort of set of features that allows you to move around, maybe in an easier way. But that also comes with a set of drawbacks. Again, I look at using EKS, for example, and I see, okay, I have to manage IAM and RBAC now, whereas if I use something like ECS or whatever the Neuromag Cloud vendor of choice,

I don't have to deal with any of this. So I think it's finding the fine balance between how you do orchestration of containers now and what works for you. And it's sustainable over the time more than about are you going to use containers? Because the chances are somebody is using containers. My experiences and workflows and constraints are radically different than that of other folks. Because for a lot of the things I'm building, these are accounts where I'm the only person that has access to them. It is me. So the idea of fine-grained permissions for users from an RBAC perspective doesn't really factor into it. Yes, yes, in theory, I should have a lot of the

systems themselves with instance roles being managed in safe and secure ways. But in many cases, the AWS account boundary is sufficient for that, depending on what it is we're talking about. But that changes when you start having a small team of people working with you and having to collaborate on these things. And we do a little bit of that with some of our consulting stuff that isn't just the shitpost stuff I build for fun. But there's multiple levels beyond that. You're clearly in a full-blown enterprise at this point, where there are a bunch of different teams working on different things, all ideally going in the same direction. And it's easy to get stuck in the weeds of having to either go through central IT for these things, which gives rise to shadow IT every time you find a corporate credit card in the wild, or it winds up being everyone can do what they want,

but then there's no consensus, there's no control. There's no architectural similarity. And I'm not sure which path is worse in some respects. How do you land on it? Right. So what I've seen done in companies that works very well, and again, to the credits of my current companies, one of the things they've done really well is build a hub of people who are going to manage solely everything that has to do with accounts access right so they control IAM security hub all of those sort of things for you there's things that are mandatory that you can't deal without you have permissions boundary that's it you have to use those things end of story but beyond that point once you have access to your account you've been given all of the access that is necessary for you to deliver application and

deploy them all the way up to production without asking permission for anybody else apart from your delivery managers, potentially. And I think from there, because there is the room to do all of this, one of the things that we've done within my business unit is that we've put in place a framework that enables developers. And when I say that, it really is a question of allowing them to do everything they have to do, focus on the code. And I know it's a little catchy phrase that you hear these days, but the developers really are the customers that we have. And all that we do is to try to make sure that they have a framework in place that allows them to do what they need and deploy the applications in a secure fashion. And the only way to do that for us was to build the tools for them that allows them to do all of that. And I honestly haven't checked a single service IAM policies in a very long time, because I know that by providing the tools to developers, they don't have this will

to go and mess with the permissions because their, they don't have this will to go and mess with the permissions because their application suddenly doesn't have the permissions. They just know that with the automation we've provided them, their application gets the access it needs and no more.

On some level, it feels like there's a story around graduated development approach, where in a dev environment, you can do basically whatever you want with a big asterisk next to it that's the same asterisk by the way next to the aws free tier but as you start elevating things into higher environments you start to see gating around things like who has access to what security reviews etc etc and ideally by the time you wind up getting into production almost no one should have access. And that access that people do have winds up being heavily gated.

That is, of course, the vision that folks have. In practice, reality is what happens instead of what we plan on. The idea of it works in theory, but not in production is, of course, why I call my staging environment theory. Does that tend to resonate as far as what you've seen in the wild?

Yeah, very much so. And when I joined the company and we put together our standard pipelines for developers to be able to do everything, the rule that I would give to my team, so I manage a small team of cloud engineers, the one rule I would say is we have access to prod because we need to provision resources. But when we're going to build the pipelines for the developers, you have to build everything in such a way that the developers will only have read-only access to the production environment. And that is only to go and see their logs.

And at least try to foster this notion that developers do not need access to production as much as possible because that avoids people go and do something they shouldn't be doing in those production environments. Now, as the pipeline progresses and applications get deployed to production, there are some operational capabilities that people need to have. And so in that case, what we do is we try to fine-tune what do people need to do and grant those people access to the account so that they can perform the jobs. And I don't have to be woken up at two in the morning. The developers are. One thing that I think is going to be a

cause of some consternation for folks, because I didn't really think about this in any meaningful sense until I started acting as a consultant, which means you're getting three years of experience for every year that you're in the wild, just by virtue of the variety of environments you encounter. On some level, there's a reasonable expectation you can have when you're at a small, scrappy startup that everyone involved knows where all the moving parts live. That tends to break down with scale. So the idea of a cloud center of excellence has been bandied around a lot. And personally, I hate the term because it implies the data center of mediocrity, which is a little on the nose for some people at times. So the idea of having a sort of a centralized Tiger team that has the expertise and has the ability to go on deep dives and sort of

loan themselves out to different teams seems to be a compromise between nobody knows what they're doing and every person involved should have an in-depth knowledge of the following list of disciplines. For example, most folks do not need an in-depth primer on AWS billing constructs. They need about as much information fixed. It's on an index card. Do you find that having the centralized concentration of cloud knowledge on a particular team works out? Or do you find that effectively doing a rotating embedding story is the better answer? It varies a lot, I think,

because it depends on the level of curiosity of the developers quite a lot. So I have a huge developer background. People in my team are probably more coming from X IT environments or this sort of operation. And then they just naturally went into the cloud. And in my opinion, it's fairly rare to find somebody that is actually good at doing both AWS and coding. I am by no means really, really great at coding.

I code pretty much every day, but I wouldn't call myself a professional developer. However, it does bring to my knowledge the fact that there are some good patterns and good practices that you can bring into building your applications in the cloud and some really bad ones. However, I think it's really down to making sure that the knowledge is here within the team. If there is a specialized team, those need really to be specialists. And I think the important thing then is to make sure that the developers and the people around you

that are curious and want to ask questions know that you are available to them to share that knowledge. Because at the end of the day, if I'm the only one with the knowledge, I'm going to be the one who is always going to be on call for this or doing that. And this is no responsibility that I want. I'm happy with a number of responsibilities, but not to be the only person to ever do this. I want to go on holidays from time to time. So at the end of the day, I suppose it really is up to what people want or expect out of their careers.

I do a job that was a passion for me since I was about 14 years old. And I've always been extremely curious to understand how things work. But I do draw the line that I don't write anything else than Python these days. And if you ask me to write Java, I'll probably change job in the flip of a second. But that's the end of it. But I enjoy understanding how Java things work so that I can help my developers make better choices with what services in OBS to use. On some level, it feels like there's a, I guess, lack of the same kind of socialization that startups have sort of been somewhat guided by, as far as core ethos goes, where, oh, whatever I'm working on, I want to reach out to other people

and, hey, I'm trying to solve this problem. What is it that you have been working on that's germane to this and how can we collaborate together? And it has nothing to do, incidentally, with the idea that, oh, big companies, people aren't friendly or aren't dedicated or aren't good or aren't well-connected. None of that. But there are so many people internally that you're spending your time focusing on, and there's so much more internal context that doesn't necessarily map to anything outside of the company that the idea of someone off the street who just solved a particular problem in a weird way could apply to what a larger company with, you know, regulatory burdens starts to have in mind, it becomes a little bit further afield. Do you think that that's accurate?

Do you think that there's still a strong sense of enterprise community that I'm just potentially not seeing in various ways because I don't work at big companies? It's a very fine line to walk. So when I joined the company, I was made aware that there's a lot of Terraform and Kubernetes, which I went the total all the way with CloudFormation and ECS. So that was one of the challenges I knew I would have. But I came with an open mind. And when I looked around,

okay, what are the Terraform modules? Because I used Terraform with anger for an entire year of suffering. And I thought, okay, well, maybe people have actually got to a point where they've built great modules that I can just pick up off the shelves and reuse or customize only a tiny little bit,

add maybe a couple of features, and that's it, move on. It's good enough for me. But as it turns out, there is, I think, a lot of the time a case where the need for standardization goes against the need for business to move on. So I think this is where you start to see silos start to being built within the company and people do their own thing and the other ones do their own. And I think it's always a really big challenge for a large company with extremely opinionated individuals to say, all right, we're going to standardize on this

thing. And it definitely was one of the big challenges that I had when I joined the company, because again, big Kubernetes and Terraform place, we're going to need to do something else. Then it was the case of saying, hey, I don't think we need Kubernetes and I definitely don't think we need Terraform for any of those reasons. So how about we do something a little different? Speaking of doing things a little bit different, you were recently featured in AWS Open Source Roundup newsletter. That was just where you, I think, came across my desk one of the first times. It was specifically around an open source project that you built, ECS Compose-X. So I assume it's like, oh, it's like Docker Compose for ECS, and also the X implies that it is extreme,

just like, you know, snack foods at the convenience store. What does it do, and where did it come from? Right, so you said most of it, right? It literally is a question where you take a Docker Compose file, and you want to deploy your services that you worked on, and all of that together, and you want to deploy it to DLBS. So ECS ComposeX is a CLI tool very much like the Copilot. I think it was released about four months just before Copilot came out. So sorry, beat you to the ball there. But with the

Docker Compose, specifications supported. And again, it was really out of, I needed to find a neat way to take my services and deploy them in the BIOS. So ComposeX is just a CLI tool that is going to parse your Docker Compose file and create CloudFormation templates out of it. Now, the X is not for Xtreme or anything like that, but it's actually coming from the fact

that it's extension fields, which is something supported in Docker Compose. And so you can do things like hdac-rds or x-dynamodb, which Docker Compose on your laptop will totally ignore, but ECS Compose X, however, will take that into account. And what it will do is if you need a database or a DynamoDB table, for example, in your Docker Compose file, you do xrds, my database,

some properties, exactly the same properties as CloudFormation, actually. And then you say, I want this service to have access to it in a real-only fashion. What ACS ComposeX is going to do is just understand what it has to do, meaning creating IAM policies, opening security groups, all of that stuff, and make all of that available to the containers in one way or another. It feels like it's a bit of a miss for Copilot not to do this. It feels like they wanted to go off in their own direction with the way that they viewed the world, which I get. I'm not saying there's anything inherently wrong with that. There's a reason that I point kubernetes the easy way.com to the ECS marketing site. But there's so much stuff out there that is shipped or made available in other ways with a Docker composeose file. And the question of, okay,

how do I take this and run it in Fargate or something? Because I don't want to run it locally for whatever reason. And the answer is, that's the neat part. You don't. And it just becomes such a clear mess. There have been questions about this since Copilot launched. There's a GitHub issue tracking getting support for this that was last updated in September. We are currently recording this at the end of March. It just doesn't seem to be something that's a priority. I mean, I will say the couple of times that I've used Copilot myself was always for greenfield experiments, never for adopting something else that already existed. And that was, it just felt like a bit of a heavy lift to me of, oh, you need to know from the beginning, this is the tool you're going to use for the thing. Docker Compose is what the ecosystem has settled on a long time ago. And

I really am disheartened by the fact that there's no direct ECS support for it today. Yeah. And it was definitely a motivation for me because I knew that ECS CLI version one was going into the sunset and there wasn't going to be anything supporting it. And so I just wanted to have Docker Compose because it's familiar to developers. And again, if you want to have adoption and have people use your thing, it has to be easy. And when I looked at Copilot the first time around, I was extremely excited because I thought, yes, thank you, Amazon, for making my life easy. I don't have to maintain this project anymore. And I'm going to be able to just lift and shift move over and be happy about it but when the specification for copilot was out and i could go through the documentation i was equally disheartened because i was like okay not for me and and something very

similar happened when they announced proton i was extremely excited about proton i was i opened a github issue on the roadmap immediately to say hey hey, are you going to support to have some of those things together or not? And the fact that the Proton templates, I mean, again, it was what, two, three years ago now? And I haven't looked at Proton since, so it was a very long time now. It made a splash when it was announced in 2020, and I really haven't seen much from it since. Well, and I haven't done anything at all with it. And literally, one of the first things I did when the project came out, because obviously, this is an open source project that we use in Sainsbury's, right? Because we deploy everything in WVCS. So why would I reinvent the wheel the third time it's been done? I might as well leverage it. But every time something like that came out,

I was seeing it as the way out of nobody is going to need me anymore, which is great. And that doesn't create a huge potential dependency on the company for me. Oh, well, we need this to carry, you know, keep working. Now it's open source,

it's on the license, you can fork it and do whatever you want with it. So from that point of view, nobody is going to ask me anything in the future. But from the point of view where I need to, as much as possible, use AWS native tools or AWS build tools, I definitely wanted every time to move over to something different. But every time I tried and I tiptoed with those alternative offerings, I just went back and said, no, this either is too new and not mature enough yet, or my tool is just better.

Right. And one of the things I've been doing for the past three years is look at the Docker ECS plugin, all of the issues, and then see all of the feature requests that people are asking for, and just do that in my project. And same with Copilot.

The only thing that Copilot does that I don't do is tell people how to do CI CD pipelines. One thing you said a second ago just sort of, I guess, sent me spiraling for a second, because I distinctly remember this particular painful part. You're right, there was an ECS CLI for a long time that has since been deprecated, but we had internal tooling built around that. When there was an issue with a particular task that failed, getting logs out of it was non-trivial. So great, here's the magic incantation that does it. I still haven't found a great way to do that with the AWS v2 CLI. And that feels like it's a gap where,

yes, I understand old tools go away, new ones show up, but, hey, I read a task. Can you tell me where the logs are? No. Well, Copilot's the new answer. Okay, can I use this to get logs from something that isn't Copilot? Oh, absolutely not. And the future is inherently terrible as a direct result. Yeah, well, I mean, again, personally, the only thing that ECS Composix does is create all the templates for you

so you can then just query it and know where everything has been created. And one of the things it definitely does create is all of the log groups because, again, least privileged permissions being something that is very dear to me, I create the log groups and I just allow the services to only write in those log groups, and that's it. Now, typically, this is not a thing that I thought Composex was going to do because that's not its purpose. It's not going to be an operational tool to troubleshoot all the things.

And this is where I think that other projects are much better suited, and I would rather use them as an extension or library of the project, as opposed to reinvent them. So if you're trying to find a tool for yourself to look at logs, I highly recommend something called AWS Logs, which is fantastic. You just say, hey, can you list the groups? Okay, can you get me the groups

and can I tell them on a terminal? And that's it, job done. So as much as I enjoy building new features into the project, for example, I think that there's a clear definition between what the project is for and what it's not. And what it's for is giving people CloudFormation templates they can reuse

in any region and deploy their services and not necessarily deal with the operational... deal with their operations. That's up to them. At the end of the day, it's really up to the user to know what they want to do with it. I'm not trying to force anybody into doing something specific. I would

agree. I think that there's a... there's value to... there's more than one way to do it. The problem is at some point, there's a tipping point where you have this proliferation of different options to the point where you end up in this analysis paralysis model where you're too busy trying to figure out what is the next clear step. And yes, that flexibility is incredibly valuable, especially when you get into, you know, large, sophisticated enterprises. Ahem, ahem. But when you're just trying to kick the tires on something new, I feel like there's a certain lack of golden path where in the event of not having an opinion on any of these things,

this is what you should do just to keep things moving forward, as opposed to here are two equal options that you can check with radio boxes, and it's not at all clear which does what or what the longer-term implications are. We've all gotten caught with the one-way doors. We didn't realize we're passing through at the time and then had to do significant technical debt repayment efforts to wind up making it right again. I just wish that those questions would be called out, but everything else, it doesn't matter. If you don't like the name of the service that you're creating, you can change it later. Or if you can't, maybe you should know now,

so you don't have, in my case, a DynamoDB table that is named test running in production forever. Yeah, you're absolutely right. And again, I think it goes back to one of the biggest challenges that I had when I joined the company, which was when I said, I think we should be using CloudFormation, I think we should be using ECS and not Terraform and Kubernetes for those reasons. And one of the reasons was the people, meaning we were a very small team,

only five cloud engineers at the time. And as I joined the company, there already was three different teams using four different CI CD tools. And they all wanted to use Kubernetes, for example. And they were all using different CICD, like I said, just now, different CICD tools. And so the real big challenge for me was how do I pitch that simplicity is what's going to allow us to deliver value for the business. Because at the end of the day, like you said, many, many times before before the AWS bill is a question of architecture right and there's a link in intricacy between the two things so the only thing that really mattered for me and the team was to find a way find a service that was going to allow to do a number of things a delivery value quickly being supported over time because one of

the things that I think people forget these days well one of the things i'm allergic to and one of the things that makes me spiral is is what i call cv driven take choices where people say hey i love this great thing i read about and i think that we should use that in production how great idea but really i don't know anything about it and it's it's then up to somebody else to maintain it long-term. And that goes to the other point, which is turnover proof is what I call it. So making tech choices that are going to be something that people will be able to use for many, many years. There's going to be a company behind the scenes

that is going to be able to support you as well as you go and use the service for the many, many years to go. I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place for years to go. I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place for them to find you? So people can find me on LinkedIn.

I'm also around on Twitter these days, although I probably about have nine followers. Oh, probably I shouldn't say that. It's fine. We'll put a link into it. We'll put a link to that in the show notes and maybe we'll come up with number 10. We never know.

Thanks again for your time. I really appreciate it. Thanks so much, Corey, for having me. John Mill, Principal Cloud Engineer at Sainsbury's. I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review

on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you go to great pains to type out, but then fails to post because the version of the tool you used to submit it has been deprecated without a viable replacement. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying.

The Duck Bill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.