Screaming in the Cloud - Navigating the Cloud-First World with Eric Pullen

Episode Date: October 8, 2024

Corey is joined by cloud economist Eric Pullen to discuss Eric’s journey at AWS that led to his current role as a cloud economist at Duckbill Group. They explore Eric’s early career building data centers and learning IT finance, highlighting how today’s cloud-first world has transformed career paths. The conversation also addresses the hype around cloud repatriation, with Eric arguing that enterprises are unlikely to return to on-prem due to the efficiency of cloud solutions. Additionally, they touch on cloud cost optimization, AWS service deprecation, and the importance of aligning cloud spending with business value rather than cutting costs blindly.Show Highlights:(1:35) Eric Pullen’s background before joining The Duckbill Group(3:22) What’s going on with cloud repatriation(6:39) Eric’s advice for getting into the IT industry(7:08) How Eric got involved with AWS(10:51) Different aspects of Eric’s time at AWS, including Well-Architected(15:02) The rise of service deprecation in AWS(17:47) Why Eric joined The Duckbill Group(22:42) Eric’s concept of consulting at scale(26:23) How cost can affect performance(32:32) Problems with standardization in enterprises(39:10) Where to learn more about Eric and his workAbout Eric PullenI'm Eric Pullen, and I live just outside of Louisville, Kentucky. I've been following Duckbill Group for a while now, and when I saw an opportunity to join as a Cloud Economist, I couldn't pass it up. Before AWS, I worked at Appriss, Inc. for over 14 years, where I was the Director of IT and helped grow several SaaS products, including VINE, JusticeXchange, and MethCheck. In 2015 I joined AWS, where I worked as a Senior Cloud Infrastructure Architect, the Global Performance Efficiency Pillar Lead for the AWS Well-Architected Framework, and most recently as a Global Solutions Architect in their Healthcare and Life Sciences (HCLS) division. During my time at AWS, I had the chance to work with some of their biggest customers, including GE, Siemens, and AstraZeneca.Outside of work, I've been married to Kelly for almost 19 years, and we have two daughters: Jordan, who is 26 and fully embracing adulthood, and Myia, who is 15. We also have two pets: Rocky, my charcoal Lab, and Turbo, our Lionhead bunny.LinksPersonal site: https://www.ericpullen.com/LinkedIn: https://www.linkedin.com/in/ericpullen/Twitter: https://x.com/ericpullenSponsorsBackblaze: https://www.backblaze.com/

I'm going to try something new. This was something that I could do very well. I think I'm, you know, uniquely suited for being able to talk about cost and articulate cost, but do it with an architecture view. Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today spent nine long years working at AWS before pivoting into his current job. Eric Pullen is a cloud economist here at the Duckbill Group. Eric, thanks for agreeing to talk to me, you know, in public. Thanks, Corey. I'm glad to be here.

And yeah, I think it's only fair that I come on your podcast since I work for you now. Exactly. It's one of those power imbalance force it way through. It's like, hey, you're being voluntold to do a thing. I love being voluntold, so it way through. It's like, hey, you're being voluntold to do a thing. I love being voluntold things, so that's great. Backblaze B2B cloud storage helps you scale applications and deliver services globally. Egress is free up to three times the amount of data you have stored and completely free between S3 compatible Backblaze

and leading CDN and compute providers like Fastly, Cloudflare, Vulture, and Coreweave. Visit Backblaze.com to learn more. Backblaze. Cloud storage built better. So where do you come from? Most of the time when I talk to folks who are interested in working here their early career, it's a delicate thing to say, but cloud economists don't just spring fully formed from the forehead of some god somewhere. There's no such thing as a junior person who is likely to succeed in this role by any angle that I can spot it from. You've been in this industry a dreadfully long time. What's your story? So I built data centers to start with. So I, you know, out of

college, worked for a medium-sized company and built a couple data centers, did all the sort of standards that you would expect. And I really had to learn the sort of financial side of IT when I did that. I was a director of IT and ran the department. So everything from power and cooling to networking and storage and all the way up to basically the OS levels where we would stop. And so you had to learn not only all the technology aspects, which we all learned, and that's the part I was most interested in, but also had to learn about what does CapEx mean and what is OpEx and how do we deal with that? And that was an interesting addition to my repertoire of skill sets. But it was also great because I got to learn not only the IT side, but also the

business side. And I think that's a great learning point for a lot of people that maybe they don't get, especially if they're junior, maybe they don't have the ability to even look at some of these things. Like, you know, what does the budgeting process look like for a mid-size or even enterprise size? I mean, those are interesting aspects to any IT type job. It feels like this is a similar path to the one that I went down. And everywhere I look these days, it feels like the door to do that sort of thing has long since closed. You're fresh out of school. You're going to go build data centers and become the director of the IT department. That sounds like a 90s to 2000 era story. Now you do

that. And the question is, oh, great. Was the company just negligent? Like we found a kid out back. We're going to let them go ahead and run all the computers here. And it feels weird now that I'm the old guy because back then it was I was the young kid and I was doing this stuff. And I do think there is a level of that that one, a lot of companies don't need that aspect anymore. I mean, how many companies are building data centers anymore? So even understanding or needing that technology skill set is just gone. Folks are insisting that cloud repatriation is a big thing. Even AWS a couple of weeks ago made a whole stink about this, but only in response to a regulator.

They said, well, this is happening. Where is it happening? The data center company's stocks are flat. Is the cloud repatriation serverless? What's the deal here? And I look at the server repatriation and all the stuff that goes with it, and people are just going crazy about it.

Let's be honest. These enterprises, they move at glacial speeds. And they've moved to cloud, which is great. And it took them years to do it. There is no way they're bringing that stuff back on-prem in a short order. So I think it's premature at best to say that. I think a lot of those companies

aren't even interested in running that again. Is that the core business? I mean, that's the thing I think people miss a lot of times. Is your core business, healthcare and life sciences companies, for example, is that your core business to run data centers? No, it's to design drugs and to do drug trials. And do you even care what a data center is? No, you just need to run compute. You need a lot of it nowadays, especially with all the gen AI, but more specifically the old AI stuff that they would do. And that's great. You need that, but you don't need to understand and run those data centers. And I think that is

an art that is going away for a lot of younger people that are getting into this industry. That's why I struggle when people ask, how did you break into this? I don't know that I could tell you an answer that would be applicable for somebody that's coming out of college today, because a lot of those conditions just didn't exist. My breakthrough job was being the network admin at Chapman University because they liked my personality when I interviewed and the technical interviewer was out sick that day. They called to offer me the job 20 minutes after I left. So I crammed like hell for three months and realized that I had gotten to the level I

needed to be. And a year later, someone approached, hey, you want to come work in corporate life? Why would I want that? Well, we'll give you a 50% raise just for showing up. Okay, I'm going to corporate life. I'll take that every day. No, I and I'm the same way. I mean, I kind of fell into mine as a career. I was working in IT when I was in college. And I did it at the, you know, campus computer centers, you know, trying Slackware on an old, I don't know, HP desktop of some sort and learning Unix through Next. If you remember the old Next stations and the cubes and the slabs and stuff, that's how I learned Unix and then Linux and then other things later. But, you know, I got to learn all that because it was just stuff to play with because I was working to save money for college tuition at that time, which was great.

I mean, it was a great environment. Today, that's a little bit more esoteric, I think. There are whole curricula aimed at these things too. Oh, you want to work in computers. Great. Join the 5,000 other students at this university that are doing the exact same thing and try and distinguish yourself. It's funny that you mentioned that. I actually taught a course in college once just out of an old teacher reached out to me at the University of Louisville and said, would you be interested in being an adjunct professor for this database course, which I was not huge into database, but it was an interesting thing to try. Glad I did it. Man, that's a difficult thing to do. And also, it's difficult for me to articulate to them where this technology was going.

This was even years ago at this point. But, you know, to be able to show people, you know, where we started and where we are today, I mean, the original database for me was Lotus M23 or D-Base and other things. Nowadays, you've got a database for everything. So it's just kind of a crazy shift that we've seen. But I do think that when we're looking at anybody trying to get into this industry, my suggestion and the one thing I do tell people all the time is try to gain as much knowledge as you can in a broad set of environments.

Don't limit yourself to just working in startups or just working at enterprises, because each of those have positives and negatives. But you need to learn from all of those different sets to learn the different skill sets you're going to need if you want to continue to grow up in the IT world. So after doing this for a period of time, you decided, you know what I'm going to do? I'm going to go work for an online bookstores computer division. You know, it's funny. I mean, I enjoyed what I did before I went to AWS, but I wanted to see what AWS was. And so in 2015, this was still early days of cloud for a lot of people. And for us, I was working at a company where we were moving things to cloud and it was great. And I kind of convinced that team that we need to go down this road after having built data centers and all that. So I wanted to see what that was like. And ProServe

had just started up inside of AWS. And so their professional services group was one where you could work remotely. And so I was an early remote worker for AWS. And a lot of their positions, even at that time, were not remote friendly. But this is one that was. And so for me, it was a great opportunity to stay where I live, which is in Kentucky, and still be able to experience Amazon and learn from them. And so I joined ProServe in 2015. And I really loved it. I got to work with some of the biggest clients. It was an interesting time for AWS because a lot of the tooling didn't exist that does now. I mean, there was, you know, all the stuff that we do to create all these accounts and all that stuff, that didn't exist. And, you know, transit gateways were not around. We were, you know, Cisco had a appliance you could install and it was...

We're going to do a whole lot of horrible things with EPC peering and hope for the best. It was bad, but it was also exciting. Come work in our cloud. An in-depth knowledge of several routing protocols is required. What are you people doing over there? You really aren't going to like the answer to that. I'm pre-VPC guy, right? I remember when we had EC2 Classic and I was running just Linux, running on, I think it was Fles Linux at the time, and running it on a single, you know, one or two machines that had nothing but a security group protecting them, which was at the time seemed like a good idea. But now with VPCs and all the constructs that they have,

it's even better and more secure. But for us, it was a godsend that we even had that capability. But when I started at ProServe, the thing that I learned very quickly is there wasn't that many people that knew this stuff outside of people that worked in AWS. So you got a community very quickly inside of the world and you had to sort of run that community and learn from all those different people very quickly because consultancies or other outside entities just didn't know how to

run a cloud. And so we were doing as much teaching as we were building because you had to help all these clients just understand what are they even going to do on this. And then the other aspect of that is that a lot of people missed even in the initial onset is how much is it going to cost me? We'll worry about that part later. God, everybody was like, oh, just make it work. And we did. And then the cost starts to come up and it's like, wow, this is expensive. Now, some companies, I will say, I worked with many that actually said, well, it's cheaper

than my on-prem chargeback model that the company was using at the time. And that was an interesting conversation to have with the internal IT teams to say, look, I can get the compute for half the cost, AWS, because of this weird chargeback. And I can get it today, and they're not going to sit there stonewalling me for eight months telling me no every chance they get.

And it was the rise of shadow IT. And every large enterprise that was doing chargebacks, I mean, they learned a lot through cloud migrations that all of the business units were basically saying, no, I'm not going to do this anymore. And it was amazing to watch them have to shift. And in some cases, they all shifted straight to cloud as well because they're like, well, they can do it cheaper than me.

I mean, I have all this sort of layers of complexity and maybe we just move everything to cloud. And some of those large enterprises did that. And you heard that at reInvents and main stages and those things, which is all great. But it was a lot of effort to get in those early days to even get people to understand what the cost sort of dimensions were around things. And right-sizing was always the first thing that people looked at. But honestly, right-sizing is only a piece of the puzzle. And I think that's where a lot of companies now are starting to experiment with other pieces as far as cost to go.

What else did you do while you were there? Nine years is a long time to work anywhere. And if you know anything about AWS, you typically, if you're there for very long, you don't stay in one position. I mean, there's probably a few exceptions to that, but most people, especially in the sales org, which technically I was in the sales org inside of AWS, you kind of move around because one, you just want to do more things,

but also because you just you need to to sort of keep your career moving on an upward trajectory. So I spent a few years doing pro serve, learned a ton, and then I got this opportunity to join the Well-Architected team. And that, for me, was a great experience to try to grow my brand. Was the team itself Well-Architected? Or was it one of those cobblers children have no shoes problem? It was an interesting group. I will say there was some of the smartest people that I worked with when I first started in that team were from all over the world. And we had all over the world. We had team members that went to Australia and then West Coast and East Coast. And the team was great. I loved working with that team.

It's like a lot of positions, though. It's difficult to articulate your value to the organization when there's not hard numbers behind it. We didn't have a sales quota. We didn't have clients that we were working with where if they used Well-Architected, they would suddenly increase. So I think we're Well-Architected. And even today, I mean, I still love Well- live well architected and I believe in a lot of its principles. But where it's kind of gone off the rails a little bit is, is it about the concepts

of well architected? Or is it about some tool that we build helps you with those concepts? Because I think it's moved towards the tool and less from the concepts. Tool is kind of a lofty term if I'm being direct. It's a checklist. That is effectively what it is. I was curious when the whole well architected framework came out. So when I dove into it, it was, oh, this is just a formalization of the stuff everyone

should be doing. There were no surprises lurking around in there. And it's one of those things, I will say this, for all of the faults of the tool, it was a way for people to not have Excel spreadsheets with the questions in them, which was an early days problem that we had of that, right? So, I mean, I think the tool was beneficial in that sense. It gave you a sort of a mechanism to use that was easy to repeat. That part was great, but using metrics against the tool, I think is a bad idea. Like how many, you know, essays are doing well-architected reviews? I don't know if that's a metric we should measure if a customer doesn't want to do it and they shouldn't do it. But again, going back to the

well-architected framework itself was the important part to me. And I think that's where a lot of people didn't necessarily understand that that was the key component. And for a lot of enterprises, they never looked at it in a holistic way when they look at workloads. And so you have operations security and performance and cost, all those things.

That's a different way to view it when you look at it in a totality. And every large enterprise I went into, even midsize companies, we start asking those questions. They didn't know how to answer the questions from one person's perspective. We would have to bring together teams to answer all the questions to even understand one singular workload that they were doing on AWS. And I think that was the power of Well Architect. It was the conversation. It was getting those teams together to ask those questions. The tool and all the other things that go with it were just secondary to that.

And I think that's where Well Architect can be powerful as a base level of knowledge, but it's something that you have to do. It's not a tool that you can run. And I think that's where a lot of people got confused with the architecture. I think that there's been a lot of noise about it. And whenever you have a formula like that or a framework, there's always some folks who are wanting to take it beyond the precepts upon which it was based

and into now the process becomes the reason land. And that is where I think people start to go awry with a lot of things, honestly. And yeah, it's true. And the other thing is, and if you know anything about the internals of AWS, you end up with competing things. Like there was also the cloud adoption framework.

Oh, don't worry. You see, you get competed both by AWS outside of the company too. It's a weird sort of system where people are encouraged to generate new instead of adding on to what already exists. And I think system is built the way it is internally, but I also understand that it makes it difficult for customers that, well, is this the thing I should use or this other thing or what's going on? This service hasn't seen a feature release in three years. Is there someone there keeping

the lights on or are you about to come out with a whole series of things and reinvent for it? Or is it going to be merged into something else or is it just going to sort of sit there doing what it does? And service deprecation is something very new for AWS. We're just now seeing the beginnings of that. And I think it's a good thing. Oh, it's a great thing. I wish they communicated more effectively as opposed to having to go out and find it individually. But if they can solve that, yeah, turn the stuff off that doesn't work. And I think there's a lot of teams that wanted things to be turned off and couldn't get them turned off because there was a real sort of hesitation in turning things off because we didn't do that. Like that was not an Amazonian thing, right? We built it as kind of a contract to have the services running for our clients.

I think what they're learning now is it's just, you can't do that forever. I mean, there are certain things, you know, having the old cloud search platform. Like, why is anybody still running that? Like, if you're not moving on to something else, then you need to be told to move on. And, you know, you give them timeframes and all that, and they definitely need to articulate this better. But there's no reason to get rid of stuff that's not being used heavily. They do, in some cases, shut it down to new users as an early step, which is great. And I get the sense that there are still customers

on these things who are like, well, we're definitely going to be turning it off on X date. No, they're not. They're not Google. When Google does a deprecation, it will be ripped out from under you. When AWS does it, I suspect at some point you drag your heels as a customer hard enough, they will fly a team of engineers out to help you migrate off. And I have actually

been first party to that. And I have seen where we have had instances where we said, look, we're going to have to do upgrades on X, Y, or Z platform. Usually it's some managed service platform that they offered. And those are ones where they said, look, if there was an exception because it was a particularly important client or had a particularly important workload, they would make exceptions for that. Now, they would put all kinds of rules and sort of parameters around it, which is great, but they would do that. And I do think that's a big positive for AWS is that they're always willing to kind of get over for customers and make sure that they're always capable of doing

what they need to do to keep running their business. So after nine years, you decided, you know what I should do? I'm going to go work with that guy with a big mouth who basically insults everything that the company I currently work for does. And instead of working on the computers directly, we're going to go talk about the money aspect instead. What's wrong with you? I don't know. I actually, so I look at it this way.

I wanted to do something different. I wanted a new challenge, but I also saw inside of AWS, things are changing. And it doesn't take a rocket scientist to see that things are kind of changing internally at AWS. It could be for the good. It could be for the bad. I can't articulate yet whether it is or isn't. I can tell you, though, from my perspective as an employee of AWS at that time, it was the risk level has increased at AWS.

So layoffs and return to team, return to office, all these things that you hear about in the public media about AWS, that weighs on you as an employee. That's certainly something you look at. Plus, I just wanted to do something new. And so I had been talking with Mike actually for a while, and we conversed about different roles over a period of time. And it was like, it just made sense now that I'm going to try something new. This was something that I could do very well. I think I'm uniquely suited for being able to talk about cost and articulate cost, but do it with an architecture view. And I think that's what's really needed for any cloud economist is you got to understand the technology and what's underlying it. And for a lot of these

companies, you have to understand where the technology even originated from. And I think that's where I built those data centers. I know how that stuff used to run. I know how it runs in cloud. And I also know where you can spend money that doesn't make sense. And so for me, it was just a great opportunity and really enjoyed my time here.

I didn't even realize how... I don't think I was unhappy at AWS. I think I enjoyed what I did, but I think I'm even more excited now than I used to be. And my wife is joking with me now. She's like, you seem much happier now than you have been in a few years. And I think it's not anything other than I'm just interested in doing a fun new thing. And I'm able to grow and learn new things at my own pace without having to worry about sort of the bureaucracy that is AWS.

Again, there's reasons for all that bureaucracy, but I still not have that, I will tell you. You're also the vanguard of a new approach that we're taking with our clients. Historically, most of our engagements have been fixed term, very defined, relatively short. You're embedded full-time at one of our larger customers 40 hours a week, and you're digging very deep

into not just the architectural optimizations, but helping get some of the changes driven across the finish line, working cross-functionally. And it's a new experiment for us, and it's going very well to the point where this is not the only one like this that we now have in flight. Imagine that. And it was fascinating to me

when we were trying to figure out what this role was going to look like. Something you stumbled onto very early on in our conversations was that you thought beyond the big obvious services, EC2, RDS, how to optimize those things. Great. Yeah. Everyone optimizes those things. There are tools that do it. We're not going to come up with much new ground on EC2 optimization. We're just not. But you focus on other areas primarily, which resonates with my own hobby horses as well, as far as how I approach these things. Yeah. And I think it's really strange that, you know, I mean, you look at the bill and, you know, we all should, if we're looking at AWS cloud spend, we should be looking at our overall bill. And, you know, I just like

to look at the top 20. Like, let's just take a really, you know, quick and easy look at what are you spending your money on? And what I find is every enterprise

is a little bit different, but obviously RDS and EC2 and maybe S3 are in the top three. That's typical. You may see CloudWatch up there,

although I hope people are working towards reducing that because there's a lot of money being spent on CloudWatch that I think is a little too much being spent on it. But, you know, neither here nor there on that one.

But then you start looking down. Look at the number five and six and seven on the list. Some of those can be really big drivers of cost. And people just aren't even focused on things like database integration service or managed Kafka, MSK, or it could even be other services. Like maybe you're using chatbots, you know, and you're using other types of sort of, you know, ElastiSearch services, other things where maybe there's opportunities to save money there that nobody's looking at because it's just not at the top of the list.

And yeah, you know, maybe it's another, you know, 30K a month that you could save just by optimizing MSK, let's say. Or Dynamo is another interesting one that I've spent some time recently diving deep into Dynamo. There's a lot of cost optimizations to be done there, and the tooling for that just doesn't seem to really exist. Unlike RDS, unlike RIs and all that stuff where you can pull these levers, some of these don't have many levers. So you have to go much deeper in from an architectural perspective to really understand what can we even do to save money on this? Is there even an opportunity to save money? You have the concept at consulting scale as well of when you see a large billing item, your immediate response is not, this is broken and shouldn't be this way. Your

immediate response is to have questions about it. Okay, what is this doing? What is the business use for it? How does this interplay with other things? Is this a misconfiguration? Is it the mistake? Or is it underpinning an incredibly important, valuable business function, and we should not look directly at it lest it stop working? Exactly. And I think that's another area where a lot of people that work, especially in FinOps space, but simply as an architecture team, if you're an architect on a team, you should be able to articulate to your upper management what the value of the thing is. Somebody looks and says, you're spending $2 million a month on Dynamo, as an example. That's a lot of money to spend on Dynamo. But if it's running a service that's generating $100

million a month, who cares? Your way of spending under what you're bringing in, it doesn't make any sense then. Versus the other way to look at it, if you're spending a couple million dollars a month on Dynamo on a service, that's an inventory service for the pins that you have in your break room, something's wrong. Like, you know, there is an articulation of what the value is. And even in days from Law Architect, we try to help companies, you know,

articulate some of the value of a workload. But specifically when it comes to, you know, looking at it from a FinOps standpoint, it's like one of the things of a workload. But specifically when it comes to looking at it from a FinOps standpoint, it's like, what are the things that you're spending money on? Does it even make sense? And so we look at things like Database Migration Service. Great service from AWS. It can do CDC, so you can have all of your data on-prem running to cloud all the time, and it stays in sync. That's a great service to have. It's not cheap, but it is. But are you even using it for that? Or is it a service where you installed it, you moved some data to the cloud,

you moved all your services to the cloud and you forgot about it? So now you're paying all this money for something where what's the perceived value of that? And so I think there's a lot of conversations when you look at further down the bill is what are these, you know, what's the use case for this stuff?

And is there a use case for it? And then having that conversation about articulating its value versus its cost, I think that's an area where a lot of companies just don't see that yet. Enterprise itself is a skill. And at a small company, great, you can more or less talk to one or two people that'll have the answer to any question you've got in their heads. Whereas in enterprise land, you get to go exploring through a whole bunch of different folk, each have a piece of the puzzle, and you've shown an adroit ability to navigate that well. I have a lot of experience in it, I guess, and I think that people tend to know that I understand

what they're dealing with in enterprise. And I think that sympathy goes a long way when you're doing these kinds of engagements, where you understand that this person that you're talking to only has the ability to affect one very narrow slice of that enterprise organization. And so talking to that person in the way that they understand things is a way that you can gain their trust enough that they can tell you what's even going on. Because otherwise, you're just another consultant that's wanting to change their world. And is it for the better or not? Who knows? But if you're able to sympathize with people, and especially as they're telling you about their problems, and a lot of these FinOps conversations are, they know maybe there's issues here or there. They don't know how to fix them, but they know that there is a concern there. And so you're helping them uncover that and say,

oh, you see this cost is increasing, but it's because of a team that's maybe adjacent to you that's causing that. Well, how can we open up that conversation? And me being sort of a third party to that is also a huge advantage. Any consultancy you've ever worked in, I'm sure this is the case, is like sometimes just being the guy that says, I have this name behind me because we're the FinOps team that we're going to help you and we're going to uncover these savings gives you a lot of cachet with these different groups to say, I want to cut across those boundaries. I need to have conversations at different levels. It's always been critical from my view of the world to be able to delve into what the

technology is actually doing. There's an engineering prerequisite of understanding how these systems talk to each other what the what the what the constraints are and understanding how making an a cost-effective change could have detrimental effects on performance because you know once you bring something down you're not allowed to save money ever again and it's funny you mentioned performance because the pillar that i led for well architected was the performance efficiency pillar and if you read through like well architected as an example you know there's all this push and pull between the different pillars, like cost and performance. Like, yeah, I can make it super performant, but it also is going to cost

you an arm and a leg. Or when you look at reliability is like, I can make this the most reliable system across multiple regions, but it's going to cost a lot of money. And so you have to kind of recognize what's the value of the service you're providing so that you can articulate what the cost should be for that service. Like, again, if you have a service where it's counting the number of pins in the break room, maybe it doesn't need to be multi-region, you know?

But if it's counting, you know, I don't know, the number of drugs that you're shipping out to pharmacies, that's an important number that should never go away. And you may need to- You're not spending enough on that. Go spin up some Manage Nat Gateway so we get fewer questions. Exactly. So I think it's always good to find out what the reason is behind spend. And sometimes that conversation has nothing to do with IT, right? This is a business problem. You're starting to understand why they're doing what they're doing. And you may even uncover that

that business problem doesn't need to be solved in the way they're solving it. So now you've affected an architectural change through conversations you have about why they're spending so much. And I think that's another critical piece of what we do, especially in these newer engagements that Duckville is doing is we're trying to dive much deeper into that and say, it's not just enough to say you can save some money with our eyes. You have to look at why you're doing what you're doing. And it takes time to do that. It's not a, you know, one week engagement and you can find all those things out. It's across the board. You have to have conversations with the organization. Organizations generally tend to be fairly poor at articulating internally the context for a lot of these things. You'll have some engineers

spinning things up that are wildly expensive without justification. But what I always find more interesting is when I encounter folks who are well-intentioned engineers, but they're treating it like it's their own personal money, where they think for some reason they can't spend more than $200 a month on their own development environment there, where it's, I assure you, you cost significantly more than the infrastructure upon which you work. We need this working. Don't worry about the cost. You're really aiming at the wrong part of the story. And it's funny you bring that up because I have had many occasions where I've seen both sides of that, right?

I have people that are unwilling to spend $200 on some AWS service to try it out and to see if it will fix a problem they don't have to build a whole other service for. Great. Or you see the other side, which is someone that is doing some minor scripting and they spun up a 12XL instance to run that in EC2 instance because they don't care. It's not their money and they wanted it to be fast, even though it's way overkill for whatever the thing is they were running.

And they leave it running forever because they don't want a fool shutting it down. There's a weird dichotomy you see, and I think that's probably true for their personal finances as well. Maybe they don't think about it for their own money. Yeah, I've noticed my personal EC2 dev box has gotten larger over the years. It's now a C7G large, and it's, that's a little beefy to be running all the time. But, you know, I need something there that works. I don't have to wait for things to complete. I'm spoiling myself.

And that's true. And spoiling yourself sometimes, like you said, is needed to save you time and effort if your job is to do building scripts or whatever. And then the money kind of works itself out. It's when you see hundreds or thousands of those that you start to wonder about, you know, and that's when you start going, well, why are you even doing this? And that's a question that you have to kind of unfurl the answer to. It could be cultural. It's out of a company. It could be that they've been dictated that this is

the way they do things. And so the first person on the team built a 12 XL and so everybody else does too. A stupendous number of things out there, just workloads are running on a single instance size. Okay. Why that? Well, the developer who was building it that day, five years ago, just pulled something out of the ether, which when you're developing is fine, but it becomes those load-bearing to-dos in the backlog. Oh yeah, we should go and figure out what the right scale is for this thing.

But once it was up and working, it's suddenly in production. And well, that's not a high priority right now. There are features to share. And the other thing that I've seen at a lot of enterprises over the years is this sort of ability that the enterprise wants to have to say, we're going to have standards. And so they want to pick a size of like EC2 or maybe it's a higher level service like RDS or something like every RDS instance is going to be a standard and there's a group that sets standards in large enterprises and they're used to doing that when they had on-prem, it made a lot of sense, right?

Economies of scale, I'm going to buy the same size server or the farms that I build internally. Okay, that made sense to have one standard size. It doesn't make any sense in AWS. I will caveat that. The time where it does is, are you getting a screaming deal on a highly specific private pricing agreement on this one instance size family and region? Because if so, that shoots a hole in everything we just said. I suspect you're not. And yeah, usually when that happens, you know, going in that that's the story. And in my experience, what I've seen working in AWS is the ones where you take advantage of that is weird instance types. GPUs, obviously a lot of people, but I saw it with F1s.

So if you have certain industries where FPGAs are heavily used for certain tasks, and so they would get private pricing for those F1 instances. Makes total sense. But that's a very sort of esoteric use case. It's a weird choice for a bastion host, but okay, you do you. Yeah, exactly. Whatever. But I do think that,

you know, a lot of times companies are just built around standardization when they're enterprises. And it gets to a certain point where that made sense for them. But I do think that's an area where

you have to look at it and say, what is, is this making sense anymore? And it's the old days too with standardization around database engines as an example, right? Everybody was standardizing on Oracle or Microsoft SQL or whatever. It's like,

yeah, that's a great concept. And I get why they would do that because a lot of the heavy lifting. Call legal. We need to figure out what the license terms are on this. But now with open source engines and things like Aurora and MySQL and Postgres and all those, pick what makes sense and let the provider run a lot of that low-level infrastructure, replicating data. I don't know if you remember, but I remember how difficult that was across data centers. Having an Oracle box and using GoldenGate to ship data from one place to the other, it broke all the time. And now you have multi-AZ for something like RDS. You never even have to think about it. It just works., that's an area where I think a lot of people, especially coming into the industry

now if they're new, is they don't even know how difficult some of these things were years ago. I spent months on that at multiple companies, just trying to get replication between two sites that aren't that far apart working. And it was brutal. In some cases, in the same facility, just in a different rack. Yeah, we had them across facility. We did GoldenGate across two data centers, and it was a pain in the butt. But then even, like, think about file replication. Look at how easy something like EFS is today. Or, you know, there's even ONTAP and other things. But that's a pain in the butt to build in a data center.

And if people haven't experienced that, I mean, it is... Well, ONTAP was the solution that you used back then. Waffle's still one of my favorite file systems. But you had to spend top dollar for NetApps to make it work. And you still had to have all these restrictions on how the two buildings were interconnected. And there's all kinds of rules around that stuff. Whereas with AWS, you just, you know, put a button and say, I want a file system and it does it. EFS is magic. I mean, I, you know, I remember the early days of EFS when it got pre-announced. It was one of the first ones that they pre-announced that didn't come out because it didn't work so well from what I saw. But then later when it did come out and now you look at it as a service today, it's a great

foundational service and it works really well for a lot of different use cases. It's not perfect for everyone, but if you're needing NFS, man, it's cheap and it's fast. It's cheap, it's fast, it replicates super well. The only problems I've had with it are more or less when I have made poor life choices. Like, okay, why does it take me seven seconds to log into my box every time when I put my home directory on EFS? Well, yeah, maybe it's time to redo my crufty decade-old ZSHRC, which makes apparently 1,500 calls across the wire for that, which it never expected to do. It's maybe stop doing horseshit and see maybe it works better. And couldn't you know it did? You know, that's another area where I always find it kind of amazing that Amazon does

eventually get around to building services on top of other foundational services. I mean, you look at like EFS is a great service, but you also look at what uses EFS. You can use it with Lambda. You can do all these interconnections. But then you even think about, I remember early days when EFS first got announced, the reason I wanted it is I had an enterprise that had FTP servers or secure FTP servers at the time. And I wanted a quick and easy way to have a centralized file system because that was a pain in the butt at the time. And EFS looked like a great option. Well, it wasn't because of a whole bunch of restrictions that it had. But lo and behold, Amazon says, oh, we could build a file transfer service that frontends S3. I requested that service when I was at BlackRock because it was, I think,

my fourth company where I had built the same thing where it's, all right, you're going to use LibNotify or or I notify on an FTP box to make sure the transfer is done and then validate it and wait a while and make sure the connection is closed and then go ahead and throw it into S3. Can I just pay you to make this go away? Because it's always for big enterprises. This is not for you sharing with your mates, but for enterprises trying to solve for enterprise workloads, it is cheap as free. And people don't realize how tedious that process was to manage file transfer. It always broke.

Yeah, it broke. And it broke in ways you couldn't figure out. And now having the file land in S3, which is where everyone wants it to land anyway, at the end of the day, to then kick off whatever other process. I think there's that whole aspect of things. When people look at on-prem workloads that do FTP and do file manipulation and then push it through a process and a workflow, that stuff is dead simple now in a cloud environment. And it's an API call away, clickety-clickety, and it costs pennies per hour, and you don't

have to worry. And it's so straightforward now for people. And again, this is the history of people coming into the industry today don't even understand how difficult some of this stuff was. Now people come in and go, why would you do that when you can just push the file to S3? I would have loved to have done that over the years, but every other enterprise didn't want to touch S3 at the time. Yeah, go ahead and just teach that giant bank over there how to use S3 instead of FTP. It's like, do I look foolish?

I wish someone had told me. Nobody ever wants to deal with that stuff. And I think that's technology changes. That's great. But being able to still have some of the old vestiges still work in a cloud-friendly way is a great advantage. And I've seen that with AWS. I think that's one of the areas where they really shine is they're trying to bridge that gap between what we used to have to do and what you can do today. And I think there's a lot of areas there that are optimization efforts that you can look at when companies are moving to cloud.

So this whole repatriation and moving stuff back is like, yeah, did you think about all the stuff we're going to have to do again? You just moved a whole bunch of workloads into cloud too. And it's okay, maybe yours are better ways to do these things instead of having to have that fleet of servers to build a message cube. Have you heard of SQS?

It amazes me, too, when you read some of those stories, and I've read a few of them, where it's like, did you just not try to optimize before you moved back? There are a few cases where I could see that it makes sense. I mean, if you're just all you're doing is, you know, I don't know, some really simple website, and maybe you could get away with just one single server and, you know, you're only serving 100 users. OK, maybe it maybe it doesn't make sense. Let's be clear. It can be done. This is not just knowledge of the ancients. We still know how to run these servers at scale. It's just a question of, is that where you want to invest time and effort? It's a decision every company

has to make. My assumption is that people are doing what's right for them. And I don't see a lot of people talking about what that cost is. I mean, every story I've read about is like, we already had teams that could do this work. Great. What if those teams go away? I don't know too many companies that are spending more on infrastructure than they are on people. Agree. And it's a weird setup to say you're going to save that much effort by moving it back

on-prem. I don't know. That whole repatriation stuff, it just doesn't make sense to me. It does not. Can I just say, I'm glad that you're going to save that much effort, you know, by moving it back on prem. I don't know. It's just that whole repatriation stuff. It just doesn't make sense to me. It does not. Can I just say, I'm glad that you're here. I appreciate the chance to work with you. And I'm enjoying, I'm enjoying seeing how this plays out. If people want to learn more, where's the best place for them to find you? Well, they can find me at the Duckville group. And so feel free to come to duckville.com and you can find out more about reaching out to one of us as a cloud economist.

I would be happy or Mike or Corey or any of us can reach out and find out what we can do to help you. Because I would love to have more effort than just the one project I'm working on. I would love to do many more of these. I think that there's opportunities abound for different enterprises that want to engage us or even small to midsize companies. I mean, everyone can save money if you're in cloud. And maybe, again, you can look at those non-EC2 RDS type things, and maybe we can help you find areas to save money that you didn't even think about. So I really want to thank you for taking the time to speak with me. I appreciate it. Thanks, Corey. I appreciate it. Eric Pullen, cloud economist

here at the Duckbill Group. I'm Corey Quinn, also cloud economist here at the Duckbill Group. And this is Screaming in the Cloud. If you enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you'll no doubt be charged way too much money for because you don't realize what that podcast platform is charging you because, eh, that's someone else's problem.