Screaming in the Cloud - Ask Me Anything with Corey Quinn

Episode Date: October 3, 2023

In this special live-recorded episode of Screaming in the Cloud, Corey interviews himself— well, kind of. Corey hosts an AMA session, answering both live and previously submitted questions from his listeners. Throughout this episode, Corey discusses misconceptions about his public persona, the nature of consulting on AWS bills, why he focuses so heavily on AWS offerings, his favorite breakfast foods, and much, much more. Corey shares insights into how he monetizes his public persona without selling out his genuine opinions on the products he advertises, his favorite and least favorite AWS services, and some tips and tricks to get the most out of re:Invent.About CoreyCorey is the Chief Cloud Economist at The Duckbill Group. Corey’s unique brand of snark combines with a deep understanding of AWS’s offerings, unlocking a level of insight that’s both penetrating and hilarious. He lives in San Francisco with his spouse and daughters.Links Referenced:lastweekinaws.com/disclosures: https://lastweekinaws.com/disclosuresduckbillgroup.com: https://duckbillgroup.com

Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. As businesses consider automation to help build and manage their hybrid cloud infrastructures, deployment speed's important, but so is cost.

Red Hat Ansible Automation Platform is available in the AWS Marketplace to help you meet your cloud spend commitments while delivering best of both worlds support. All right, thank you all for coming. Let's begin and see how this whole thing shakes out, which is fun and exciting. And for some godforsaken reason, the lights like to turn off. So we're going to see if that continues. I've been doing Screaming in the Cloud for about give give or take, 500 episodes now, which is more than a little bit ridiculous. And I figured it would be a nice change of pace if I could, instead of reaching out and talking to folks who are innovative leaders in this space and whatnot, if I could instead interview my own favorite guest myself, because the entire point

is I'm usually the one sitting here asking questions. So I'm instead going to now gather questions from you folks and feel free to drop some of them into the comments, but I've solicited a bunch of them. I'm going to work through them and see what you folks want to know about me. I generally try to be fairly transparent, but let's have fun with it. To be clear, if this is your first exposure to my Screaming in the Cloud podcast show, it's generally an interview show talking with people involved with the business of cloud. It's not intended to be snarky because not everyone enjoys thinking on their feet quite like that. But rather a conversation with people about what they're passionate about.

I'm passionate about the sound of my own voice. That's the theme of this entire episode. So there are a few that have come through that are in no particular order. I'm going to wind up powering through them. And again, throw some into the comments if you want to have other ones added. If you're listening to this in the usual screaming in the cloud place, well, send me questions and I am thrilled to wind up passing out more of them. The first one, great one to start, comes with someone asking a question about the video feed. What's with the Minecraft pickaxe on the wall?

It's made out of foam. One of my favorite stories, and despite having a bunch of stuff on my wall that is interesting and is stuff that I've created, years ago, I wrote a blog post talking about how machine learning is effectively selling digital pickaxes into a gold rush because the cloud companies pushing it are all selling

things such as, you know, they're taking expensive compute, large amounts of storage, and charging us by the hour for it. And in response, Amanda, who runs machine learning analyst relations at AWS, sent me that by way of retaliation. And it remains one of my absolute favorite gifts. It's, where is all this creativity in the machine learning marketing? No, instead it's, we built a robot that can think, but what are we going to do with it now? Microsoft Excel, come up with some of that creativity, that energy, and put it into the marketing side of the world. Okay, someone else asked, Brooke asks, what do I think is people's biggest misconception about me? That's a good one. Part of it has been my misconception for the long time about what

the audience is. When I started doing this, the only people who ever wound up asking me anything or talking to me about anything in social media already knew who I was. So I didn't feel the need to explain who I am and what I do. So people sometimes only see the witty banter on Twitter and whatnot and think that I'm just here to make fun of things. They don't notice, for example, that my jokes are never calling out individual people unless they're basically a U.S. senator. And they're not there to make individual humans feel bad about collectively poor corporate decision-making, I would say across the board, people think that I'm trying to be meaner than I am. I'm going to be honest and say it's a little bit insulting, just from the perspective of if I really had an ax to grind against people who work at Amazon, for example. Is this the best I'd be

able to do? I'd like to think that I could at least smack a little bit harder. Speaking of, we do have a question that people sent in in advance. When was the last time that Mike Julian gave me that look? Easy. Would have been two days ago because we were both in the same room up in Seattle. I made a ridiculous pun and he just stared at me. I don't remember what the pun is, but I am an incorrigible punster. And as a result, Mike has learned that whatever he does when I make a pun, he cannot encourage me. But don't piss. That's right. They're no longer puns. They're dad jokes. A pun becomes a dad joke once the punchline becomes apparent. Yes. Okay. The next one is, what is my favorite AWS joke? The easy answer

is something cynical and ridiculous, but that's just punching down at barrier service teams. It's not my goal. My personal favorite is the genie joke where a guy rubs a lamp, genie comes out and says, you can have a billion dollars if you can spend a hundred million dollars in a month and you're not allowed to waste it or give it away. And the person says, okay, those are the rules. Like, okay, can I use AWS? And the genie says, well, okay, there's one more rule. I think that's kind of fun. Let's see another one, a hardball question. Given the emphasis on right-sizing for meager cost savings and the amount of engineering work required to make real architectural changes to get costs down? How do you approach cost controls in companies

largely running other people's software? There are not as many companies as you might think where dialing in the specifics of a given application across the board is going to result in meaningful savings. Yes, yes, if you're running something at hyperscale, it makes an awful lot of sense, but most workloads don't do that. The mistakes that you most often see are misconfigurations for not knowing this arcane bit of AWS trivia as a good example. There are often things you can do with relatively small amounts of effort.

Beyond a certain point, things are going to cost what they're going to cost without a massive re-architecture, and I don't advise people do that because no one is going to be happy re-architecting just for cost reasons. It doesn't go well. Someone asked, I'm quite critical of AWS, which does build trust with the audience. Has AWS tried to get you to market some of their services, and would I be open to do that? That's a great question. Yes, sometimes they do. You can tell

this because they wind up buying ads in the newsletter or the podcast, and they're all disclaimed as a sponsored piece of content. I do have an analyst arrangement with a couple of different cloud companies, as mentioned last week in aws.com slash disclosures. And the reason behind that is because you can buy my attention to look at your product and talk to you in depth about it, but you cannot buy my opinion on it.

And those engagements are always tied to, let's talk about what the public is seeing about this. Now, sometimes I write about the things that I'm talking about because that's where my mind goes. But it's not about, okay, now go and talk about this because we're paying you to and don't disclose that you have a financial relationship. No, that is called fraud. I figure I can sell you as an audience out exactly once, so I'd better be able to charge enough money to never have to work again. Like when you see me suddenly talking about multi-cloud being great and I become a VP at IBM,

about three to six months after that, no one will ever hear from me again because I love nesting doll yacht money. It'll be great. Let's see. The next one I have on my prepared list here is, tell me about a time I got AWS to create a pie chart. I wish I'd see less of it.

Every once in a while, I'll talk to a team and they're like, well, we've prepared a PowerPoint deck to show you what we're talking about. Amazon is famously not a PowerPoint company and I don't know why people feel the need to repeatedly prove that point to me because slides are not always the best way to convey complex information. I prefer to read documents and then have a conversation about them, as Amazon tends to do.

The visual approach and the bullet lists and all the rest are just frustrating. If I'm going to do a pie chart, it's going to be in service of a joke. It's not going to be anything that is the best way to convey information in almost any sense. How many internal documents do I think reference me by name at AWS is another one. And I don't know the answer to documents, but someone sent me a screenshot once of searching for my name in their Slack internal nonsense thing. And it was about 10,000 messages referenced me that it found. I don't know what they were saying.

I have to assume on some level, just something that does a belt feed from my Twitter account where it lists my name or something. But I choose to believe that, no, they actually are talking about me to that level of extreme. Let's see.

Let's turn back to the chat for a sec. Because otherwise, it just sounds like I'm doing all prepared stuff. And I'm thrilled to do that, but I'm also thrilled to wind up fielding questions from folks who are playing along on these things. I love your talk, Heresy in the Church of Docker. Do I have any more speaking gigs planned? Well, today's Wednesday, and this Friday I have a talk that's going out at the CDK Community Day. I also have a couple of things coming up their internal corporate presentations at various places but at the moment no i suspect i'll be giving a talk if they accept it at scale in pasadena in march of next

year but at the moment i'm mostly focused on reinvent just because that is eight short weeks away and i more or less destroy the second half of my year because, well, holidays are for other people. We're going to talk about clouds as Amazon and the rest of us dance to the tune that they play. Look in my crystal ball. What will the industry look like in 5, 10, or 20 years? Which is a fun one. You shouldn't listen to me on this at all. I was the person telling you that virtualization was a flash in the pan, that cloud was never going to catch on,

that Kubernetes and containers had a bunch of problems that were unlikely to be solved. And I'm actually kind of enthused about serverless, which probably means it's going to flop. I am bad at predicting overall trends, but I have no problem admitting that, wow, I was completely wrong on that point, which apparently is a rarer skill than it should be. I don't know what the future

of the industry holds. I know that we're seeing some AI value shaping up. I think that there's going to be a bit of a downturn in that sector once people realize that just calling something AI doesn't mean you make wild VC piles of money anymore. But there will be use cases that filter out of it. Don't know what they're going to look like yet, but I'm excited to see it. Okay. Have any of the AWS services increased costs in the last year? I was having a hard time finding historical pricing charts for services. There have been repricing stories.

There have been SMS charges in India and Pinpoint and a few other things that wound up increasing because of a government tariff on them. And that cost was passed on. Next February, they're going to be charging for public IPv4 addresses. But those tend to be the exceptions. The way that most costs tend to increase

have been either it becomes far cheaper for AWS to provide a service and they don't cut the cost. Data transfer being a good example. They'll also often have stories in that they're going to start launching a bunch of new things. And you'll notice that AWS bills tend to grow in time. Part of that's growth. Part of that is just cruft because people don't go back and clean things up.

But by and large, I have not seen this thing that used to cost you a dollar is now going to cost you $2. That's not how AWS does pricing, thankfully. Everyone's always been scared of something like that happening. I think that when we start seeing actual increases like that, that's when it's time to start taking a long, hard look at the way that the industry is shaping up. I don't think we're there yet. Okay. Any plans for a last week in Azure or last week in GCP? Good question. If so, I won't be the person writing it. I don't think that it's reasonable

to expect someone to keep up with multiple large companies and their releases. I'd also say that Azure and GCP don't release updates to services with the relentless cadence that AWS does. The reason I built the thing to start with is simply because it was difficult to gather all the information in one place,

at least the stuff that I cared about with an economic impact. And by the time I'd done that, it was, well, this is 80% of the way toward just republishing it for other people. I expected someone was going to point me at a thing, so I didn't have to do it. And instead, everyone signed up. I don't see the need for it. I hope that in those spaces, they're better at telling their own story to the point where the only reason someone would care about a newsletter would be just my sarcasm tied into whatever was released.

But that's not something that I'm paying as much attention to, just because my customers are on AWS. My stuff is largely built on AWS. It's what I have to care about. Let's see here. What do I look forward to at reInvent? Not being at reInvent anymore.

I'm there for eight nights a year. That is shitty cloud Hanukkah come to life for me. I'm there to set things up in advance. I'm there to tear things down at the end. And I'm trying to have way too many meetings in the middle of all of that. I am useless for the rest of the year after reInvent. So I just basically go home and breathe into a bag forever. I had a revelation last year about

replay, which is that I don't have to go to it if I don't want to go. And I don't like the cold, the repetitive music, the giant crowds. I want to go read a book in a bathtub and call it a night. And that's what I hope to do in practice. I'll probably go grab dinner with other people who feel the same way. I also love the drink up I do there every year over at Atomic Liquors. I believe this year, we're partnering with the folks over at Red Monk because a lot of the people we want to talk to are in the same groups. It's just a fun event. Show up. Let us buy you drinks. There's no badge scanning or any nonsense like that. We just want to talk to people who care to come out and visit. I love doing that. It's probably my favorite part of reInvent other than not being at reInvent.

It's going to be on November 29th of this year. If you're listening to this, please come on by if you're unfortunate enough to be in Las Vegas. The one else had a good question I want to talk about here. I'm a TAM for AWS. Cost optimization is one of our functions. What do you wish we would do better after all the easy button things, such as picking the right instance and family, savings plans, RIs, turning off or delete orphaned resources, watching out for inefficient data transfer patterns, etc. I'm going to back up and say that you're begging the

question here, in that you are doing the easy things, at least not at scale, not globally. I used to think that all of my customer engagements would be, okay, after the easy stuff, what's next? I love those projects. But in so many cases, I show up and those easy things have not been done. Well, that just means that your customers haven't been asking their TAM. Every customer I've had has asked their TAM first. Should we ask the free expert or the one that charges us a large but reasonable fixed fee?

Let's try the free thing first. The quality of that advice is uneven. I wish that there were at least a solid baseline. I would love to get to a point where I can assume that I can go ahead and be able to just say, okay, you've clearly got your RI stuff, your right sizing, your deleting stuff you're not using, taking care of. Now let's look at the serious architecture stuff.

It's just rare that I get to see it. What tool, feature, or widget do I wish AWS would build into the budget console? I want to be able to set a dollar figure. Maybe it's zero. Maybe it's $20. Maybe it is irrelevant. But above whatever I set,

the account will not charge me above that figure, period. If that means they have to turn things off, if that means they have to delete portions of data, great. But I want that assurance because even now when I kick the tires in a new service, I get worried that I'm going to wind up with a surprise bill because I didn't understand some very subtle interplay of the dynamics. And if I'm worried about that, everyone else is going to wind up getting caught by that stuff too. I want the freedom to experiment and if it smacks into a wall, okay, cool. Though that's $20, that was worth learning that, whatever. I want the freedom to experiment and if it smacks into a wall okay cool though that's 20 that was worth learning that whatever i want the ability to not be charged on reasonable overages i'm not

worried about it turning from 20 into 40 i'm worried about turning from 20 into 300 000 like there's the ooh that's gonna have a dent on the quarterlies style of number all right someone also asked what is the one thing that AWS could do that I believe would reduce costs for both AWS and their customers and no canceling reInvent doesn't out? I don't think about it in that way because believe it or not, most of my customers don't come to me asking to reduce their bill. They think they do at the start, but what they're trying to do is understand it. They're trying to predict, yes, they want to turn off the waste and the rest. But by and large, there are very few AWS offerings that you take a look at and realize what you're getting for it and say, that's too

expensive. It can be expensive for certain use cases. But the dangerous part is when the costs are unpredictable. What's it going to cost me to run this big application in my data center? The answer is usually, well, run it for a month and then we'll know. But that's an expensive and dangerous way to go about finding things out. I think that customers don't care about reducing costs as much as they think. They care about controlling them, predicting them, and understanding them. So how would they make things less expensive? I don't know. I suspect that data transfer, if they were to reduce that, at least cross AZ or eliminate it, ideally, you'd start seeing a lot more compute usage in multiple AZs. I've had multiple clients who are not spinning things up in multi-AZ specifically because

they'll take the reliability trade-off over the extreme cost of all the replication flowing back and forth. Aside from that, they mostly get a lot of the value right in how they price things, which I don't think that people have heard me say before, but it is true. Someone asked a question here of, are any major trends that I'm seeing in EDP slash PPA negotiations? Yeah. Lately, in particular, used to be that you would have a marketplace as the fallback, where it used to be that 50 cents of every dollar you spent on marketplace would count. Now it's 100% up to a quarter of your commit. Great. But when you have a long-term

commitment deal with Amazon, now they're starting to put all your other vendors onto the AWS Marketplace so you can have a bigger commit and thus a bigger discount, which incidentally, the discount does not apply to Marketplace spend. A lot of folks are uncomfortable with having Amazon as the middleman between all of their vendor relationships. And a lot of the vendors aren't super thrilled with having to pay percentages of existing customer relationships to Amazon for what they perceive to be remarkably little value.

That's the current one. I'm not seeing generative AI play a significant stake in this yet. People are still experimenting with it. I'm not seeing, well, we a significant stake in this yet. People are still experimenting with it. I'm not seeing, well, we're spending $100 million a year, but make that $150 because of generative AI. That's expensive to play with gen AI stuff, but it's not driving the business spend yet.

That's the big trend that I'm seeing over the past, I'd say, a few months. Do I use AWS for personal projects? The first problem there is, well, what's a personal project versus a work thing? My life is starting to flow in a bunch of weird different ways. The answer is yes.

Most of the stuff that I build for funsies is on top of AWS, though there are exceptions. Should I is the follow-up question. And the answer to that is it depends. The person's worrying about cost overruns. So am I. I tend to not be a big fan of uncontrolled downside risk when something winds up getting exposed. I think that there are going to be a lot of caveats there.

I know what I'm doing. And I also have the backstop in my case of, I figure I can have a big billing screw up where I have to bend the knee and apologize and beg for a concession from AWS once. It'll probably be on a billboard or something one of these days. Lord knows I have it coming to me. But that's something I can use as a get-out-of-jail-free card. Most people can't make that guarantee. And so I would take... Depending on the environment that you know and what you want to build, there are a lot of other options. Buying a fixed-fee VPS somewhere, if that's how you tend to think about things, might very well be cost-effective for you, depending on what you're building. There's no straight answer to this. state actors. No, I don't. And the reason behind that is that a lot of Azure spend is not necessarily Azure usage. It's being rolled into enterprise agreements. Customers negotiate as part of their

on-premise stuff, their operating system licenses, their office licensing, and the rest. The business world is not going to stop using Excel and Word and PowerPoint and Outlook. They're not going to stop putting Windows on desktop stuff. And largely, customers don't care about security. They say they do. They often believe that they do. But I see where the bills are. I see what people spend on feature development. I see what they spend on core infrastructure. And I see what they spend on security services. And I have conversations about budgeting with, what are you doing with a lot of these things? Companies generally don't care about this until right after they really should have cared.

And maybe that's a rational effect. I mean, take a look at most breaches. And a year later, their stock price is larger than it was when they disclosed the breach. Sure, maybe they're burning through their ablative CISO, but the business itself tends to succeed. I wish that there were bigger consequences for this. I have talked to folks who will not put specific workloads on Azure as a result of this. Will you talk about that publicly? No, because who can afford to upset Microsoft? I used to have guests from Microsoft on my show regularly. They don't talk to me and haven't for a couple of years. Scott Guthrie, the head of

Azure, has been on this show. The problem I have is that once you start criticizing their security posture, they go quiet. They clearly don't like me, but their options were basically to either ice me out or play around with my seven seats for office licensing, which, okay, whatever. They don't have a stick to hit me with in the way that they do most companies. And whether that's true or not, that they're going to lash out like that, companies don't want to take the risk of calling Microsoft out in public. Too big to be criticized is sort of how that works.

Let's see, someone else asks, how can a startup get the most out of its startup status with AWS? You're not going to get what you think you want from AWS in this context. Ooh, we're going to be a featured partner, so they market us. I've yet to hear a story about how being featured by AWS for something has dramatically changed the fortunes of a startup. Usually, they'll do that when there's either a big social mission, then you never hear about

the company again, or they're a darling of the industry that's taking the world by fire, and they're already at that upward swing. And AWS wants to hang out with those successful people in public and be seen to do so. The actual way that startup stuff is going to manifest itself well for you from AWS is largely in the form of credits as you go through Activate

or one of their other programs. But be careful, treat them like actual money, not this free thing you don't have to worry about. One day they expire or run out and suddenly you're going from having no dollars going to AWS to 10 grand a month, and people aren't prepared for that. It's, wait, you mean this costs money? Oh my god. You have to approach it with a sense of discipline. But yeah, if you can do that, yeah, free money and a free cloud bill for a few years, that's not nothing.

I also would question the idea of being able to ask a giant company that's worth a trillion and a half dollars on advice for how to be a startup. I find that one's always a little on the humorous side myself. What do I think is the most underrated service or feature release from 2023? Full disclosures, this means I'll make some content about it, says Brooke over at AWS. Oh, that's a good question. Try to remember when various things have come out, and it all tends to run together. I think that people are criticizing AWS

for charging for IPv4 an awful lot, and I think that that is a terrific change just because I've seen how wasteful companies are with public IP addresses, which are basically an exhausted or rapidly exhausting resource. And they just... You spend tens or hundreds of thousands of these things

and don't use reason to think about that. It'll be one of the best things that we've seen for IPv6 adoption once AWS figures out how to make that work. And I would say that there's a lot to be said for since IPv4 is exhausted already. Now we're talking about, can we get them on the secondary markets?

You need a reasonable IP plan to get some of those. And well, we just give them to customers and they throw them away. I want AWS to continue to be able to get those for the stuff that the rest of us are working on, not because one big company uses a million of them, just because, oh, what do you mean private IP addresses? What might those be?

That's part of it. I would say that there's also been, thinking back on this. It's on song, but Compute Optimizer is doing a lot better at recommending things than it used to be. It was originally just giving crap advice. And over time, it started giving advice

that's actually solid and backs up what I've seen. It's not perfect. And I keep forgetting it's there because for some godforsaken reason, it's its own standalone service rather than living in the billing console where it belongs. And no one's excited about a service like that to the point where they talk about or create content about it.

But it's good. And it's getting better all the time. That's probably a good one. They recently announced the ability for it to do GPU instances, which, okay, great. For people who care about that, awesome. But it's not exciting. Even I don't think I paid much attention to it in the newsletter.

Does it make economic sense to bring your own IP addresses to AWS instead of paying their fees? Bring your own IP, if you bring your own allocation to AWS, costs you nothing in terms of AWS costs. You take a look at the market rate per IP address versus what AWS costs. You'll hit break-even within your first year if you do it. So yeah, it makes perfect economic sense to do it if you have the allocation and if you have the resourcing, as well as the ability to throw people at the problem to do the migration. It can

be a little hairy if you're not careful, but the economics benefit is clear on that once you account for those variables. Let's see here. We've also got tagging. Everyone nods their heads. They know it's the key

to controlling things, but how effective are people at actually tagging, especially with new to cloud? They're terrible at it. They're never going to tag things appropriately. Automation's the way to do it

because otherwise, you're going to spend the rest of your life chasing developers and asking them to tag things appropriately and then they won't. And then they'll feel bad about it. No one enjoys that conversation. So having derived tags and the rest or failing that, having some deployment gate as early in the process as possible. Oh, what's the tag for this is the only way you're going to start to see coverage on this. And ideally, someday you'll go back and tag a bunch of pre-existing stuff. But

it's honestly the thing that everyone hates the most on this. I have never seen a company that says, we are thrilled with our tag coverage. We're nailing it. The only time you see that is pure greenfield, everything done without ClickOps, and those environments are vanishingly rare. Outside of telecom, are customers using local zones more or at all? Very, very limited as far as what their usage looks like on them. Because it doesn't buy you as much as you'd think for most workloads. The real benefit is it's a little bit more expensive, but it's also in specific cities where there are not AWS regions.

And at least in the United States, where the majority of my clients are, there is not meaningful latency differences. For example, from in Los Angeles versus up to Oregon, since no one should be using the Northern California region because it's really expensive. It's a 20 millisecond round trip, which in most cases for most workloads is fine.

Gaming companies are a big exception to this. Getting anything they can as close to the customer as possible is their entire goal, which very often means they don't even go with some of the cloud providers in some places. That's one of those actual multi-cloud workloads that you want to be able to run anywhere that you can get a baseline computer up to run a container or a golden image or something. That is the usual case. The rest

for local zones is largely going to be driven by specific one-off weird things. Good question. Let's see. Is S3 Intelligent Tiering good enough or is it worth trying to do it yourself? Your default choice for almost everything should be Intelligent Tiering in 2023. It winds up costing you more only in very specific circumstances that are unlikely to be anything other than a corner case for what you're doing. And the exceptions to this are large workloads that are running a lot of S3 stuff where the lifecycle is very well understood, environments where you're not going to be storing your data for more than

30 days in any case, and you can do a lifecycle policy around it. Other than those use cases, yeah, the monitoring fee is not significant in any environment I've ever seen, and people touch their data a lot less than they believe. So, okay, there's a monitoring fee for object, yes, but it also cuts your raw storage

cost in half for things that aren't frequently touched. So, you know, think about it. Run your own numbers and also be aware that first month, as it transitions in, you're going to see massive transition charges per object. But once it's in intelligent tiering, there's no further transition charges, which is nice. Let's see here.

We're all in on serverless. Oh, good, someone drank the Kool-Aid too. And for our use cases, it works great. Do I find other customers moving to it and succeeding? Yeah, I do when they're moving to it. Because for certain workloads, it makes an awful lot of sense. For others, it requires complete reimagining of whatever it is that you're doing. The early successes were just doing these periodic jobs.

Now we're seeing full applications built on top of event-driven architectures, which is really neat to see. But trying to retrofit something that was never built with that in mind can be more trouble than it's worth. And there are quarter cases where building something on serverless would cost significantly more than building it in a serverful way. But it's time has come for an awful lot of stuff. Now, what I don't subscribe to is this belief that,

oh, if you're not building something serverless, you're doing it totally wrong. No, that is not true. That has never been true. Let's see. What else have we got here? Oh, following up on local zones, how about outposts?

Do I see much adoption? What's the primary use case or cases? My customers inherently are coming to me because of a large AWS bill. If they're running outposts, it is extremely unlikely that they are putting significant portions of their spend through the outposts. It tends to be something of a rounding error, which means I don't spend a lot of time focusing on it. They obviously have some existing data center workloads and data center

facilities where they're going to take an AWS-provided rack and slap it in there. But it's not going to be in the top 10 or even top 20 list of service spend in almost every case as a result, so it doesn't come up. One of the big secrets of how we approach things is we start with the big number first and then work our way down

instead of going alphabetically. So yes, I've seen customers using them and the customers I've talked to at reInventor using them are very happy with them for the use cases, but it's not a common approach. I'm not a huge fan of the rest. Someone said that Basecamp saved a million and a half a year by leaving AWS. I know you say repatriation isn't a thing people are doing, but has my view changed at all since you published that blog post? No, because everyone's asking me about Basecamp and its repatriation. And that's the only use case that they've got for this. Let's further point out that a million and a half a year

is not as many engineers as you might think it is when you wind up tying that all together. And now those engineers are spending time running that environment. Does it make sense for them? Probably. I don't know their specific context. I know that a million and a half dollars a year, even if they had to spend that for the marketing coverage that they're getting as a result of this, makes perfect sense. But cloud has never been about raw cost savings. It's about feature velocity. If you have a data center and you move it to the cloud, you're not going to recoup that investment for at least five years. Migrations are inherently expensive. It does not create the benefits that people often believe that they do. That becomes a painful problem for folks. I would say that there's a lot more noise than there are

real-world stories coming out about these things. Now, I do occasionally see a specific workload that is moved back to a data center for a variety of reasons, occasionally cost, but not always. And I see proof-of-concept projects that they don't pursue and then turn off. Some people like to call that a repatriation. No, all it is, we tried it, it didn't do what we wanted it to do, so we didn't proceed. If you try that with any other project, no one says, oh, you're migrating off of it. No, you're not. You tested it, it didn't do what it needed to do. I do see net new workloads going into data centers, but that's not the same thing. Let's see. Are the talks at reInvent worth it anymore? I went to a lot of

the early reInvents and haven't in about five years. I found back then that even the level 400 talks left a lot to be desired. Okay. I'm not a fan of attending conference talks most of the time just because there's so many things I need to do at all these events that I would rather spend the time building relationships and having conversations. The talks are going to be on YouTube a week later. So I would rather get to know the people building the service so I can ask them how to inappropriately use it as a database six months later than asking questions about the talk. Conferenceware is often a thing. Reinvent always tends to have an AWS employee on stage as well. And I'm not saying that makes these talks less authentic, but they're also not

going to get through slide review of, well, we tried to build this onto this AWS service, and it was a terrible experience. Let's tell you about that as a war story. They're going to shoot that down instantly, even though failure stories are so compelling about, here's what didn't work for us and how we got there. It's the lessons learned type of thing. Whenever you have as much control as reInvent exhibits over its speakers, you know that a lot of those anecdotes are going to be significantly watered down. This is not to impugn any of the speakers themselves.

This is the corporate mind continuing to grow to a point where risk mitigation and downside protection becomes the primary driving goal. Let's pull up another one from the prepared list here. My most annoying overpriced or unnecessary charge service in AWS. AWS Config. It's a tax on using the cloud as the cloud. When you have a high config bill, it's because it charges you every time you change the configuration of something you have out there. It means you're spinning up and spinning down EC2 instances, whereas you're going to have a super low config bill if you treat it like a big, dumb data center.

It's a tax on accepting the promises under which cloud has been sold, and it's necessary for a number of other things like Security Hub, Control Towers, Magic deploys it everywhere and makes it annoying to turn off. And I think that that is a pure rent-seeking charge. Because people aren't incurring config charges if they're not already using a lot of AWS things.

Not every service needs to make money in a vacuum. It's, well, we don't charge anything for this because our users are going to spend an awful lot of money on storing things in S3 to use our service. Great. That's a good thing. You don't have to pile charge upon charge upon charge upon charge. It drives me a little bit nuts. Let's see what else we have here as far as questions go. Which AWS service delights me the most? Depends on the week. S3 has always been a great service just because it winds up turning big storage that used to require a lot of maintenance and care to something I don't have to think about very much.

It's getting smarter and smarter all the time. The biggest lie is the simple in its name, simple storage service. At this point, if that's simple, I really don't want to know what you think complex would look like. By following me on Twitter, someone gets a lot of value from things I mention offhandedly as things everybody just knows.

For example, which services are quasi-deprecated or outdated, or what common practices are anti-patterns? Is there a way to learn this kind of thing all in one go, as in a website or a book that reduces AWS to, these are the handful of services everybody actually uses, and these are the most commonly sensible ways to do it. I wish. The problem is that a lot of the stuff that everyone knows, no, it's stuff that at most maybe half of the people who are engaging with it knew. They find out by hearing it from other people the way that you do

or by trying something and failing and realizing, ooh, this doesn't work the way that I want it to. It's one of the more insidious forms of cloud lock-in. You know how a service works, how a service breaks, what the constraints are around when it starts and it stops. And that becomes something that's a hell of a lot scarier when you have to realize, I'm going to pick a new provider instead and relearn all of those things. The reason I build things on AWS these days is honestly because I know the ways it sucks. I know the painful sharp edges. I don't have to guess where they might be hiding. I'm not saying that these

sharp edges aren't painful, but when you know they're there in advance, you can do an awful lot to guard against that. Do I believe the big two, AWS and Azure, cloud providers have agreed between themselves not to launch any price wars as they already have an effective monopoly between them and no one win in a price war? I don't know if there's ever an explicit agreement on that, but business people aren't foolish. Okay, we're going to cut our cost of a service instantly to undercut a competitor. Every serious competitor is going to do the same thing. The only reason to do that is if you believe your margins are so wildly superior to your competitors that you can drive them under by doing that, or if you have the ability to subsidize your losses longer than

they can remain a going concern, Microsoft and Amazon and Google are not in a position where, all right, we're going to drive them under. They can both subsidize losses basically forever on a lot of these things, and they realize it's a game you don't win in, I suspect. The real pricing pressure on that stuff seems to come from customers. When, all right, I know it's big and expensive up front to buy a SAN, but when that starts costing me less than S3 on a per petabyte basis, that's when you start to see a lot of pricing changing in the market. The one thing I haven't seen that take effect on is data transfer.

You could be forgiven for believing that data transfer still costs as much as it did in the 1990s. It does not. Is AWS as far behind in AI as they appear? I think a lot of folks are in the big company space, and they're all stammering, well, we've been doing this for 20 years. Great. Then why are all of your generative AI services, A, bad? B, why is Alexa so terrible? C, why is it so clear that everything you have pre-announced and not brought to market

was very clearly not envisioned as a product to be going to market this year until 300 days ago when ChatGipity burst onto the scene and OpenAI stole a march on everyone? Companies are sprinting to position themselves as leaders in the AI space, despite the fact that they've gotten lapped by basically a small startup at seven years old. Everyone is trying to work the word AI into things, but it always feels contrived to me. Frankly, it tells me that I need to start tuning the space out for a year

until things settle down and people stop describing metric math or anomaly detection as AI. Stop it. So yeah, I'd say if anything, they're worse than they appear as far as from behind goes. I mostly focus on AWS.

Will I ever cover Azure? There are certain things that would cause me to do that, but that's because I don't want to be the last Pearl consultancy as the entire world has moved off to Python. Effectively, my focus on AWS is because that's where the painful problems

I know how to fix live. But that's not a suicide pact. I'm not going to ride that down in flames. But I can retool for a different cloud provider if that's what the industry starts doing far faster than AWS can go from its current market-leading status to irrelevance. There are certain triggers that would cause me to do that.

But at the time, I don't see them in the near term and I don't have any plans to begin covering other things. As mentioned, people want me to talk about the things I'm good at, not the thing that makes me completely nonsensical. Which AWS services look like a good idea, but pricing-wise, they're going to kill you once you have any scale, especially the ones that look okay pricing-wise, but aren't really, and it's hard to know going in. CloudTrail data events,

S3 bucket access logging, any of the logging services really, manage NAT gateways in a bunch of cases. There's a lot that starts to get really expensive once you hit certain points of scale. With a corollary that everyone thinks that everything they're building is going to scale globally, and that's not true. I don't build things as a general rule with the idea that I'm going to get 10 million users on it tomorrow. Because by the time I get from nothing to substantial workloads, I'm going to have multiple refactors of what I've done. I want to get things out the door as fast as possible. And if that means that later in time, oh, I accidentally built Pinterest, what am I going to do? Well, okay, yeah, I'm going to need

to rebuild a whole bunch of stuff, but I'll have the user traffic and mind share and market share to finance that growth. Early optimization on stuff like this causes a lot more problems than it solves. Best practices and anti-patterns in managing AWS costs. For context, you once told me about a role that I had taken that you'd seen lots of companies try to create that role and then said that the person rarely lasts more than a few months because it just isn't effective. You were right, by the way. Imagine that. I sometimes know what I'm talking about. When it comes to managing costs, understand what your goal is here, what you're actually trying to achieve. Understand it's going to be a cross-functional work between people in finance and people in engineering. It is first and foremost an engineering problem. You learn that

at your peril. And making someone be the human gateway to spin things up means that they're going to quit basically instantly. Stop trying to shame different teams without understanding their constraints. Savings plans are a great example. They apply biggest discount first, which is what you want, less money going out the door to Amazon,

but that makes it look like anything with a low discount percentage, like any workload running on top of Microsoft Windows, is not being responsible because they're always on demand. And you're inappropriately shaming a team for something completely out of their control. There's a point where optimization no longer makes sense.

Don't apply it to Greenfield projects or Skunkworks things. You want to see if the thing's going to work first. You can optimize it later. Starting out with a step one, spend as little as possible is generally not a recipe for success. What else have we got here? I've seen some things fly by in the chat that are probably worth mentioning here. Some of it is just random nonsense, but other things are, I'm sure, tied to various questions here. With geopolitics shaping up to govern tech data differently in each country, does it make sense to even build a globally distributed B2B SaaS?

Okay, I'm going to tackle this one in a way that people will probably view as a bit of an attack. But it's something I see asked a lot by folks trying to come up with business ideas. At the outset, I'm a big believer in if you're building something, solve it for a problem and a use case that you intrinsically understand. That is going to mean the customers with whom you speak. Very often, the way business is done in different countries and different cultures means that in some cases, this thing that's a terrific idea in one country is not going to see market adoption somewhere else. There's a better approach to build

for the market you have and the one you're addressing rather than aspirational builds. I would also say that it potentially makes sense if there are certain things you know are going to happen, like, okay, we validated our market. And yeah, it turns out that we're building an image resizing site. Great. People in Germany and in the US all both need to resize images. But you know, going in that there's going to be a data residency requirement. So architecting from day one with an idea that you can have a partition that winds up storing its data separately is always going to be to your benefit. I find aligning whatever you're building with the idea of not being creepy

is often a great plan. And there's always the bring your own storage approach. Right. As a customer, you can decide where your data gets stored in your account. Charge more for that, sure. But then that becomes their problem. Anything that gets you out of the regulatory critical path is usually a good idea. But with all the problems I would have building a business, that is so far down the list for almost any use case I could ever see pursuing

that it's just one of those... You have a half hour conversation with someone who's been down that path before, if you think it might apply to what you're doing, but then get back to the hard stuff. Worry on the first two or three steps rather than step 90, just because you'll get there eventually. You don't want to make your future life harder, but you also don't want to spend all your time optimizing early before you validate you're actually building something useful. What unique feature of AWS do I most want to see on other cloud providers and vice versa? The vice versa is easy. I love that Google Cloud by default has everything in this project, which is their account equivalent, can talk to everything else, which means that humans aren't just allowing permissions to the universe because it's hard.

And I also like that billing is tied to an individual project. Terminate all billable resources in this project as a button click away, and that's great. Now, what do I wish other cloud providers would take from AWS? Quite honestly, the customer obsession. It's still real. I know it sounds like it's a funny talking point or that people who talk about this the most tend to be cultists, but they care about customer problems. Back when no one had ever heard of me before, my AWS bill was seven bucks. Whenever I had a problem with a service, and I talked about this in passing to folks, Amazonians showed up out of nowhere to help make sure that my problem got answered, that I was taken care of, that I understood what I was misunderstanding,

or in some cases, the feedback went to the product team. I see too many companies across the board convinced that they themselves know best about what customers need. That occasionally can be true, but not consistently. When customers are screaming for something, give them what they need, or frankly, get out of the way so someone else can.

I mean, I know someone's expecting to name a service or something, but we've gotten past the point to my mind of trying to do apples to oranges comparison in terms of different service offerings. If you want to build a website using any reasonable technology, there's a whole bunch of companies now that have the entire stack for you. Pick one. Have fun. We've got time for a few more here. Also, feel free to drop more questions in. I'm thrilled to wind up answering any of these things. Here's one about Babelfish, for example,

from Justin Brodley. Have I seen anyone using Babelfish, for example, from Justin Brodley. Have I seen anyone using Babelfish in the wild? It seems it was a great idea that didn't really work or had major trade-offs. It's a free open-source project that translates from one kind of database SQL to a different kind of database SQL. There have been a whole bunch of attempts at this over the years, and in practice, none of them

have really panned out. I have seen no indications that Babelfish is different. If someone at AWS who works on this or is a customer using Babelfish and say, wait, that's not true, please tell me. Because all I'm saying is I have not seen it and I don't expect that I will,

but I'm always willing to be wrong. Please, if I say something at some point that someone disagrees with, please reach out to me. I don't intend to perpetuate misinformation. Purely hypothetically. Yeah, it's always great to ask things hypothetically. In the companies I work with, which group typically manages purchasing savings plans? The ops team? Finance? Some mix of both? It depends. The sad answer is, what's a savings plan? Ask the company, and then we have

an educational path to go down. Often it is individual teams buying them ad hoc, which can work, cannot, as long as everyone's on the same page. Central planning in a bunch of companies past a certain point of sophistication is where everything winds up leading to. And that is usually going to be a series of discussions, ideally run by that group in a cross-functional way. They can be cost engineering. They can be optimization engineering. I've heard it described in a bunch of different ways.

But that is increasingly as the sophistication of your business and the magnitude of your spend increases, the sophistication of how you approach this should change as well. Early on, it's often just some VP of engineering at a startup like, that's a lot of money running the analyzer and clicking the button to buy what it says. That's not a bad first pass attempt. And then I think getting smaller and smaller buys as you continue to proceed, means you can start, it no longer becomes the big giant annual decision,

instead it becomes part of a frequently used process. That works pretty well too. Is there anything else that I want to make sure I get to before we wind up running this down? To the folks in the comments, this is your last chance to throw random awkward questions my way. Is there anything else that I want to make sure I get to before we wind up running this down? To the folks in the comments, this is your last chance to throw random awkward questions my way. I'm thrilled to wind up taking any slings, arrows, etc. that you care to throw my way. Going once, going twice style.

What is the most esoteric or shocking item on the AWS bill that you ever found with one of your customers? All right, it's been long enough and I can say it without naming the customer, so that'll be fun. My personal favorite was a high five-figure bill for Route 53. I joke about using Route 53 as a database. It can be, but there are better options. I would say that there are a whole bunch of use cases for Route 53, and it's a great service, but when it's that much money, it occasions comment. It turned out that we had

discovered, in fact, a data exfiltration in progress, which made it now a rather clever security incident. And this call will now be ending for the day, and we're going to go fix that. Thanks. It's like, I want a customer testimonial on that one. But for obvious reasons, we didn't get one. But that was probably the most shocking thing.

The depressing thing that I see the most, and this is the core of the cost problem, is not when the numbers are high. It's when I ask about a line item that drives significant spend and the customer is surprised. I don't like it when customers don't know what they're spending money on. If your service surprises customers when they realize

what it costs, you have failed. Because a lot of things are expensive and customers know that and they're willing to take the value in return for the cost. That's fine. But tricking customers does not serve anyone well, even your own long-term interests i promise have i ever had to reject a potential client because they had a tangled mess that was impossible to tackle or is there always a way it's never the technology that will cause us not to pursue working with a given company what will is like if you go to our website at duckbillgroup.com, you're not going to see a buy here button where you add one consulting please to your shopping cart and

call it a day. It's a series of conversations. And what we try to make sure is what is your goal? Who's aligned with it? What are the problems you're having in getting there? And what does success look like? Who else is involved in this? And it often becomes clear that people don't like the current situation, but there's no outcome with which they would be satisfied. Or they want something that we do not do. For example, we want you to come in and implement all of your findings. We are advisory. We do not know the specifics of your environment and or your deployment processes or the rest. We're not an engineering shop. We charge fixed fee. And part of the way we can do that

is by controlling the scope of what we do. Well, yeah, we have some AWS bills we really care about is our GCP bill or our data dog bill. Great. We don't focus on either of those things. I mean, I can just come in and sound confident, but that's not what adding value as a consultant is about.

It's about being authoritatively correct. Great question, though. How often do I receive GovCloud cost optimization requests? Does the compliance and regulation that these customers typically have keep them from making the needed changes? It doesn't happen often. And part of the big reason behind that is that if you're in GovCloud, it's probably because you are a significant governmental entity. There's not a lot of private sector in GovCloud for almost every workload. Yes, there are exceptions. We don't tend to do a whole lot with them. And the government

procurement process is a beast. We can sell and service three to five commercial engagements in the time it takes to negotiate a single GovCloud agreement with a customer. So it just isn't something that we focus. We don't have the scale to wind up tackling that down. Let's also be clear that in many cases, governments don't view money the same way as enterprise, which in part is a good thing. But it also means that this cloud thing is too expensive is never the stated problem. Good question. Waffles or pancakes is another one. I tend to go with eggs, personally. It just feels like empty filler in the morning. I mean, you could put syrup on anything if you're bold enough. So if it's just a syrup delivery vehicle, there are other paths to go. And I believe we might have exhausted the question pool. So I want to thank you all

for taking the time to talk with me. Once again, I am cloud economist Corey Quinn, and this is a very special live episode of Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review wherever you can, or a thumbs up, or whatever it is. Like and subscribe, obviously. Whereas if you've hated this podcast, same thing, five-star review, but also go ahead and leave an insulting comment, usually around something I've said about a service that you deeply care about because it's tied to your paycheck. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill

Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.