Screaming in the Cloud - The re:Invent Wheel in the Sky Keeps on Turning with Pete Cheslock

Episode Date: January 18, 2022

About PeteI enjoy improving companies organizational structures, providing insight into building and growing autonomous high functioning, high performing technical teams. I'm fascinated by the dynamics of high performance, and take great pride in building and supporting those teams. I also enjoy the intricacies of Systems Architecture, Design, and Implementation work. I like to use modern tools to solve difficult technology problems. I'm most excited by Automation, Observability, Data Engineering.  I'm a product minded technologist. For the last 20 years working from Internet Service Providers and Hosting Companies to modern SaaS hosted on Cloud providers. I like to understand how people use the products that I build, and I like to build things that last a long time.I consider product needs, business requirements, and technical capabilities when building products or planning new features. I work to understand the user and how and why they consume a service. All of our actions can impact many different ways, and I enjoy understanding how services, product teams, and business units work. I like to find ways to take one team's success and apply it more broadly, leveling up the entire business.  I like to get things done. I'm not too fond of unnecessary processes that slow down progress. I like iterative improvements, bringing new features into users' hands as quickly as possible, even if they are tiny changes. I want to share what I learn—both internal to a company and external to a broader community. I enjoy the business side of technology as much as the technical side. I went back to school and received my MBA to understand the language of business. I enjoyed my product and finance classes the most. I like to understand the financial impact of product decisions. I don't like waste (in time or money), and I also believe premature optimization is the root of all evil.Links:Last Tweet in AWS: https://lasttweetinaws.comTwitter: https://twitter.com/petecheslockLinkedIn: https://www.linkedin.com/in/petecheslock/

Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production.

I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you and watch for the wits. Using one of the vanilla cloud caching services, these folks have you covered with the go-to managed Redis service for global caching and primary database capabilities, Redis Enterprise. Set up a meeting with a Redis expert during reInvent, and you'll not only learn how you can become a Redis hero,

but also have a chance to win some fun and exciting prizes. To learn more and deploy not only a cache, but a single operational data platform for one Redis experience, visit redis.com slash hero. That's R-E-D-I-S dot com slash hero. And my thanks to my friends at Redis for sponsoring my ridiculous nonsense. Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined, as is tradition, for a post-reinvent wrap-up a month or so later, once everything has time to settle,

by my friend and yours, Pete Cheslock. Pete, how are ya? I am doing fantastic. New year, new me. That's what I'm going with. That's the problem. I keep hoping for that, but every time I turn around, it's still me.

And, you know, honestly, I wouldn't wish that on anyone. Exactly. I wouldn't wish you on me either, but somehow I keep coming back for this. So in 2020 or 2020, as the children say, reInvent was fully virtual and that felt weird. Then reInvent 2021 was a hybrid event, which, let's be serious here, is not really those things. They had a crappy online thing and then a differently crappy thing in person. But it didn't feel real to me because you weren't there. That is part of the reInvent traditions. There's a midnight madness thing.

There's a keynote where they announce a bunch of nonsense. And then Pete and I go and have brunch at the last day of reInvent and decompress and more or less talk smack about everything that crosses our minds. And you weren't there this year. I had to backfill you with Tim Banks. You know, the person that I backfilled you with here at the Duckbill Group as a principal cloud economist. You know, you got a great upgrade in hot takes, I feel like, with Tim. In other ways, too, but it's rude of me to say that to you directly.

So, yeah, his hot takes, I feel like, with Tim. In other ways, too, but it's rude of me to say that to you directly. So, yeah, his hot takes are spectacular. He was going to be doing this with me, except you cannot mess with tradition. You really can't. You can't. I'm trying to think how many, is this the third year? Third or fourth.

Yeah, it's at least three. Yeah, I, it was, I don't want to say I was sad to not be there because with everything going on, it was still, it's still weird out there. But I am always, I'm just that weird person who actually likes reInvent, but not for, I feel like the reasons people think. It's, again, I'm such an extroverted type person that it's so great to have these like, this serendipity to reinvent the people that you run into and the conversations that you have and prior like in 2019 i think was a great example

because that was the last one i had gone to um you know having so many conversations so quickly because everyone is there right it's like this this magnet that attracts technologists and venture capital and product builders and all this other stuff. It's all compressed into that five-day span, I think is the biggest part that makes it so great. And the fear in people's eyes when they see me. And it was fun. I had a pair of masks with me. One of them was a standard mask and no one recognizes anyone because masks. And the other was a printout of my ridiculous face, which was horrifyingly uncanny, but also made it very easy for people to identify me. And depending upon was a printout of my ridiculous face, which was horrifyingly uncanny, but also made it very easy

for people to identify me. And depending upon how social I was feeling, I would wear one or the other and it worked flawlessly. That was worth doing. They really managed to thread the needle as well

before Omicron hit, but after the horrors of last year. So I feel like if it were going on right now, it would not be going on right now. Yeah. I talk about really, yeah, really just hitting it timing-wise.

Like, not that they could have planned for any of this, but like, as things were kind of not too crazy and before they got all crazy again, it's just like, wow, like, you know, they really couldn't have done the event at any other time. And it's like purely due to luck. I mean, absolute 100%. That's the amazing power of frugality, because the reason it's. And it's like purely due to luck. I mean, absolute 100%. That's the amazing power of frugality because the reason it's then is it's the week after Thanksgiving every year when everything is dirt cheap. And you know, if there's one thing that a 1.7, sorry, there's stocks in the toilet, a $1.6 trillion company is very concerned about,

it's saving money at every opportunity. Well, the one thing that I was most curious about, so I was at the first reInvent in 2012, I think it was. And there was, it was quaint, right? There was 4000 people there, I want to say it was in the thousands of people. Now, granted, still a big conference, but it was in the SANS Convention Center. It was in that giant room, the same number of people, where people's booths were like tables, like eight by 10 tables, right? It had almost a DevOps Days feel to it. And I was kind of curious if this one had any of those feelings. Like, did it evoke it being more quaint and personable, or was it just as soulless as it probably has been in recent years?

This was fairly soulless because they reduced the footprint of the event. They dropped from two expo halls down to one. They cut the number of venues, but they still had what felt like 20,000 people or something there. It was still crowded. It was still packed. And I've done some diligent follow-ups afterwards, and there have been very few cases of COVID that came out of it. I quarantined for a week in a hotel so I don't come back and kill my young kids for the wrong reasons. And that was sort of like the worst part of it on some level where it's like, great, now I can sit alone at a hotel and do some catch up

and all the rest. But all right, I'd kind of like to go home. I'm not used to being on the road that much. Yeah, I think we're all a little bit out of practice. I haven't been on a plane in years. I mean, the travel I've done more recently has been in my car from point A to point B, like a direct thing. Actually, a good friend of mine who's not in technology at all

had to travel for business. And he also has young kids who are under five five so when he got back he actually hid in a room in their house and quarantined himself in the room but they uh i thought this was kind of funny they never told the kids he was home uh because they knew that like the house was haunted like they don't go in the west wing sort of level of nonsense that that is kind of amazing honestly like like we were hanging out with the family because they're our neighbors.

And it was like, oh yeah, like he's in the guest room right now. Kids have no idea. I'm like, oh my God, like I can't even imagine. Yeah. So let's talk a little bit about the releases of reInvent. And I'm going to lead up with something that may seem uncharitable, but I don't think it necessarily is.

There weren't the usual torrent of new releases for ridiculous nonsense in the same way that there have been previously. There was no, this service talks to satellites in space.

I mean, sure, there was some IoT stuff to manage fleets of cars and giant piles of robots. And cool, I don't have those particular problems. I'm trying to run a website over here. So, okay, great. There were enhancements to a number of different services that were, in many cases, appreciated, in other cases, irrelevant.

Werner said in his keynote that it was about focusing on primitives this year. And why do we have so many services? It's because you asked for it as customers. Yeah, you asked for it. Pete, what have you been asking for, Pete? Because I know what I've been asking for, and it wasn't that. It is, it's amazing to see a company continually say yes to everything.

And somehow, despite their best efforts, be successful at doing it, no other company could do that. Imagine any other software technology business out there that just builds everything the customers ask for. That's like, from a product management business standpoint, that is like, rule 101 is listen to your customers, but don't say yes to everything.

Most companies can't navigate the transition between offering the same software in the cloud and on a customer facility. So it's like, ooh, an on-prem version. I don't know, that almost broke the company the last time we tried it. Whereas you have Amazon, whose product strategy is, yes, being able to put together a whole bunch of things. I also will challenge the assertion that it's the primitives that customers want. They don't want to build a data center out of popsicle sticks themselves. They want to get something that solves a problem. And this has been a long-term realization for me.

I used to work at Media Temple as a senior systems engineer running WordPress at extremely large scale. My websites now run on WordPress, and I have the good sense to pay WP Engine to handle it for me instead of doing it myself because it's not the most productive use of my time. I want things higher up the stack. I assure you, I pay more to WP Engine than it would cost me to run these things myself from an infrastructure point of view,

but not in terms of my time. What I see sometimes is the worst of all worlds is that AWS is trying to charge for that value-added pricing without adding the value that goes along with it. Because you've still got to build a lot of this stuff yourself. It's still a very janky experience.

You're reduced to Googling random blog posts to figure out how this thing is supposed to work. And the best documentation comes from externally. Whereas with a company that's built around offering solutions like this, great. In the fullness of time, I really suspect that if this doesn't change, their customers are going to just be those people who build solutions out this, great. In the fullness of time, I really suspect that if this doesn't change, their customers are going to just be those people

who build solutions out of these things and let those companies capture the up-the-stack margin, which I have no problem with, but they do because Amazon is a company that lies awake at night, actively worrying that someone somewhere who isn't them might possibly be making money somehow. I think MongoDB is a perfect example of,

like, look at their stock price over the last whatever years. Like, they, I feel like everyone called for the death of MongoDB every time Amazon came out with their new things. Yet, they're still a multi-billion dollar company because I can just, give me an API endpoint and you scale the database. Look at all the high profile hires that Mongo is making out of AWS. And I can't shake the feeling they're sitting there going, yeah, who's losing important things out of production now?

It's everyone is exodus in there. I did one of those ridiculous graphics with the naming, all the people that went over there and in the, with the, with the hurricane evacuation traffic picture. And there's one car going the other way went over there and with the hurricane evacuation traffic picture, and there's one car going the other way that I just labeled with reinvent sponsorship check because yeah, they had a top tier sponsorship and it was great. I've got to say, I've been pretty down on MongoDB for a while for a variety of excellent reasons based upon more or less how they treated customers who were in pain. And I'd mostly written it off. I don't do that anymore.

Not because I inherently believe the technology has changed, though I'm told it has, but by the number of people who I deeply respect who are going over there and telling me, no, no, this is good. Congratulations. I've often said you cannot buy authenticity and I don't think that they are,

but the people who are working there, I do not believe that these people are, but the people who are working there, I would, I do not believe that these people are, yeah, well, you bought my opinion. You can buy their attention, not their opinion. If someone changes their opinion based upon where they work, I kind of question everything they're telling me. It's like, oh, you're just here to sell something you don't believe in. Welcome aboard. Right. Yeah. There's a, there's an interview question I like to ask, which is what's something that you used to believe in very strongly that you've more recently changed your mind on? And out of politeness, because it usually throws people back a little bit and they're like, oh, wow, let me think about that.

And I'm like, okay, while you think about that, I want to give you mine. Yeah. Which is, in the past, my strongly held belief was we had to run everything ourselves. You'd own your availability was the line. No, I'm not buying Datadog. I can build my own metric stack just fine. Thank you very much. No, I'm not going to use these outsourced load balancers or databases because I need to own my availability. And what I realized is that all of those decisions led to actually delivering and focusing on things that were not the core product.

And so now I've really flipped 180. That if anything that you're building that does not directly relate to the core product, i.e. how your business makes money, should 100% be outsourced to an expert that is better than you. Mongo knows how to run Mongo better than you. What does your company do?

Oh, we handle expense reports. Oh, what are you working on this month? I'm building a load balancer. It's like, that doesn't add the value. Don't do that. Right, exactly. And so it's so interesting, I think, to hear Werner say that, you know,

we're just building primitives and you asked for this. And I think that concept maybe would work years ago when you had a lot of builders who needed tools. But I don't think we have any, like, we don't have as many builders as before. It's like, I think we have people who need more complete solutions. And that's probably why all these businesses are being super successful against Amazon. I'm wondering if it comes down to a cloud economic story, specifically that my cloud bill is always going to be variable and it's difficult to predict.

Whereas if I just use EC2 instances and I build load balancers or whatnot myself, well, yeah, it's a lot more work, but I can predict accurately what my staff compensation costs are more effectively than I can predict what a CapEx charge would be or what the AWS bill is going to be. I'm wondering if that might in some ways shape it.

Well, I feel like how people get better in managing their costs, right? You'll eventually move to a world where like, yep, okay, first we turned off waste, right? Step one is waste. Steps two is understanding your spend better to optimize. But like step three, like the, you know, galaxy brain meme of Amazon cost stuff is all like unit economic stuff where trying to better understand the actual cost deliver an actual feature. And yeah, I think that actually gets really hard when you give kind of spread your product across like a slew of services that have varying levels of cost, varying levels of tagging so you can attribute it. Like, it's really hard. Honestly, it's pretty easy. If I have 1,000 EC2 servers with very

specific tags, I can very easily figure out what cost to deliver a product. But if I have... If I have CoriBuild, I know what Cori's going to cost, and I know how many servers he's going to use. Great. If I have PeteBuild, Pete's good at things. It'll cut that server bill in half because he actually knows how to wind up being efficient with things. Okay, great. You can start calculating things out that way.

I don't think that's an intentional choice that companies are making, but I feel like that might be a natural outgrowth of it. Yeah. And there's still, I think, a lot of the old school mentality of the not invented here. We have to own our availability. You can still own your availability by using these other vendors. And honestly, it's it's really heartening to see so many companies realize that and realize that I don't need to get everything from Amazon. And honestly, like, in some things, like I look at a cloud Amazon bill, and I think to myself, it would be easier if you just did everything from Amazon versus having these 10 other vendors. But those 10 other vendors are going to be a lot better at running the product that they build as a service than you probably will be running it

yourself or even Amazon's interpretation of that product. A few other things that came out that I thought were interesting, at least the direction they're going in, the changes to S3's intelligent tiering are great with instant retrieval on Glacier. I feel like that honestly was, they talk a good story, but I feel like that was a competitive response to Google offering the same thing. That smacks of a large company with its use case saying, you've got two choices here. And they're like, well, okay, crap, we're going to build it then. Or alternately, looking at the changes that they're making to intelligent tiering, they're now shifting that to being the default that as far as recommendations go, there are a couple of drawbacks to it, but not many. And it's, it's getting easier now to not have the mental

overhead of trying to figure out exactly what your life cycle policies are. Yeah. There are some quarter cases where, okay, if I adjust this just so, then I could save 10% on that monitoring fee or whatnot. Yeah, but look at how much work that's going to take you to curate and make sure that you're not doing something silly. That feels like it is such an in the margins issue. It's like, well, how much data are you storing?

Four exabytes. Okay, yeah, you probably want some people doing exactly that, but that's not most of us. is purely in optimizing our data lifecycle for that data. Till a point, right? Till you've optimized it, the 80%, basically. You optimize the first 80, that's probably, air quote, easy. The last 20 is going to be incredibly hard. Maybe you never even do that. But at lower levels of scale,

I don't think the economics actually work out to have a team managing your data lifecycle of S3. But the fact that now AWS can largely do it for you in the background. Now, there's so many things you have to think about and understand even what your data is there because not all data is the same. And since S3 is basically a big giant database you can query, you got to really think about some of that stuff. But honestly, what I... I don't know if... I have no idea if this is even being worked on, but what I would love to see, you know, hashtag AWS wishlist is now we have countless tiers of EBS volumes,

EBS volumes that can be dynamically modified without touching the, you know, the physical host. Meaning with an API call, you could change from the GP2 to GP3 or IO whatever, right? Or back again if it doesn't pan out. Or back again, right? And so for companies with large amounts of spend, you know, the economics makes sense that you should have a team that is analyzing

your volume's usage and modifying that daily, right? Like, you can modify that daily and I don't know if there's anyone out there that's actually doing it at that level yet. And they probably should. Like if you've got millions of dollars in EBS, like there's legit savings that you're probably leaving on the table without doing that. But that's what I'm waiting for Amazon to do for me, right? I want intelligent tiering for EBS. Because if you're telling me I can API call and you'll move

my data and make that better. It could be like your auto scaling for DynamoDB, for example. It gives you the capacity you need 20 minutes after you needed it, but fine, whatever. Because if I can schedule stuff like that, great. I know what time of day the runs are going to kick off that beat up the disks. I know when end of month reporting fires off. I know what my usage pattern is going to be by and large. Yeah. Part of the problem too is that I look what my usage pattern is going to be by and large. Yeah.

Part of the problem too is that I look at this stuff and I get excited about it with the intelligent tiering. At the Duckbill Group, we've got a few hundred S3 buckets lurking around and thinking, all right, I've got to go through

and do some changes on this and implement all of that. Our S3 bill is something like 50 bucks a month or something ridiculous like that. It's a, no, that really isn't a thing. I have a screenshot bucket

that I have an app installed, I think called Dropshare, that hooks up's a, no, that really isn't a thing. Like I have a screenshot bucket that I have an app installed, I think called Dropshare that hooks up to anytime I hit a shortcut, I drag with the mouse to select whatever I want and boom, it's up there. And the URL is now copied to my clipboard. I can paste that wherever I want. And I'm thinking like, yeah, there's no cleanup on that. There's no lifecycle policy that's turning into anything. I should really go back and age some of it out and do the rest and start doing some lifecycle management. I've been using this thing for years and I think it's now a whopping, what,

20 cents a month for that bucket? I just don't care. Other than this voice in the back of my head, that's an unbounded growth problem. Cool, when it hits 20 bucks a month, then I'll consider it. But until then, I just don't, it does not matter. Yeah, I think, yeah, scale changes everything. Start adding some zeros and percentages turn into meaningful numbers.

And honestly, back on the EBS thing, the one thing that really changed my perspective of EBS in general is, especially coming from the early days, right? One terabyte volume, it was a hard drive in a thing it was a virtual lun and a sand somewhere probably um nowadays and even like many years after those original ebs volumes like all the limits you get in ebs those are actually artificial limits right uh if you're like my my ebs volume is too slow it's not because like the hard drive it's on is too slow that's an artificial limit that is likely put in place due to your volume choice and so like once you realize that in your head then your concept of how you store data on ebs should change dramatically oh it was at a blog post recently talking about like with io2 and the limits and everything and there was an architecture thing

okay so let's say this is insufficient and the quarter million IOPS a second that you're able to get is not there. And I'm sitting there thinking, that is just ludicrous data volume and data interactivity model. And it's one of those, like I'm sitting here trying to think about, like I haven't had to deal with a problem like that

in a decade just because it's, huh, turns out getting this one thing that's super fast is kind of expensive. If you paralyze it out, that's usually the right answer. And that's how the internet has mostly evolved. But there are use cases for which that doesn't work. And I'm excited to see it. I don't want to pay for it, mind you, but it's nice to see it. Yeah, it's kind of fun to go into the Amazon calculator and price out one of the like IO2 volumes and like maxed out. It's like, I don't know, like 50,000 a month or a hundred, like it's some just absolutely

absurd number. But the beauty of it is that if you needed that volume for an hour to run some intensive data processing task, you can have it for an hour and then just kill it when you're done. Right? Like that is what is most impressive. Oh yeah. I copied 130 gigs of data to an EFS volume, which was a while ago, which EFS has gone from, this is a piece of junk to one of my favorite services. It really is, just because of its utility and different ways of doing things. I didn't have the foresight

to use a second EFS volume for this. So it was unzipping a whole bunch of small files onto it. Great. It took a long time for me to go through it. All right, now that I'm done with that, I want to clean all this up. My answer was to ultimately spin up a compute node and wind up running a whole bunch, like 400 simultaneous RMRFs on that long thing. And it was just like, this feels foolish and dumb, but here

we are. And I'm looking at the stats on it because the instance was, all right, at that point, the load average in the instance was like 200 or something like that. And the EFS volume was like, ooh, wow, you're really churning on this. I'm now at like 5% of the limit. Like, okay, great. It turns out I'm really bad at computers. Yeah, well, that's really the trick. It's like, yeah, sure, you can have a quarter million IOPS per second. But like, what's going to break before you even hit that limit?

Probably many other things. Oh, yeah. Like, it feels like on some level, something gets to that point. It's a misconfiguration somewhere. But honestly, that's. Probably many other things. Oh yeah. Like it feels like on some level, something gets to that point. It's a misconfiguration somewhere, but honestly, that's the thing I find weirdest about the world in which we live is that at a small scale, in my $5 a month shit posting account, great. If I screw something up and cost myself a couple hundred bucks in misconfiguration, it's going to stand out. At large scale, it doesn't matter if you're spending $50 million a year or $500 million a year on AWS and someone leaks your creds and someone spins up a whole bunch of Bitcoin miners

somewhere else, you aren't going to see that on your bill until they're mining basically all the Bitcoin. It just gets lost in the background. I'm waiting for those. I'm actually waiting for the next level of them to get smarter because maybe you have like an aggressive tagging system and you're monitoring for untagged instances. But the move here would be first get the creds and query for like the most used tags and start applying those tags to your Bitcoin mining instances.

Just clone a bunch of tags. Congratulations. You now have a second BI Elasticsearch cluster that you're running yourself. Good work of tags. Congratulations. You now have a second BI Elasticsearch cluster that you're running yourself. Good work. Yeah. Yeah. People won't find that

until someone comes along after the fact and be like, why don't we have two of these things? And you're like, must be a DR thing. That's now CPU. Yeah, exactly. Oh, the terrible ideas.

Please, hackers, don't take our terrible ideas. I had a whole thing I did on Twitter years ago talking about how I would wind up using the AWS marketplace for an embezzlement scheme. Namely,

I would just wind up spinning up something that had like a five cent an hour charge or whatnot on it. Just basically rebadged the CentOS community AMI or whatnot.

Great. And then write a blog post not attached to me that explains how to do a thing that I'm going to be doing in production in a week or two anyway. Like how to build an auto-scaling group and reference that AMI. Then when it comes, if it ever comes out, like, wow, why are we having all these marketplace

charges on this? I just followed the blog post like it said here. And it's like, oh, okay, you're a dumbass. In the end, that's the way to do it. A month goes by and suddenly it came out that someone had done something similar, only they wound up rebadging these community things on the marketplace and charging big money for it. And I'm sitting there going like, that was a joke. It wasn't a how-to. But yeah, every time I make these jokes, I worry someone's going to do it. Welcome to Large Scale Fraud with Corey Quinn. Oh yeah. Fraud at scale is really the important thing here. This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still

dreaming of deploying apps instead of hello world demos? Allow me to introduce you to Oracle's always free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And let me be clear here, it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always

like to put asterisk next to the word free. This is actually free. No asterisk. Start now. Visit snark.cloud slash oci-free. That's snark.cloud slash oci-free. I still remember a year ago now at reInvent 2021, was it? Or was it 2020? Whenever they came out with, I want to say, it wasn't GP3 or maybe it was. Regardless, there was a new EBS volume type that came out that you were playing with to see how it worked and you experimented with it. And the next morning, you looked at the, I checked Slack and you're like, well, my experiments yesterday cost us $5,000. And at first, my response is instructive on this because first it was, oh my God, what's going to happen now? And it's like, first,

hang on a second. First off, that seems suspect, but let's assume it's real. I assumed it was real at the outset. It's, oh, right. This is not my personal $5 a month toy box account. We are a company. We can absolutely pay that because it's like, I could absolutely reach out, call it a favor. I made a mistake and I need a favor on the bill, please. To AWS. And I would never live it down. Let's be clear. For a $7,000 mistake, I would almost certainly eat it as opposed to having to prostrate myself like that in front of Amazon. Like, no, no, no. I want one of those. Like, if it's like, okay, you're going to like set back the company roadmap by six months if you have to pay this. Do you want to do it? Like, fine, I'll eat some crow. But okay. And then followed immediately by, wow, if Pete of all people can mess this up,

customers are going to be doomed here. We should figure out what happened. And I'm doing the math. Pete, what did you actually do? And you're sitting there and you're saying, well, I had like a 20 gig volume that I did this. And I'm doing the numbers and it's like, how sure are you when you say gigabyte that you were, that that actually means what you think it did? Like,

were you off by a lot? Like, did you mean exabyte? Like what's the deal here? Multiple factors. Yeah. How, how much, how many IOFs did you give that thing, buddy? And it turned out what had happened was that when they launched this, they had mispriced it in the system by a factor of a million. So it was fun. I think by the end of it, all of your experimentation was somewhere between five to seven cents, which is why you don't work here anymore, because no one cost me seven cents of money to give to Amazon. I buy what? Get out.

Yeah. How dare you, sir? Yeah. That was amazing to see as someone who has done definitely made screw-ups that have cost real money you know s3 list requests are always a fun one at scale but that one was supremely fun to see the um that was a scary one because another one they'd done previously was they had messed up light sale pricing where people would log in and like okay so what is my light

sale instance going to cost? And I swear to you, this is true. It was back in 2017 or so. The answer is like $4.3 billion. Because when you see that, you just start laughing because you know it's a mistake. You know that they're not going to actually demand that you spend $4.3 billion for a single instance unless it's running SAP. And great. It's a laugh, and it's clearly a misprice, and it's clearly a bug that's going to get fixed. I just spun up this new EBS volume that no one fully understands yet, and it cost me thousands of dollars. That's the sort of thing that, no, no, I could actually see that happening. There are instances now that cost something like a hundred bucks an hour or whatnot to run. I can see spinning up the wrong thing by mistake and getting bitten by it. There's a bunch of fun configuration mistakes you can make that

will, he, he, he, why can't I see that bill spike from orbit? And that's the scary thing. It's the original CI and CD problem of the per hour billing. That was super common of like, yeah, like an i3 16XL server is pretty cheap per hour. But if you're charged per hour and you spin up a bunch for five minutes, you will be shocked by what you see there. Yeah, mistakes will show. And I get it. It's also people as individuals are very different psychologically than companies are. With companies, it's one of those, great, we're optimizing to bring in more revenue and we don't really care about saving money at all costs. Whereas people generally have something that looks a lot like a fixed income in the form of a salary or whatnot. So it is easier for us to cut spend than it is for us to go out and make more money. Like, I don't want to get a second job or pitch my boss on stuff. And yeah. So all in all,

the routing out the rest of what happened at reInvent, this is the problem, is that they have a bunch of minor things like SageMaker inference recommender. Yeah, I don't care. Anything from SageMaker, I mostly tend to ignore for safety. I did like the way they described Amplify Studio

because they made it sound like a WYSIWYG, drag and drop, build a React app. It's not. It basically, you can do that in Figma and then it can hook it up to some things in some cases. It's not what I want to, but it's not what I want it to be,

which is honey code except good. But we'll get there some of the year, maybe. There's a lot of stuff that was, you know, it's the classic like preview, which sure, like from a product standpoint, it's great. They have a level of scale where they can say, here's this thing we're building, which could be in just a twinkle in a product manager's eye, call it preview, and get thousands of people who would be happy to test it out and give you feedback.

And it's great that you have that capability. But I often look at so much stuff and I'm like, yeah, that's really cool, but can I have it now? Or you can't even get into the preview plan even though you have that specific problem. And it's largely just because either your scale isn't big enough or you don't have a good enough relationship with your account manager or, I don't know, countless other reasons. The thing that really throws me too is the pre-announcements

that come a year or so in advance, like the outposts, smaller ones are finally available. But it feels like when they do too many pre-announcements or no big marquee service announcements, as much as they talk about, we're getting back to fundamental. No, you have a bunch of teams that blew the deadline. That's really what it is. Let's not call it anything else. Another one that I think is causing trouble for folks, I'm fortunate in that I don't do much work with Oracle databases or Microsoft SQL databases, but they extended RDS custom to Microsoft SQL

at the AppSQL server at reInvent this year, which means when this comes down to things I actually use, we're going to have a problem. Because historically, the lesson has always been, if I want to run my own databases and tweak everything, I do it on top of an EC2 instance. If I want to manage relational database service, great, I use RDS. RDS custom basically gives you root into the RDS instance,

which means, among other things, yes, you can now use RDS to run containers, but it lets you do a lot of things that are right in between. So how do you position this? When should I use RDS to run containers, but it lets you do a lot of things that are right in between. So how do you position this? When should I use RDS custom? Can you give me an easy answer to that question? And they used a lot of words to say, no, they cannot. It's basically completely blowing apart the messaging and positioning of both of those services in some unfortunate ways. We'll learn as we go. Yeah. Honestly, it's like, why would I use this or how would I use this? And this is, I think, fundamentally what's hard when you just say yes to everything. It's like

they, in many cases, I don't think like, I don't want to say they don't understand why they're doing this, but if it's not like there's a visionary who's like, this fits into this multi-year roadmap, that roadmap is largely, if that roadmap is largely generated by the customers asking for it,

then it's not like, oh, we're building towards this North Star of RDS being whatever. You might say that, but your roadmap is probably getting moved all over the place because this company that pays you $1 billion a year is saying, I would give you $2 billion a year for all of my Oracle databases, but I need this specific thing. I can't imagine a scenario that they would say, oh, well, we're building towards this North Star, and that's not on the way there, right? They'd be like, new North Star,

another billion dollars, please. Yep. Probably the worst release of reInvent from my perspective is RUM, Real User Monitoring for CloudWatch. And to be clear, I wrote a shit-posting Twitter threading client called LastTweetinAWS. Go to LastTweetinAWS.com.

You can all use it. It's free. I just built this for my own purposes. And I've instrumented it with RUM. Now, Real User Monitoring is something that a lot of monitoring vendors use, and also CloudWatch now. And what that is, is it embeds a listener

into the JavaScript that runs on client load, and it winds up looking. And what that is, is it embeds a listener into the JavaScript that runs on client load. And it winds up looking at what's going on, loading times, et cetera, as you can see when users are unhappy. I have no problem with this other than that, you know, liking users.

What's up with that? But then, okay, now what this does is unlike every other ROM tool out there, which charges per session, meaning I am going to be doing a webpage load, which charges per session, meaning I am going to be doing a web page load, it charges per data item, which includes HTTP errors or JavaScript errors,

et cetera. Which means that if you have a high transaction volume site and suddenly your CDN takes a nap like Fastly did for an hour last year, suddenly your bill is stratospheric for this because errors abound and cascade. And you can have thousands of errors on a single page load for these things. And it is going to be visible from orbit, at least with a per session basis thing. When you start to go viral, you understand that, okay, this is probably going to cost me some more on these things.

And oops, I guess I should write less compelling content. Fine. This is one of those one misconfiguration away and you are wailing and gnashing teeth. Now, this is a new service. I believe that they will waive these surprise bills in the event that things like that happen. But it's going to take a while and you're going to be worrying the whole time if you've rolled this out naively. So it's, I just don't like the price. How many people will actively avoid that service, right? And honestly, choose a competitor because the competitor could be

five times more expensive, right? On face value. But it's the certainty of it. It's the uncertainty that of what Amazon will charge you. No one wants the surprise bill. Well, vendor is saying that they'll give us this contract for $10,000. I'm going to pay $10,000, even though rum might be a fraction of that price. It's honestly, a lot of these product analytics tools and monitoring tools, you'll often see they price via MAU, monthly active user, or some sort of user-based pricing, like the number of people coming to your site. And I feel like at least then, if you are trying to optimize for lots of users on your site, and more users means more revenue, then if your spend is going up, but your revenue is also

going up, that's a win-win. But if it's like someone... Your third-party vendor dies and you're spewing out errors, or someone upgraded something and it spews out errors that no one would normally see. That's the thing. Unless you're popping open that JavaScript console, you're not seeing any of those errors, yet somehow it's directly impacting your bottom line. There is something vaguely Machiavellian about that.

How do I get my developers to care about errors on the consoles? How about we make it extortionately expensive for them not to? It's all right, then. Here we go. And then talk about now you're in a scenario where you're working on things that don't directly impact

the product. You're basically just sweeping up the floor and trying to remove errors that maybe don't actually affect anything. They're not actually an error. Yeah, I really do wonder what the right answer is going

to be. We'll find out. Again, we live, we learn. But it's also, how long does it take a service that has bad pricing at launch or an unfortunate story around it to outrun that reputation? People are still scared of Glacier because of its original restore pricing, which was non-deterministic for any sensible human being. And in some cases led to, I used to spend in 20 to 30 bucks a month on this. Why was I just charged two grand? Right. Scare people like that. They don't come back. I'm trying to actually remember which service it is that, that was, um, that basically gave you an estimate, right? Like turn it on for a month and it would give you an estimate of how much this was going to cost you when billing started. It was either detective or guard duty.

Yeah, it was. Yeah, that's exactly right. It was one of those two. And honestly, that was unbelievably refreshing to see. Listen, you have the data, Amazon. You know what this is going to cost me. So don't make me spend all this time to go and figure out the cost. If you have all this data already, just tell me, right? And if I look at it and go, yeah, wow, like turning this on in my environment is going to cost me X dollars. Like, yeah, that's a trade-off I want to make. I'll spend that. But, you know, with some of the... And that is a little bit of a worry on some of the intelligent tiering on S3 is that the recommendation is likely going to be everything goes to intelligent tiering first.

It's the GP3 story. Put everything on GP3, then move it to the proper volume. Move it to an SC or an ST or an IO. Like, GP3 is where you start. And I wonder if that's going to be the S3 story. Except I went through a wizard yesterday to launch an EC2 instance, and it defaults on the free tier

to GP2, which does not thrill me. I also still don't understand for the life of me why in some regions the free tier is a T2 instance, and it defaults on the free tier to GP2, which does not thrill me. I also still don't understand for the life of me why in some regions the free tier is a T2 instance when T3 is available. My guess is that they've got some free, they got a bunch of T2s lying around. Well, one of the most notable announcements at reInvent that most people didn't pay attention to

is their ability now to run legacy instance types on top of Nitro, which really speaks to what's going on behind the scenes of, we can get rid of all that old hardware and emulate the old M1s on modern equipment. So, because you can still have that legacy ancient instance, but now we're able to wind up greening our data centers, which is part of their big sustainability push, with their sustainability pillar for the well-architected framework. They're talking more about what the green choices in cloud are, which is super handy,

not just because of the economic impact, because we could use this pretty directly to reverse engineer their various margins on a per-service or per-offering basis, which I'm not sure they're aware of yet, but oh, they're going to be. And that really winds up being a win for the planet, obviously, but also something that is, I guess, puts a little bit of choice on customers. The challenge I've got is with my serverless stuff that I build out, if I spend, the Google search I make to figure out what the most economic, the most sustainable way to do that is, is going to have a bigger carbon impact than the app itself. That seems to be something that is important at scale. But if you're not at scale, it's one of those that don't worry about it. Because let's face it, the cloud providers, all of them, are going to have a better sustainability story than you are running

this in your own data centers or on a Raspberry Pi that's always plugged into the wall. Yeah. I mean, you got to remember, Amazon builds their own power plants to power their data centers like that's that's the level they play at right um their their economies of scale are so entirely they're they're they're so entirely different than anything that you could possibly even imagine um so it's it's it's something that like i'm sure people will want to choose choose for um but uh you know if i i would honestly say like if we really cared about our computing costs and the carbon footprint of it, I would love to actually know the carbon footprint of all of the JavaScript trackers that when I go to various news sites and it loads the whatever thousands of trackers and

tracking me all over. What is the carbon impact of some of those choices that I actually could control as either a consumer or a business person? I really hope that it turns into something that makes a meaningful difference and it's not just greenwashing. But we'll see. In the fullness of time, we're going to figure that out.

They're also launching some mainframe stuff. That's great. I don't deal with a lot of customers that are doing things with that in any meaningful sense. There is no AWS 400, so all right. Yeah, I think, like, that's great. I don't deal with a lot of customers that are doing things with that in any meaningful sense. There is no AWS 400, so all right. Yeah, I think, honestly, like, I did talk to a friend of mine who's in a big, big old enterprise and has a mainframe,

and they're actually replacing their mainframe with Lambda. Like, they're peeling off, which is, like, a great move, taking the monolith, right, and peeling off the individual components of what it can do into, you know, these discrete Lambda functions, which I like a great move, taking the monolith, right? And peeling off the individual components of what it can do into these discrete Lambda functions, which I thought was really fascinating.

Again, it's a five year long journey to do something like that. And not everyone wants to wait five years, especially if their support's about to run out for that giant box in the giant warehouse. The thing that I also noticed, and this is probably the, I guess one of the, talk about swing and a miss on pricing. They have a, what is it? There's a, they have a VPC IP address

manager, which tracks the VPC, the IP addresses assigned to your VPCs that are allocated versus not. And it's 20 cents a month per IP address. It's like, okay, so you're competing against a Google sheet or an Excel spreadsheet, which is what people are using for these things now, only you're making it extortionately expensive. Yeah, what kind of value does that provide for 20, I mean, like, again... It's Infoblox or someone like that

that offers it, where they become less, they become more cost-effective as soon as you hit 500 IP addresses. And it's just, like, this is what I'm talking about. Like, I know it does not cost AWS that kind of money to store an IP address. And it's just, like, this is what I'm talking about. Like, I know it does not cost AWS that kind of money to store an IP address. You can store that in a Route 53 text record for less money, for God's sake. And that's one of those,

like, ah, we can extract some value pricing here. Like, I don't know if it's a good product or not. Given its pricing, I don't give a shit because it's going to be too expensive for anything beyond trivial usage. So it's swinging amiss from that perspective. It's just looking at that, I laugh and I don't look at it again. And I'm not hugely price

sensitive. I want to be clear on that. It's just that is just loony tuned clown shoes pricing. Yeah, it's honestly like in many cases, I think the thing that I have seen, you know, in the past few years is in many cases, it can honestly feel like Amazon is nickel and diming their customers in so many ways. The explosion of making it easy to create multiple Amazon accounts has a direct impact to waste in the cloud because there's a lot of stuff you have to have per account. And the more accounts you have, those costs grow exponentially as you have these different places. You lose out on the economies of scale when you have these different places. You kind of lose out on the economies of scale when you have a smaller number of accounts. And yeah, it's hard to optimize for that. If you're trying to reduce your spend, it's challenging to say, well, by making a change here,

we'll save $10,000 in this account. That doesn't seem like a lot when we're spending millions. Well, hold on a second, you'll save $10,000 per account and you have 500 accounts or you have 1,000 accounts or something like that. Or almost cost avoidance of this cost is growing unbounded in all of your accounts. It's tiny right now. So now would be the time you want to do something with it. But again, for a lot of companies that have adopted the practice of endless Amazon accounts, they've almost gone... It's the classic, I've got 8,000 GitHub repositories for my source code. That feels just as bad as having one GitHub repository for your repo. I don't know what the balance is there. But anytime these different types of services come out, it feels like,

oh, wow, I'm going to get nickel and dime for it. This ties into the repost launch, which is the rebranding of their forums, where, okay, great. It was a little crufty and it needs to modernize, but it still ties your identity to an IAM account or the root email address for an Amazon account, which is great.

This is completely worthless because as soon as I change jobs, I lose my identity, my history, the rest on this forum. I'm not using it. It shows that there's a lack of awareness that everyone is going to have multiple accounts with which they interact and that people are going to deal with the platform longer than any individual account will. It's just a continual swing and a miss on things like that. And it gets back to the billing question of, okay, when I spin up an account, do I want them to just continue billing me because don't turn this off, this is important, or do I

want there to be a hard boundary where if you're about to charge me, turn it off, turn off the thing that's about to cost me money. And people hem and haw, like this is an insurmountable problem, but I think the way to solve it is let me specify that intent when I provision the account where it's, this is a production account for a bank. I really don't want you turning it off versus I'm a student learner who thinks that a managed NAT gateway might be a good thing. Yeah. I want you to turn off my demo hello world app that will teach me what's going on rather than surprising me with a five figure bill at the end of the month. Yeah. It's, Yeah, it shouldn't be that hard.

I mean, but again, I guess everything's hard at scale. Oh, yeah. But still, I feel like every time I log into Cost Explorer and I look at, and this is years, it's still not fixed. Not that it's even possible to fix, but on the first day of the month, you look at Cost Explorer and look at what Amazon is estimating your monthly bill is going to be. It's like, because of your, you know... Your support fees and your RI purchases and savings plans purchases. All those things happen, right?

First of the month. And it's like, yeah, your bill's going to be $800,000 this year. And it's like, shouldn't it be like $1,000? Like, you know, it's the little things like that. The one-off charges like, oh, you're route 53 zone and all the stuff that gets charged in a monthly cadence, which is fine, whatever. I mean, I'm okay with it, but it's also the like, be careful when that happens. I feel like there's a way to make that user experience less jarring. Yeah. Cause that, I mean, in my scenario, companies that I've worked at, there's been multiple times that a non-technical person will look at that data and go into immediate freakout mode.

And that's never something that you want to have happen because now that's just adding a lot of stress and anxiety into a company that is with inaccurate data. The answer you're giving someone is just wrong. Perhaps you shouldn't even give it to them if it's that wrong. Yeah, I'm looking forward to seeing what happens this coming year. We're already seeing promising stuff. They give people a little timeline

on how long in advance these things record. Late last night, AWS released a new console experience. When you log into the AWS console now, there's a new console experience. When you log into the AWS console now, there's a new beta thing. And I gave it some grief on Twitter because I'm still me, but I like the direction it's going. It lets you customize your view with widgets and whatnot. And until they start selling widgets on Marketplace or having sponsored widgets you can't remove, I like it, which is, you know, guarantee at some point. But it shows things

like I can move the cost stuff. I can move the outage stuff up around. I can have the things that are going on in my account, but who I am means I can shift this around. If I'm a finance manager, cool. I can remove all the stuff. It's like, hey, you want to get started spinning up an EC2 instance? Absolutely not. Do I want to get told like how to get certified? Probably not. Do I want to know what the current bill is and whether I list of favorites that I've pinned, whatever services are? Yeah, I absolutely do. This is starting to get there. Yeah. I wonder if it really is a way to start almost hedging on organizations, having a wider group of people accessing AWS. I mean, in previous companies, I absolutely gave access to the console for tools like QuickSight, for tools like Athena, for the data brew stuff, the glue data brew, giving non-technical people access to be able to do these UI ETL tasks. A wider group of a company is getting access into Amazon. So I think anything

that Amazon does to improve that experience for the non-SREs, like the people who would traditionally log in, that is an investment definitely worth making. Well, what could non-engineering types possibly be doing in the AWS console? I don't know, jackhole, maybe paying the bill? just a thought here. It's the, there are people who look at these things from a variety of different places and you are, you have such sprawl in the AWS world that there are different personas by a landslide. If I'm building Twitter for pets, you probably don't want to be pitching your mainframe migration services to me the same way that you would if I were a 200 year old insurance company.

Yeah, exactly. And the number of those products are going to grow, the number of personas are going to grow. And yeah, they'll have to do something if they want to actually maintain that experience so that every person can have the experience that they want and not be distracted. Oh, what's this? Let me go test this out. And it's like, you get a one-time charge for $10,000 because that's how it's charged. That's not an experience that people like. No, they really don't. Pete, I want to thank you for spending the time to chat with me again, as is our tradition. I'm hoping we can do it in person this year when we go at the end of the 2022 to reinvent again, or that no one goes in person, but this hybrid nonsense is for the birds.

Yeah, I very much would love to get back to another one. And yeah, like I think there could be an interesting kind of merging here of our annual reinvent recap slash live brunch, you know, stream, you know, hot takes after a long week. Oh, yeah. The real way that you know that it's a good joke is when one of us says something, the other one sprays scrambled eggs out of their nose. Yeah, that's the way to do it. Exactly. Exactly.

Pete, thank you so much. If people want to learn more about what you're up to, hopefully, you know, come back. We miss you. But you're unaffiliated. You're a startup advisor. Where can people find you to learn more if they, for some unforgivable reason, don't know who or what a Pete Cheslock is? Yeah, I think the easiest place to find me is always on Twitter. I'm just at Pete Cheslock. My DMs are always open and I'm always down to expand my network and chat with folks. And yeah, right now I'm just, as I jokingly say, professionally unaffiliated. I do some startup advisory work and have been largely just kind of

honestly checking out the state of the economy. Like there's a lot of really interesting companies out there and some interesting problems to solve. And, you know, trying to spend some of my time, you know, learning more about what companies are up to nowadays. So yeah, if you got some interesting problems, you know, you can follow my Twitter or go to LinkedIn if you want some great, you know, business hot takes about, you know, shit posting, basically. Same thing. Pete, thanks so much for joining me. I appreciate it. Thanks for having me. Pete Cheslock, startup advisor, professionally unaffiliated and recurring reInvent analyst pal

of mine. I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please give a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please give a five-star review on your podcast platform of choice, along with an angry comment calling me a jackass, because do I know how long it took you personally to price CloudWatch ROM? If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS. We tailor recommendations to your business and we get to the point.

Visit duckbillgroup.com to get started. This has been a HumblePod production. Stay humble.