Screaming in the Cloud - The Appalachian Cloud Trail: Hiking, Cloud Economics, and Finding Perspective

Episode Date: June 11, 2026

What happens when two cloud economists leave AWS  behind and spend six days hiking 60 miles on the Appalachian Trail? Corey Quinn sits down with Caleb Hurd to share stories from the trail, i...ncluding exploding sleeping pads, heroic shuttle drivers, lost phones, and the unique community that makes long-distance hiking special. Along the way, they draw surprising parallels between backpacking and cloud economics, discussing everything from serverless architecture and cloud cost optimization to the hidden challenges of on-prem infrastructure. It's a conversation about technology, adventure, perspective, and why sometimes the best way to solve complex problems is to step away from them entirely.Show highlights:(00:00) Why Hiking Hooks You(00:15) Meet Caleb on the Trail(01:31) Trail Miles and Ultralight Parallels(05:24) The Sleeping Pad Blowout(07:46) Shepherd Saves the Day(09:43) Trail Community and Cloud Community(11:07) Post Trail Perspective and Inside Jokes(15:35) Back to Work On Prem vs Cloud Pain(25:47) Server-less Spend and Lambda Sprawl(32:29) Wrap Up Where to Find CalebAbout Caleb: Caleb Hurd is a Cloud Economist at Duckbill, where he helps enterprises make sense of their cloud spend. Before moving to the cost side of the house, Caleb spent years in the trenches building and operating large-scale cloud environments and leading the engineering teams behind them across companies ranging from healthcare tech to enterprise Saas. He also founded CostOps.cloud, an AWS cost consulting practice, and is a vocal advocate for engineering-led FinOps — arguing that the people closest to the architecture should be the ones driving cost strategy, not spreadsheet jockeys in finance. Caleb holds a degree from Georgia Tech and made an unconventional journey into tech from a background in carpentry, which may explain his preference for building things over just talking about them. He's based in Atlanta.Links:LinkedIn: https://www.linkedin.com/in/calebrhurd/Sponsored by: duckbillhq.com

Transcript
Discussion (0)
Starting point is 00:00:00 I think, honestly, I love hiking. I've been section hiking the Appalachian Trail before I join me on this current leg. And I think what I love most about it is the community. Welcome to Screaming in the Cloud. I'm Corey Quinn, and today's guest has had just about enough of me. Caleb Hurd is many things. He's periodically a cloud economist here at Duckbill,
Starting point is 00:00:28 but he was also my hiking buddy for a 60-mile section hike of the Appalachian Trail. Caleb, thank you. for continuing to speak with me. You're welcome, Corey. You're welcome. It's a pleasure to be here. And I would say six days on the trail with you is very much like screaming in the cloud
Starting point is 00:00:46 and last week in AWS in that. You don't really know why you do it. But then at the end of it, you feel like you should have done it and you appreciate it. This episode is sponsored in part by my day job, Duck Bill. Do you have a horrifying AWS bill? That can mean a lot of things, predicting what it's going to be, determining what it should be, negotiating your next long-term
Starting point is 00:01:09 contract with AWS, or just figuring out why it increasingly resembles a phone number, but nobody seems to quite know why that is. To learn more, visit duckbillhq.com. Remember, you can't duck the duck bill bill, which my CEO reliably informs me is absolutely not our slogan. Thank you. I still remember a few of the things you. said along the way. For example, you confess that you thought that a lot of my jokes on Twitter were AI-driven until you started talking to me and realized, no, I talk exactly like my Twitter account does. Yeah, pretty much sarcasm remains my first language spoken at home. Yeah, we wound up doing, what was it, 10 miles a day on average for six days at the end of which everything
Starting point is 00:01:53 hurt. It was fun. Carrying obsessively about how much everything we carried weighed was certainly a new experience. And then talking to folks who were doing three, four times our mileage, good for them, good for them. I choose happiness and life. Yeah, I don't think managing ounces that you carry in your pack is a new experience for you at all because that's exactly what you do, tuning AWS bills. I feel like managing weight is like you load everything and you need and then halfway through the trip, you start throwing in off the trail, not that we would ever litter. and you end up at the end of it with what you exactly needed.
Starting point is 00:02:33 Oh, it's great too. My back is super strong from carrying AWS marketing all these years. So I don't know why I was so focused on being ultra lightweight. You had a knack, though, for finding very expensive things along the trail. Just people had littered,
Starting point is 00:02:46 and you picked them up at one for to be a good citizen. Like, you found a headlamp that was not inexpensive and also a pound and a half just because someone hated themselves. I can see why they wanted to leave that behind. Please don't litter in the woods.
Starting point is 00:02:59 But, yeah, you had a knack for being very lucky around things like that. Yeah, it's true. It's true. And for the listeners, for the podcast, day one, I was convinced that Corey had memorized all of his jokes and AI generated content. Day two, I started to suspect it might actually be coming out of him naturally. Day three through six, I realized this is actually just how Corey talks. And, in fact, AI is the one catching up. and he probably doesn't generate any of his content.
Starting point is 00:03:30 Not really. Sometimes I'll workshop titles and that's that's about enough. Honestly, in this case, I don't even need to. The title writes itself, Appalachian Cloud Trail. It's beautiful. A wonderful podcast title. I mean, the best part of it was we didn't have to think about clouds for the entire trip. The internet can't find us there. And you learn a lot about yourself when you're huddled at a tent in a rainstorm or in your case,
Starting point is 00:03:55 trying fruitlessly to convince me to switch to a half. all the time or when my sleeping pad exploded under me at 10.30 at night. Then as of lying there in the cold, dark rocks, I get to hear the American singing retrievers a hundred yards away. The only difference between them and coyotes is my unwillingness to admit when I've made a terrible mistake. And it was just a series of fun things that at the time didn't seem like fun, but now I'm nostalgic about them because anything's better than dealing with AWS bills. I don't disagree. And what really the highlight of the trip, and you probably should take a step back and we should we should walk them through it in chronological order but uh was being
Starting point is 00:04:32 saved by shepherd our uh our shuttle driver yeah why don't you begin that story because otherwise it looks like i'm dunking on you and it's more fun if you dunk on yourself please take it away we'll get to the cloud stuff don't worry but i want this is the story worth telling it's it's phenomenal it's it's a three part and so i'll try to go through it rapidly and you uh jumped in when i miss the detail but we started off parking my vehicle at the bottom and then we had a shuttle driver driver's top and his name was gone bottom top there was a lot of uphill in a lot of our hiking it was there we read what something like a mile in elevation oh it was it was something crazy I don't even remember it was it was it was significant and every day was hard it we might have had one flat day
Starting point is 00:05:17 out of six like it was pretty wild or one flat night when my sleeping pad exploded but we'll get that please continue definitely had a flat night so uh couple days in, I don't know, night two, I can't remember which exact night. I slept great. I heard nothing other than the coyotes, which apparently are not coyotes. But apparently in the middle of the night, Corey's sleeping pad blew up. And early in the night, too, it was like 11 p.m. or something like that. Eight inch blowout. We're not talking a slow leak that you get to find. We're talking all the air left in about a second and a half. In fact, the next morning, you asked if I had gone to sleep with ice skates on. That's right.
Starting point is 00:05:55 It was pretty wild. It was an unrepairable thing. So, of course, Corey, he had a great night. He managed to turn his backpack into a sleeping bag. I left a review on the Durstyn subreddit. It was the Kwaka 55. And I said, this is a solid 8 out of 10 as a backpack. Two out of 10 as a sleeping pad.
Starting point is 00:06:15 That's exactly right. So we were trying to figure out the next day how to not have Corey sleeping on the ground. And I was like, you know what? I should text Tom or shuttle driver because. And, oh, I want to be clear, before this point, you were extraordinarily kind of like, well, you should have woken me up. Like, why? Like, like, my, one of our, like, we both have young kids. Part of the reason we want to go get
Starting point is 00:06:32 lost in the woods. It's like, like, like, they're like standing by the side of your hammock. My blanket fell on the floor. I had a bad dream. I want some water. Like, no, there's nothing you can do. I'm just going to, like, I'm already not getting a great night's sleep. Why did I make it both of us? Because I'm not a jerk. That's true. He did let me sleep. But, um, that's accurate. So the next day, me with my full nights rest in a beautiful haymach, by the way, which you should switch to you sometime for you. We woke up and, well, he, Corey didn't wake up. He was already awake. We texted Tom and basically we're like, hey, could you shuttle us into this town nearby, Hot Springs? And we can buy a sleeping pad. Because we're another two nights out and I wasn't going to do that to my back. Right. No,
Starting point is 00:07:15 that's not, that's not how nobody wants to do that for multiple nights in a row. And Tom is like, hey, why don't I just run you one out? And so next thing you know, we hiked to a road, and our shuttle driver runs us a sleeping pad out that Corey uses the rest of the trip. It was phenomenal. So those are the first two parts of the story that involved Tom. I liked it because it wasn't as lightweight, but it also didn't explode. We'll get there. There's an advantage to a sleeping pad that functions. And then after we rolled in the hot springs, slightly before we rolled in the hot springs, I realized that I had left my phone at the campsite where we had just walked out of. And there was no way that I was going to hike, you know, multiple miles back to go retrieve it.
Starting point is 00:08:02 I started doing the math in my head of like, how much do phones cost these days? So we ended up at a hospital. It's the via I on. That doesn't sound like a bad thing to litter in the woods. And maybe we're uplifting the technological development of the local squirrel population. Why not? That's right. That's right.
Starting point is 00:08:18 No, I was leaving a relic to be discovered by future. generations up there. But anyway, so we mentioned it to Tom, who was at the hostel, he happened to be one of the people running the hostel, and his trail name is Shepard, which we quickly discovered why it was Shepard. And he's like, you know what, let me just trail run back. I can just trail on back and get your phone for you. And next thing you know, this guy, who's an absolute legend, trail runs back to where we camped at, retrieves my phone and runs back before the sun even us. It was wild. It's super easy to picture this guy because picture a guy that can just run up the side of a mountain like that to find a phone with weak-ass directions for where this is, find it and come
Starting point is 00:09:03 back. That's exactly what he looks like. But it was great. So how will I know what's your campsite? Well, there's a tree that looks like it's about to give out at any moment and obliterate the fire pit, which, okay, gray, he knows nothing about us, whether we know what we're talking about in the woods. And he's walking around the area. And then he sees that giant tree about to obliterate a fire. Fiber, like, yep, this is definitely the site. Holy crap. Yeah, like we started setting up. You saw it from the other side and you said, yeah, well, let's move about 20 feet that
Starting point is 00:09:30 way. Choose life. Fine. And he just did this. He pulled a rabbit out of a hat for us. It was the weirdest thing. And you could, like, for us, it's his legendary story. And for him, it was any given Tuesday.
Starting point is 00:09:42 Yeah. I think, honestly, I love hiking. I've been section hiking the Appalachian Trail before he joined me on this current leg. and I think what I love most about it is the community. Like, just people are willing to just jump in and help. If you need something out on the trail, you can't just go to a city, you know, to a town casually and grab it. People, you know, I, for example, there was a guy passing us whose water filter gave out. And so Courtney and I both have water filters.
Starting point is 00:10:09 His is better than mine. I won't admit that. Don't tell Corey that. It'll get to his head. And so it's like, you know what, I can use that water filter. I'll give you my water filter. And it's not just me. like the whole trail has that communal feeling to it, which makes the whole experience even
Starting point is 00:10:23 that much better. We ran to an old guy with a dog who was three miles from water. We gave in the last of ours because we're young enough that we could still deal with getting there, getting a little dehydrated, and then just drinking three gallons of it. We want, it's the thing you do. Like, who needs it more? Exactly, exactly. I have a similar feeling doing cloud economy work in that there's so few people in this space,
Starting point is 00:10:47 especially ones that have heavy duty engineering backgrounds. And it's a similar community where every day should jumps in, like there's a lot of sharing and knowledge, there's a lot of opinions. There's a lot of esoteric information that's out there. And every time I come across somebody in my space that works in this field, they just freely give that knowledge. It's one of the things I enjoy about working in the space.
Starting point is 00:11:07 Yeah. One thing that I found, the reason I like doing stuff like this, it's been a month since we got back. And I'm, it's weird how quickly you get accustomed to it. But I'm also making. get a point to notice it now. I don't know if you're listening to this. You're just hearing my voice, my dulcet tones in a mic. But if you're watching this on the YouTube's, this is my first recording from our new Duck Bill office in downtown. And I'm sitting in a comfy chair at a desk with a
Starting point is 00:11:32 sparkle water in my hand. And everything is comfortable. Everything is clean. I don't have to dig a hole if I want to poop. I mean, I can, but that's generally frowned upon in the office environment. And yeah, and I don't have to care how much something weighs. It's if I'm going to, am I going to take this home tonight or not? Well, weight isn't really a factor on that. So it's just, it's a different way of living on some point. At some point, it makes me more thankful for what you've got. Like I still wander the grocery store in awe compared to how much of a pain
Starting point is 00:12:06 of the neck it is to feed yourself in the woods. Yeah, no, I think that's a great, like perspective is everything. And it's so easy to go through life fast. I think the best way I can describe coming off the trail, and it being a month in, this is probably, this feeling's already passed for me. But the first probably solid week when I come back from a long hike, everything feels fast. Meetings feel fast. Conversations feel fast. Cars feel fast.
Starting point is 00:12:32 You know, even schedule, like, even my calendar, everything just feels fast. When you, when you're out on the trail, everything's serial. You have, like, one task to do. There's no multitasking. And everything happens in order. order and your brain can follow it in order. And it's a real reminder that like cognitive overload in your day to day, it's a real thing. Like simplifying what you have in front of you and doing it serially really is an effective strategy. Yeah. What's also wild is I then got to spend a whole
Starting point is 00:13:03 bunch of time harassing you after the fact with references to the callback jokes we had on it. Like we decided that everything was heavy. Why don't we just build gear full of helium so that it actually has a negative weight. And that means you can take a lot more with it. And like three days later, I'm at home and I sit bolt up right and text you. It's, I figured it out. We're going to sell this and we're going to brand it. And here's how most sleeping bags are filled with down. These are the only ones that are filled with up. And you did the courtesy of not responding to that. And that was great. And thanks to the magic of AI image editing, I wound up finding, like, I did some rain gear that looked like Rudolph. And I called it Rudolph the Red Nose Rainier. And that was awesome.
Starting point is 00:13:45 I went with a blister on my heel when I switched away from toe socks because it turns out I am lactose intolerant. That was one of my favorites. It was great. It was just great being able to throw the dumb jokes at someone who will absolutely smack them back. No, it was, we had a phenomenal time. It was, it really was. It's something I wish I could do more of. And not just me.
Starting point is 00:14:08 I wish everybody, I wish there was like a public holiday where everybody went and spent a week on the trail because I think we would all view, differently. And what do you do when you're trying to climb up a mountain that never seems to freaking end? Well, you come up with trail facts about Tom. For example, he doesn't use GPS positioning himself. The satellite just realigned to match his trail out of respect. He passes the white blaze. They repaint themselves. You know, usual stuff. One would expect Chuck Norris fact style. Only Tom is still alive because that man will never die. Well, and that's true. I refuse to believe those headlines. Another example of having time is we were headed up one particularly bad climb at the end of the day, so we were both exhausted. This was towards the end of the hike,
Starting point is 00:14:53 so we were even more exhausted. And I think I turned to Corey and it was like, I think it got about an hour of climbing, you should tell me a joke. And he proceeded to tell me the world's longest joke. And I think it is titled the world's longest joke. And I kid you not, it was at least, what, an hour in the telling Corey? Oh, yeah, it had to be. Yeah. And I didn't feel one single step at like time just basically teleported me to the top when he finally got to the punchline, which was very much worth it, by the way. Exactly. Maybe one of these days when I have nothing better to do.
Starting point is 00:15:23 I'll record an hour-long podcast of just that joke. I, you know, I'm here for it. 100%. I would be your audience member. I could hop on and laugh and interject appropriately. Yeah. It was an awful lot of fun, just doing something completely different. And then we went back to work and worried about cloudful.
Starting point is 00:15:42 finance and the rest. One of the reasons I love working with you is that you've been doing this almost as long as I have. You were the guy that went on stage a while back and talked about putting Kubernetes clusters in every Chick-fil-A restaurant. I don't know if that's the secret ingredient in their sandwiches or not, but good job. That's, everything tastes better with Kubernetes. Maybe it's like trail hunger where you're hungry, so everything tastes better. With Kubernetes, it's the suffering that infuses the sandwich that makes everyone just enjoy it more. Maybe that's the knack. That's right.
Starting point is 00:16:17 It's like black-grown diamonds versus real diamonds. It is the suffering that makes the diamond. Jesus. Is that a conflict diamond? Well, not until I fight you for it. Yeah, exactly. That's funny. No, so, yeah, it's, I've been doing this for a long time.
Starting point is 00:16:34 I think I really enjoy doing cloud economy work. I do consulting for Duck Bill. I also have a full-time job. I'm too. And one of the reasons I enjoy it so much is because in order to be really good at it, you have to have walked the gauntlet of racket and stacking as an engineer so you know on-prem pieces. You have to walk the gauntlet of operational work keeping systems alive because cost is only
Starting point is 00:17:03 one element of consideration for an organization. And if you bust through the door and you're like, hey, these are the, you know, you could save a ton of money. If you could just turn production off between the hours of 11 and 6 a.m., people are going to look at you like you're crazy. The United States Social Security Administration basically does, but that's neither here or there. It's some sort of mainframe backup job that means that they have actual, I'm not kidding, business hours for their website. I don't doubt that in the least. I mean, you know what? Maybe they're actually ahead of the curve.
Starting point is 00:17:33 Maybe everybody should take their example. Maybe it's not them needing to come. The computers are tired, Caleb. Let them sleep. Yeah, you know, everything comes back around, right? Maybe we should all regress to 1980s models. That's one of the things I've been helping people with recently is convincing them not to go back to on-prem. It's funny.
Starting point is 00:17:52 We've had a number of clients approach us to this. And I swear every one of them is going to think that we're calling them out. But no, no, this is a larger trend. And one of the fun things that you and I got to do because we're hiking through the woods with not much else to talk about at one point was try to out-traum. was try to out trauma each other with what's going to suck the most that people don't see coming about building out a data center. And we talk about things like racknots and bad batches of cables and Dell screwing things up as is their nature and Cisco doing the same as is their nature.
Starting point is 00:18:22 And then you have remote hands, which are great, and you have smart hands which has something really insulting about remote hands by a juxtaposition there. And all the trauma and the rest. and we move, like the speed of JSON APIs, which is when you have a guy named Jason who opens a Service Now ticket by a fax and eventually implements it. I'm not saying that that's on-prem is always a bad idea,
Starting point is 00:18:45 but the companies that have never done it before are opening up a bag of pain that the clouds have done such a good job abstracting away that people don't realize it's there or believe that it exists to the degree that we're talking about it. You and I collectively have damn near 30 years of experience working in day, data centers and the like. The fact that we do not work there anymore is not because we are stupid.
Starting point is 00:19:08 This episode is sponsored by my own company, Duck Bill. Having trouble with your AWS bill, perhaps it's time to renegotiate a contract with them. Maybe you're just wondering how to predict what's going on in the wide world of AWS. Well, that's where Duck Bill comes in to help. Remember, you can't duck the Duck Bill bill bill, which I am reliably informed by my business partner, is absolutely not our motto. To learn more, visit duckbillhq.com. Yeah, I mean, at the end of the day, like, at least your cloud bill edges up over time
Starting point is 00:19:45 and you get investigated and figure out why and then troubleshoot it and lower that bill. But with data centers, like, you can wake up tomorrow and, you know, it can be a catastrophic issue or outage and or a... Through no fault of your own. Well, cloud take outages too. Yeah, but the difference there is one, everyone is going to be down at the same time.
Starting point is 00:20:06 And two, if it's a foundational infrastructure issue, they have a team of the best people in the world that are working to fix that faster than you will. It's, there is no if-answer, but around that, it's a different universe. Yeah, yeah. My DR strategy, if U.S. East One goes down, is I'm going to go grab Corey and we're going to shelter it out in the Appalachian Mountains and it happens. And or depending on whether the world is still survivable at that point, we will also, make all the money consulting people with very expensive problems all of a sudden. Because people generally don't plant, well, like we've seen the regional outages in the Middle East. Yeah, geopolitics matter. If we see a U.S. East One regional outage, that says something
Starting point is 00:20:47 concerning from a physical infrastructure layer after it goes about a week, because this is about 100 square miles in northern Virginia we're talking about. Yeah, actually, you got a good point. If U.S. East One was down for more than a week, you would see. Pacific Coast Trail. Pacific Coast Trail. Yeah, we don't want to be on the Appalachian. that point. Yeah, exactly. Or and or once we come back from the trail, we'll retool duck bill to focus on migrating people out of the cloud onto on-prem. Ducks are migratory. I mean, you know, arguably, we could make that argument. More duck less bill now. More duck less bill. Amazing. So what was your, what was your favorite part of a hike in with
Starting point is 00:21:26 me, Corey? It's weird, because at the time, the thing you look forward to the most is we hit the halfway point and managed to eat like 5,000 calories in a meal. That was awesome. There was one place in Hot Springs. It was dialing in their pizza oven and it was, they were so good at it. There was a sign that said, best pizza in town. That was a fucking lie. I love the fact that both of us were starving and still wouldn't eat that piece of crap. And then we went to the actual pizzeria and it was freaking incredible. Everything else at that place was amazing. But it was a, but it was very much a curiously, the two of us got to meet up with Chris,
Starting point is 00:22:04 my podcast producer. So it was fun being able to sit down and talk a little bit of shop about this. We tried to convince him to use Kubernetes for something, mostly as a prank. And it was fun just being able to let go of a lot of the work stuff. It was also nice to not have to worry about the complex things and instead worry about the simple things that are a little bit more survival oriented.
Starting point is 00:22:29 Like, okay, I don't like to look at those clouds. We should probably find a decent camping spot so we're not surprised off the side of the mountain. Wow, we're getting a little low on water. Do we believe that that next water source ahead is going to be sufficient or should we use this crappy one while we've got it? These are decisions that are great. Frankly, you also sleep like the dead after your second night out there
Starting point is 00:22:50 just because you're bone tired. Yeah, even if it is on a backpack. You know, we take what we can get. I did send, to their credit, it was a Nemo Tenser Elite. This thing's about a $250 sleeping pad that weighs about as much as a postage stamp. And it's a great pad, but it's delicate. And to their credit, they replaced the entire thing with a position of just write the RMA number on it in permanent ink.
Starting point is 00:23:15 Great, we're mailing you a new one. Sorry for the trouble, which was super appreciated. But also now the problem I've got is one of these already failed me. would I trust it as my only sleeping option going forward? I don't know. There's a question of faith. And this, of course, is similar to cloud providers. When you know how a cloud provider fails
Starting point is 00:23:38 and how they handle that failure from a business perspective, when you're dealing with a new provider, well, you don't know how that's going to break. And furthermore, there's a terrific opportunity for them to either gain or lose you as a customer forever. I'm saying nice things about Nemo. Their tensor prod line is pretty great. it's delicate, but it worked super well
Starting point is 00:23:56 if they'd been a jerk to me, oh, that would be the headline instead and I would have been tearing into them, but they did the right thing. There's a people, I think cloud providers forget the human component. Like, Google can't wait to not talk to humans as fast as humanly possible. Do I get, do we get a split the product placement fee for Nemo there? You know, I miss the days when I used to sponsor this with something other than Duck Bill itself, just because it's great. Hey, why not?
Starting point is 00:24:20 At some point, you start to wonder, like, whether the audience, is no longer aligned. Well, you went from advertising enterprise SaaS and observability products to camping gear. Bad business or do you know something? We don't. You never know. But it's nice not to have to be cloud Billy Mays
Starting point is 00:24:38 pitching for something different every week. Yeah, no, it was a great time. And there's so many parallels between, you know, camping and doing cloud economists work from, you know, planning and managing the load you're going to carry to, you know, going through AWS bills and trying to understand services and how that impacts organizations and the communal aspect of the trail versus the communal aspect of helping these organizations, you know, everybody piling and helping some people being
Starting point is 00:25:05 resistant to, you know, some people being like, you're here to just to make me look bad. It's like, no, no, I'm not here to make you look bad. I really am here to try to help the organization. Well, you have to be careful how far you stretch that analogy to the breaking point. Otherwise, it starts to sound like a really bad, long-form LinkedIn post. What being brutally surprised by a raccoon taught me about B2B sales. Like, I mean, you know, I can see that. I can see being brutally surprised by a raccoon helping me with B2B sales.
Starting point is 00:25:34 I think I would hate both things just as equally much. Yeah, raccoons are great except by surprise. That's right. Oh, man. But, yeah, it was, it was a good time. So I guess my question now is, since we're back and ostensibly having to think about work again, What do you see that's exciting coming down the pike right now as far as the wide world of cloud economics? I have a list of my own, but this is not one of those areas as right or wrong.
Starting point is 00:26:01 Yeah, it's a great question. One of the things that I've seen emerging that I have mixed feelings about, I used to have stronger feelings about it, but now I think there's arguments that can be made for both sides is I'm starting to see serverless within the AWS ecosystem. And when I say it with serverless, I'm referring to Lambda. I'm referring to MongoDB and other serverless functions that don't require constant lows being run, even like Aurora serverless to a certain extent. I'm starting to see these functions start to become more part of the main production systems that I find. Yeah, this is going to sound weird.
Starting point is 00:26:41 I want to flag this out because companies have been using these things for 10 years at this point. we are not late to the adoption curve, but what we're seeing is this being a significant part of the core production application line, a significant driver of spend. Yes, it was replacing batch jobs a long time ago. Now it's in the critical path in a way that it wasn't previously in large enterprises.
Starting point is 00:27:06 Yeah, small companies have been doing this forever, but that's the beautiful part of being a small company. You can put a funny animal on your website, oops, we're down, and there's the raccoon looking embarrassed. and great, it comes back up an hour later. Things are generally okay. Enterprises don't have that luxury. Which I'm a big fan of, like, the small projects that I built for myself, for example,
Starting point is 00:27:27 I build them all serverless because I don't know what kind of load I'm going to get. I want to be able to scale up if whatever I'm working on goes bananas, and I want to be able to not pay a bunch of money if it doesn't. So it's kind of an ideal solution for a lot of things I build. I think for the first time about a month ago, I came across a customer who probably 90% of their bill and there were a fairly significant spend per year, annual spend per year, but probably 80 to 90 percent of their services were all serverless technologies. I think ECS is probably the closest thing that I would say was constant.
Starting point is 00:27:59 They had like a minimum number of ECS jobs that they had kind of tuned their application down to for some things. It was interesting because evaluating that customer, there were areas that I could give advice on, you know, how they were doing CloudTrail and CloudWatch and some CSP pieces, etc. but I couldn't really tune down their CPU and memory loads because they were so heavy on serverless, they were already using pretty much exactly what they needed to. Now, the negative, the downside of that is because people aren't historically been running most of their loads in these particular technologies, the one-to-one ratio of serverless is more
Starting point is 00:28:40 expensive. Like if you took the per call cost for, you know, Lambda or the per call cost for DynamoDB and you compared it to running your own large cluster doing that thing, it's always going to be a two, three, sometimes four to one ratio as far as dollars to dollars. Where that's a little deceptive is that people don't run perfectly efficient solutions when they manage. If you have perfect four knowledge and can run it exactly 100% with no overhead and your people's time to manage that cluster is free, yeah. Sure, maybe, but we're talking math, not reality. Yeah, so what was really interesting was like where I started to see it break down a little bit is the tooling isn't there yet to really manage serverless technology at scale.
Starting point is 00:29:24 So if you've got thousands and thousands of lambdas, tuning that, managing it and controlling it, you kind of destroy the 80-20 rule, right? So the nice thing about traditional ways of solving cloud problems is the top 20% of your services are the heavy hitters. they're expensive. If you tune those top 20%, you're going to get 80% of your bill impacted. That's sort of the 80-20 rule. Don't optimize the tiny thing that gets fired once a month. Maybe optimize the thing that gets fired 18 times per user request.
Starting point is 00:29:54 And also question your architecture if that's what it's doing. Also true. But the serverless approach tends to kind of peanut butter your code, repose, services, etc. and stretch it out. And so you know long you really have that 80-20 rule. If you want to go tune something, you really have to go and revisit a thousand land this or a thousand different services to go, you know, change how you're doing a particular thing. So I think there's a sprawl problem that a lot of people experienced when containers first at the market or finally figuring out
Starting point is 00:30:29 like that sprawl and making things too easy and too easy to manage will translate into making more services, which creates sprawl. So I think my opinion with this trend, I really like it. As a cloud economist, I love the idea that people are choosing a technology that is efficient and cost effective. But I also think as a technologist that's run these systems before, I think you're going to see people start hitting a very similar sprawl problem that they hit with all this has happened before. And it doesn't, the rise of vibe coding is going to make this worse instead of better. Like, oh, you have some bad code? We have a solution to that. It's called a lot more code. It builds on top of everything. It doesn't get additive. It doesn't
Starting point is 00:31:10 refactor on its own. Let's continue to build, build, build, build, the foundation could be rotten. Yeah, observability in those sorts of environments becomes a lot harder too. I mean, one of the upsides of a monolith is that, you know, you instrument it to write to logs in an opinionated way, and it doesn't matter what somebody does in the future, for the most part. You're going to be able to go get that information, that observability, and go drill down into it. But when you've got 2,000 landos that were written over a period of five years by, you know, 15 different people. Coverage is uneven. It's weird. Some of the best environments you see are the most, some of the best environments you see architecturally are some of the newest, just because people
Starting point is 00:31:49 have gotten into it, not having to learn the early mistakes the rest of us did. Yeah, again, I strongly agree with that. But I think there's every new trend, every new technology, there is some downside that we just don't see yet. I haven't seen. I haven't seen. in the last 10 years in this field, I haven't seen any trend emerge that doesn't have some sort of payoff later that you just don't know yet. So me personally, if I was at the helm of a very large enterprise organization, I would be fine being made out of late adopter, but I wouldn't want to be the tip of the spear on a lot of these new technologies. No, spend your innovation tokens where it matters on the thing your company actually does. Caleb, I want to thank you for taking the time
Starting point is 00:32:31 to speak with me. If people want to learn more, where's the best? place for them to find you. Yeah, you can hit me up on LinkedIn. I think it's, let me actually look up my LinkedIn. I should have this memorized here. It's Caleb R-H-H-H-U-R-D. But yeah, they can hit me up on LinkedIn. Awesome. Or find you on the trail. Thank you for putting up with me as long as you have and not murdering me out in the middle of the woods. It's an absolute pleasure. Yeah, you too, Corey. It's always fun. Caleb Hurd, Cloud Economist here at Duck Bill. I am Corey Quinn. Chief Cloud here at Duck Bill, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice,
Starting point is 00:33:10 whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry, insulting comment that then will be gathered, printed out, and hand-delivered to us on the trail by Tom.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.