Screaming in the Cloud - The Appalachian Cloud Trail: Hiking, Cloud Economics, and Finding Perspective
Episode Date: June 11, 2026What happens when two cloud economists leave AWS behind and spend six days hiking 60 miles on the Appalachian Trail? Corey Quinn sits down with Caleb Hurd to share stories from the trail, i...ncluding exploding sleeping pads, heroic shuttle drivers, lost phones, and the unique community that makes long-distance hiking special. Along the way, they draw surprising parallels between backpacking and cloud economics, discussing everything from serverless architecture and cloud cost optimization to the hidden challenges of on-prem infrastructure. It's a conversation about technology, adventure, perspective, and why sometimes the best way to solve complex problems is to step away from them entirely.Show highlights:(00:00) Why Hiking Hooks You(00:15) Meet Caleb on the Trail(01:31) Trail Miles and Ultralight Parallels(05:24) The Sleeping Pad Blowout(07:46) Shepherd Saves the Day(09:43) Trail Community and Cloud Community(11:07) Post Trail Perspective and Inside Jokes(15:35) Back to Work On Prem vs Cloud Pain(25:47) Server-less Spend and Lambda Sprawl(32:29) Wrap Up Where to Find CalebAbout Caleb: Caleb Hurd is a Cloud Economist at Duckbill, where he helps enterprises make sense of their cloud spend. Before moving to the cost side of the house, Caleb spent years in the trenches building and operating large-scale cloud environments and leading the engineering teams behind them across companies ranging from healthcare tech to enterprise Saas. He also founded CostOps.cloud, an AWS cost consulting practice, and is a vocal advocate for engineering-led FinOps — arguing that the people closest to the architecture should be the ones driving cost strategy, not spreadsheet jockeys in finance. Caleb holds a degree from Georgia Tech and made an unconventional journey into tech from a background in carpentry, which may explain his preference for building things over just talking about them. He's based in Atlanta.Links:LinkedIn: https://www.linkedin.com/in/calebrhurd/Sponsored by: duckbillhq.com
Transcript
Discussion (0)
I think, honestly, I love hiking.
I've been section hiking the Appalachian Trail
before I join me on this current leg.
And I think what I love most about it is the community.
Welcome to Screaming in the Cloud.
I'm Corey Quinn, and today's guest has had just about enough of me.
Caleb Hurd is many things.
He's periodically a cloud economist here at Duckbill,
but he was also my hiking buddy for a 60-mile section hike of the Appalachian Trail.
Caleb, thank you.
for continuing to speak with me.
You're welcome, Corey.
You're welcome.
It's a pleasure to be here.
And I would say six days on the trail with you
is very much like screaming in the cloud
and last week in AWS in that.
You don't really know why you do it.
But then at the end of it,
you feel like you should have done it and you appreciate it.
This episode is sponsored in part by my day job, Duck Bill.
Do you have a horrifying AWS bill?
That can mean a lot of things,
predicting what it's going to be, determining what it should be, negotiating your next long-term
contract with AWS, or just figuring out why it increasingly resembles a phone number, but
nobody seems to quite know why that is. To learn more, visit duckbillhq.com. Remember, you can't duck
the duck bill bill, which my CEO reliably informs me is absolutely not our slogan.
Thank you. I still remember a few of the things you.
said along the way. For example, you confess that you thought that a lot of my jokes on Twitter
were AI-driven until you started talking to me and realized, no, I talk exactly like my Twitter
account does. Yeah, pretty much sarcasm remains my first language spoken at home. Yeah, we
wound up doing, what was it, 10 miles a day on average for six days at the end of which everything
hurt. It was fun. Carrying obsessively about how much everything we carried weighed was certainly
a new experience. And then talking to folks who were doing three, four times our mileage,
good for them, good for them. I choose happiness and life. Yeah, I don't think managing ounces
that you carry in your pack is a new experience for you at all because that's exactly what you
do, tuning AWS bills. I feel like managing weight is like you load everything and you need
and then halfway through the trip, you start throwing in off the trail, not that we would ever litter.
and you end up at the end of it
with what you exactly needed.
Oh, it's great too.
My back is super strong
from carrying AWS marketing all these years.
So I don't know why I was so focused
on being ultra lightweight.
You had a knack, though,
for finding very expensive things along the trail.
Just people had littered,
and you picked them up at one
for to be a good citizen.
Like, you found a headlamp
that was not inexpensive
and also a pound and a half
just because someone hated themselves.
I can see why they wanted to leave that behind.
Please don't litter in the woods.
But, yeah, you had a knack for being very lucky around things like that.
Yeah, it's true.
It's true.
And for the listeners, for the podcast, day one, I was convinced that Corey had memorized all of his jokes and AI generated content.
Day two, I started to suspect it might actually be coming out of him naturally.
Day three through six, I realized this is actually just how Corey talks.
And, in fact, AI is the one catching up.
and he probably doesn't generate any of his content.
Not really. Sometimes I'll workshop titles and that's that's about enough.
Honestly, in this case, I don't even need to.
The title writes itself, Appalachian Cloud Trail.
It's beautiful.
A wonderful podcast title.
I mean, the best part of it was we didn't have to think about clouds for the entire trip.
The internet can't find us there.
And you learn a lot about yourself when you're huddled at a tent in a rainstorm or in your case,
trying fruitlessly to convince me to switch to a half.
all the time or when my sleeping pad exploded under me at 10.30 at night. Then as of lying there
in the cold, dark rocks, I get to hear the American singing retrievers a hundred yards away.
The only difference between them and coyotes is my unwillingness to admit when I've made a
terrible mistake. And it was just a series of fun things that at the time didn't seem like
fun, but now I'm nostalgic about them because anything's better than dealing with AWS bills.
I don't disagree. And what really the highlight of the trip, and you probably should
take a step back and we should we should walk them through it in chronological order but uh was being
saved by shepherd our uh our shuttle driver yeah why don't you begin that story because otherwise it looks
like i'm dunking on you and it's more fun if you dunk on yourself please take it away we'll get to
the cloud stuff don't worry but i want this is the story worth telling it's it's phenomenal it's it's a
three part and so i'll try to go through it rapidly and you uh jumped in when i miss the detail but
we started off parking my vehicle at the bottom and then we had a shuttle driver driver's
top and his name was gone bottom top there was a lot of uphill in a lot of our hiking it was
there we read what something like a mile in elevation oh it was it was something crazy I don't even
remember it was it was it was significant and every day was hard it we might have had one flat day
out of six like it was pretty wild or one flat night when my sleeping pad exploded but we'll get
that please continue definitely had a flat night so uh
couple days in, I don't know, night two, I can't remember which exact night. I slept great. I
heard nothing other than the coyotes, which apparently are not coyotes. But apparently in the
middle of the night, Corey's sleeping pad blew up. And early in the night, too, it was like 11 p.m.
or something like that. Eight inch blowout. We're not talking a slow leak that you get to find.
We're talking all the air left in about a second and a half. In fact, the next morning,
you asked if I had gone to sleep with ice skates on. That's right.
It was pretty wild.
It was an unrepairable thing.
So, of course, Corey, he had a great night.
He managed to turn his backpack into a sleeping bag.
I left a review on the Durstyn subreddit.
It was the Kwaka 55.
And I said, this is a solid 8 out of 10 as a backpack.
Two out of 10 as a sleeping pad.
That's exactly right.
So we were trying to figure out the next day
how to not have Corey sleeping on the ground.
And I was like, you know what?
I should text Tom or shuttle driver because.
And, oh, I want to be clear,
before this point, you were extraordinarily kind of like, well, you should have woken me up.
Like, why? Like, like, my, one of our, like, we both have young kids. Part of the reason we want to go get
lost in the woods. It's like, like, like, they're like standing by the side of your hammock.
My blanket fell on the floor. I had a bad dream. I want some water. Like, no, there's nothing you can
do. I'm just going to, like, I'm already not getting a great night's sleep. Why did I make it both of us?
Because I'm not a jerk. That's true. He did let me sleep. But, um, that's accurate. So the next day, me with
my full nights rest in a beautiful haymach, by the way, which you should switch to you sometime for you.
We woke up and, well, he, Corey didn't wake up. He was already awake. We texted Tom and basically
we're like, hey, could you shuttle us into this town nearby, Hot Springs? And we can buy a sleeping pad.
Because we're another two nights out and I wasn't going to do that to my back. Right. No,
that's not, that's not how nobody wants to do that for multiple nights in a row. And Tom is like,
hey, why don't I just run you one out? And so next thing you know, we hiked to a road,
and our shuttle driver runs us a sleeping pad out that Corey uses the rest of the trip. It was phenomenal.
So those are the first two parts of the story that involved Tom. I liked it because it wasn't as
lightweight, but it also didn't explode. We'll get there. There's an advantage to a sleeping pad that
functions. And then after we rolled in the hot springs, slightly before we rolled in the hot springs,
I realized that I had left my phone at the campsite where we had just walked out of.
And there was no way that I was going to hike, you know, multiple miles back to go retrieve it.
I started doing the math in my head of like, how much do phones cost these days?
So we ended up at a hospital.
It's the via I on.
That doesn't sound like a bad thing to litter in the woods.
And maybe we're uplifting the technological development of the local squirrel population.
Why not?
That's right.
That's right.
No, I was leaving a relic to be discovered by future.
generations up there. But anyway, so we mentioned it to Tom, who was at the hostel, he happened to
be one of the people running the hostel, and his trail name is Shepard, which we quickly discovered
why it was Shepard. And he's like, you know what, let me just trail run back. I can just
trail on back and get your phone for you. And next thing you know, this guy, who's an absolute
legend, trail runs back to where we camped at, retrieves my phone and runs back before the sun even
us. It was wild. It's super easy to picture this guy because picture a guy that can just run up the side
of a mountain like that to find a phone with weak-ass directions for where this is, find it and come
back. That's exactly what he looks like. But it was great. So how will I know what's your campsite?
Well, there's a tree that looks like it's about to give out at any moment and obliterate the fire pit,
which, okay, gray, he knows nothing about us, whether we know what we're talking about in the woods.
And he's walking around the area. And then he sees that giant tree about to obliterate a fire.
Fiber, like, yep, this is definitely the site.
Holy crap.
Yeah, like we started setting up.
You saw it from the other side and you said, yeah, well, let's move about 20 feet that
way.
Choose life.
Fine.
And he just did this.
He pulled a rabbit out of a hat for us.
It was the weirdest thing.
And you could, like, for us, it's his legendary story.
And for him, it was any given Tuesday.
Yeah.
I think, honestly, I love hiking.
I've been section hiking the Appalachian Trail before he joined me on this current leg.
and I think what I love most about it is the community.
Like, just people are willing to just jump in and help.
If you need something out on the trail, you can't just go to a city, you know, to a town casually and grab it.
People, you know, I, for example, there was a guy passing us whose water filter gave out.
And so Courtney and I both have water filters.
His is better than mine.
I won't admit that.
Don't tell Corey that.
It'll get to his head.
And so it's like, you know what, I can use that water filter.
I'll give you my water filter.
And it's not just me.
like the whole trail has that communal feeling to it, which makes the whole experience even
that much better.
We ran to an old guy with a dog who was three miles from water.
We gave in the last of ours because we're young enough that we could still deal with
getting there, getting a little dehydrated, and then just drinking three gallons of it.
We want, it's the thing you do.
Like, who needs it more?
Exactly, exactly.
I have a similar feeling doing cloud economy work in that there's so few people in this space,
especially ones that have heavy duty engineering backgrounds.
And it's a similar community where every day should jumps in,
like there's a lot of sharing and knowledge,
there's a lot of opinions.
There's a lot of esoteric information that's out there.
And every time I come across somebody in my space that works in this field,
they just freely give that knowledge.
It's one of the things I enjoy about working in the space.
Yeah.
One thing that I found, the reason I like doing stuff like this,
it's been a month since we got back.
And I'm, it's weird how quickly you get accustomed to it.
But I'm also making.
get a point to notice it now. I don't know if you're listening to this. You're just hearing my
voice, my dulcet tones in a mic. But if you're watching this on the YouTube's, this is my first
recording from our new Duck Bill office in downtown. And I'm sitting in a comfy chair at a desk with a
sparkle water in my hand. And everything is comfortable. Everything is clean. I don't have to
dig a hole if I want to poop. I mean, I can, but that's generally frowned upon in the office environment.
And yeah, and I don't have to care how much something weighs.
It's if I'm going to, am I going to take this home tonight or not?
Well, weight isn't really a factor on that.
So it's just, it's a different way of living on some point.
At some point, it makes me more thankful for what you've got.
Like I still wander the grocery store in awe compared to how much of a pain
of the neck it is to feed yourself in the woods.
Yeah, no, I think that's a great, like perspective is everything.
And it's so easy to go through life fast.
I think the best way I can describe coming off the trail, and it being a month in, this is probably, this feeling's already passed for me.
But the first probably solid week when I come back from a long hike, everything feels fast.
Meetings feel fast.
Conversations feel fast.
Cars feel fast.
You know, even schedule, like, even my calendar, everything just feels fast.
When you, when you're out on the trail, everything's serial.
You have, like, one task to do.
There's no multitasking.
And everything happens in order.
order and your brain can follow it in order. And it's a real reminder that like cognitive overload
in your day to day, it's a real thing. Like simplifying what you have in front of you and doing it
serially really is an effective strategy. Yeah. What's also wild is I then got to spend a whole
bunch of time harassing you after the fact with references to the callback jokes we had on it. Like we
decided that everything was heavy. Why don't we just build gear full of helium so that it actually has a
negative weight. And that means you can take a lot more with it. And like three days later,
I'm at home and I sit bolt up right and text you. It's, I figured it out. We're going to sell this and
we're going to brand it. And here's how most sleeping bags are filled with down. These are the only ones
that are filled with up. And you did the courtesy of not responding to that. And that was great.
And thanks to the magic of AI image editing, I wound up finding, like, I did some rain gear that
looked like Rudolph. And I called it Rudolph the Red Nose Rainier. And that was awesome.
I went with a blister on my heel when I switched away from toe socks because it turns out I am lactose intolerant.
That was one of my favorites.
It was great.
It was just great being able to throw the dumb jokes at someone who will absolutely smack them back.
No, it was, we had a phenomenal time.
It was, it really was.
It's something I wish I could do more of.
And not just me.
I wish everybody, I wish there was like a public holiday where everybody went and spent a week on the trail because I think we would all view,
differently. And what do you do when you're trying to climb up a mountain that never seems to
freaking end? Well, you come up with trail facts about Tom. For example, he doesn't use GPS
positioning himself. The satellite just realigned to match his trail out of respect. He passes
the white blaze. They repaint themselves. You know, usual stuff. One would expect Chuck Norris fact
style. Only Tom is still alive because that man will never die. Well, and that's true. I refuse to
believe those headlines. Another example of having time is we were headed up one particularly bad
climb at the end of the day, so we were both exhausted. This was towards the end of the hike,
so we were even more exhausted. And I think I turned to Corey and it was like, I think it got about
an hour of climbing, you should tell me a joke. And he proceeded to tell me the world's longest joke.
And I think it is titled the world's longest joke. And I kid you not, it was at least,
what, an hour in the telling Corey? Oh, yeah, it had to be. Yeah. And I didn't feel
one single step at like time just basically teleported me to the top when he finally got
to the punchline, which was very much worth it, by the way.
Exactly.
Maybe one of these days when I have nothing better to do.
I'll record an hour-long podcast of just that joke.
I, you know, I'm here for it.
100%.
I would be your audience member.
I could hop on and laugh and interject appropriately.
Yeah.
It was an awful lot of fun, just doing something completely different.
And then we went back to work and worried about cloudful.
finance and the rest. One of the reasons I love working with you is that you've been doing this
almost as long as I have. You were the guy that went on stage a while back and talked about
putting Kubernetes clusters in every Chick-fil-A restaurant. I don't know if that's the secret
ingredient in their sandwiches or not, but good job. That's, everything tastes better with
Kubernetes. Maybe it's like trail hunger where you're hungry, so everything tastes better. With
Kubernetes, it's the suffering that infuses the sandwich that makes everyone just enjoy it more.
Maybe that's the knack.
That's right.
It's like black-grown diamonds versus real diamonds.
It is the suffering that makes the diamond.
Jesus.
Is that a conflict diamond?
Well, not until I fight you for it.
Yeah, exactly.
That's funny.
No, so, yeah, it's, I've been doing this for a long time.
I think I really enjoy doing cloud economy work.
I do consulting for Duck Bill.
I also have a full-time job.
I'm too.
And one of the reasons I enjoy it so much is because in order to be really good at it,
you have to have walked the gauntlet of racket and stacking as an engineer so you know
on-prem pieces.
You have to walk the gauntlet of operational work keeping systems alive because cost is only
one element of consideration for an organization.
And if you bust through the door and you're like, hey, these are the, you know, you could save a ton of money.
If you could just turn production off between the hours of 11 and 6 a.m., people are going to look at you like you're crazy.
The United States Social Security Administration basically does, but that's neither here or there.
It's some sort of mainframe backup job that means that they have actual, I'm not kidding, business hours for their website.
I don't doubt that in the least.
I mean, you know what?
Maybe they're actually ahead of the curve.
Maybe everybody should take their example.
Maybe it's not them needing to come.
The computers are tired, Caleb.
Let them sleep.
Yeah, you know, everything comes back around, right?
Maybe we should all regress to 1980s models.
That's one of the things I've been helping people with recently is convincing them not to go back to on-prem.
It's funny.
We've had a number of clients approach us to this.
And I swear every one of them is going to think that we're calling them out.
But no, no, this is a larger trend.
And one of the fun things that you and I got to do because we're hiking through the woods with not much else to talk about at one point was try to out-traum.
was try to out trauma each other with what's going to suck the most that people don't see coming
about building out a data center.
And we talk about things like racknots and bad batches of cables and Dell screwing things up
as is their nature and Cisco doing the same as is their nature.
And then you have remote hands, which are great, and you have smart hands which has something
really insulting about remote hands by a juxtaposition there.
And all the trauma and the rest.
and we move, like the speed of JSON APIs,
which is when you have a guy named Jason
who opens a Service Now ticket by a fax
and eventually implements it.
I'm not saying that that's on-prem is always a bad idea,
but the companies that have never done it before
are opening up a bag of pain
that the clouds have done such a good job abstracting away
that people don't realize it's there
or believe that it exists to the degree that we're talking about it.
You and I collectively have damn near 30 years
of experience working in day,
data centers and the like. The fact that we do not work there anymore is not because we are stupid.
This episode is sponsored by my own company, Duck Bill. Having trouble with your AWS bill,
perhaps it's time to renegotiate a contract with them. Maybe you're just wondering how to
predict what's going on in the wide world of AWS. Well, that's where Duck Bill comes in to help.
Remember, you can't duck the Duck Bill bill bill, which I am reliably informed by my business partner,
is absolutely not our motto.
To learn more, visit duckbillhq.com.
Yeah, I mean, at the end of the day, like,
at least your cloud bill edges up over time
and you get investigated and figure out why
and then troubleshoot it and lower that bill.
But with data centers, like, you can wake up tomorrow
and, you know, it can be a catastrophic issue or outage
and or a...
Through no fault of your own.
Well, cloud take outages too.
Yeah, but the difference there is one, everyone is going to be down at the same time.
And two, if it's a foundational infrastructure issue, they have a team of the best people in the world that are working to fix that faster than you will.
It's, there is no if-answer, but around that, it's a different universe.
Yeah, yeah.
My DR strategy, if U.S. East One goes down, is I'm going to go grab Corey and we're going to shelter it out in the Appalachian Mountains and it happens.
And or depending on whether the world is still survivable at that point, we will also,
make all the money consulting people with very expensive problems all of a sudden.
Because people generally don't plant, well, like we've seen the regional outages in the Middle
East. Yeah, geopolitics matter. If we see a U.S. East One regional outage, that says something
concerning from a physical infrastructure layer after it goes about a week, because this is
about 100 square miles in northern Virginia we're talking about. Yeah, actually, you got a good point.
If U.S. East One was down for more than a week, you would see. Pacific Coast Trail. Pacific Coast
Trail. Yeah, we don't want to be on the Appalachian.
that point. Yeah, exactly. Or and or once we come back from the trail, we'll retool
duck bill to focus on migrating people out of the cloud onto on-prem. Ducks are migratory.
I mean, you know, arguably, we could make that argument. More duck less bill now.
More duck less bill. Amazing. So what was your, what was your favorite part of a hike in with
me, Corey? It's weird, because at the time, the thing you look forward to the most is we hit the
halfway point and managed to eat like 5,000 calories in a meal. That was awesome. There was one
place in Hot Springs. It was dialing in their pizza oven and it was, they were so good at it.
There was a sign that said, best pizza in town. That was a fucking lie. I love the fact that
both of us were starving and still wouldn't eat that piece of crap. And then we went to the actual
pizzeria and it was freaking incredible. Everything else at that place was amazing. But it was a,
but it was very much a
curiously, the two of us got to meet up with Chris,
my podcast producer.
So it was fun being able to sit down and talk a little bit of shop about this.
We tried to convince him to use Kubernetes for something,
mostly as a prank.
And it was fun just being able to let go of a lot of the work stuff.
It was also nice to not have to worry about the complex things
and instead worry about the simple things
that are a little bit more survival oriented.
Like, okay, I don't like to look at those clouds.
We should probably find a decent camping spot
so we're not surprised off the side of the mountain.
Wow, we're getting a little low on water.
Do we believe that that next water source ahead is going to be sufficient
or should we use this crappy one while we've got it?
These are decisions that are great.
Frankly, you also sleep like the dead after your second night out there
just because you're bone tired.
Yeah, even if it is on a backpack.
You know, we take what we can get.
I did send, to their credit, it was a Nemo Tenser Elite.
This thing's about a $250 sleeping pad that weighs about as much as a postage stamp.
And it's a great pad, but it's delicate.
And to their credit, they replaced the entire thing with a position of just write the
RMA number on it in permanent ink.
Great, we're mailing you a new one.
Sorry for the trouble, which was super appreciated.
But also now the problem I've got is one of these already failed me.
would I trust it as my only sleeping option going forward?
I don't know.
There's a question of faith.
And this, of course, is similar to cloud providers.
When you know how a cloud provider fails
and how they handle that failure from a business perspective,
when you're dealing with a new provider,
well, you don't know how that's going to break.
And furthermore, there's a terrific opportunity
for them to either gain or lose you as a customer forever.
I'm saying nice things about Nemo.
Their tensor prod line is pretty great.
it's delicate, but it worked super well
if they'd been a jerk to me, oh, that would be the headline instead
and I would have been tearing into them, but they did the right thing.
There's a people, I think cloud providers forget the human component.
Like, Google can't wait to not talk to humans as fast as humanly possible.
Do I get, do we get a split the product placement fee for Nemo there?
You know, I miss the days when I used to sponsor this with something other than Duck Bill itself,
just because it's great.
Hey, why not?
At some point, you start to wonder, like, whether the audience,
is no longer aligned.
Well, you went from advertising enterprise SaaS
and observability products to camping gear.
Bad business or do you know something?
We don't.
You never know.
But it's nice not to have to be cloud Billy Mays
pitching for something different every week.
Yeah, no, it was a great time.
And there's so many parallels between, you know,
camping and doing cloud economists work
from, you know, planning and managing the load
you're going to carry to, you know, going through AWS bills and trying to understand services
and how that impacts organizations and the communal aspect of the trail versus the communal aspect
of helping these organizations, you know, everybody piling and helping some people being
resistant to, you know, some people being like, you're here to just to make me look bad.
It's like, no, no, I'm not here to make you look bad. I really am here to try to help the
organization.
Well, you have to be careful how far you stretch that analogy to the breaking point.
Otherwise, it starts to sound like a really bad, long-form LinkedIn post.
What being brutally surprised by a raccoon taught me about B2B sales.
Like, I mean, you know, I can see that.
I can see being brutally surprised by a raccoon helping me with B2B sales.
I think I would hate both things just as equally much.
Yeah, raccoons are great except by surprise.
That's right.
Oh, man.
But, yeah, it was, it was a good time.
So I guess my question now is, since we're back and ostensibly having to think about work again,
What do you see that's exciting coming down the pike right now as far as the wide world of cloud economics?
I have a list of my own, but this is not one of those areas as right or wrong.
Yeah, it's a great question.
One of the things that I've seen emerging that I have mixed feelings about, I used to have stronger feelings about it,
but now I think there's arguments that can be made for both sides is I'm starting to see serverless within the AWS ecosystem.
And when I say it with serverless, I'm referring to Lambda.
I'm referring to MongoDB and other serverless functions that don't require constant lows being run,
even like Aurora serverless to a certain extent.
I'm starting to see these functions start to become more part of the main production systems that I find.
Yeah, this is going to sound weird.
I want to flag this out because companies have been using these things for 10 years at this point.
we are not late to the adoption curve,
but what we're seeing is this being a significant part
of the core production application line,
a significant driver of spend.
Yes, it was replacing batch jobs a long time ago.
Now it's in the critical path
in a way that it wasn't previously in large enterprises.
Yeah, small companies have been doing this forever,
but that's the beautiful part of being a small company.
You can put a funny animal on your website,
oops, we're down, and there's the raccoon looking embarrassed.
and great, it comes back up an hour later.
Things are generally okay.
Enterprises don't have that luxury.
Which I'm a big fan of, like, the small projects that I built for myself, for example,
I build them all serverless because I don't know what kind of load I'm going to get.
I want to be able to scale up if whatever I'm working on goes bananas,
and I want to be able to not pay a bunch of money if it doesn't.
So it's kind of an ideal solution for a lot of things I build.
I think for the first time about a month ago, I came across a customer who probably 90% of their bill
and there were a fairly significant spend per year, annual spend per year,
but probably 80 to 90 percent of their services were all serverless technologies.
I think ECS is probably the closest thing that I would say was constant.
They had like a minimum number of ECS jobs that they had kind of tuned their application down to for some things.
It was interesting because evaluating that customer,
there were areas that I could give advice on, you know,
how they were doing CloudTrail and CloudWatch and some CSP pieces, etc.
but I couldn't really tune down their CPU and memory loads because they were so heavy on serverless,
they were already using pretty much exactly what they needed to.
Now, the negative, the downside of that is because people aren't historically been running
most of their loads in these particular technologies, the one-to-one ratio of serverless is more
expensive. Like if you took the per call cost for, you know, Lambda or the per call cost for
DynamoDB and you compared it to running your own large cluster doing that thing, it's always
going to be a two, three, sometimes four to one ratio as far as dollars to dollars.
Where that's a little deceptive is that people don't run perfectly efficient solutions when they
manage. If you have perfect four knowledge and can run it exactly 100% with no overhead and
your people's time to manage that cluster is free, yeah.
Sure, maybe, but we're talking math, not reality.
Yeah, so what was really interesting was like where I started to see it break down a little bit is the tooling isn't there yet to really manage serverless technology at scale.
So if you've got thousands and thousands of lambdas, tuning that, managing it and controlling it, you kind of destroy the 80-20 rule, right?
So the nice thing about traditional ways of solving cloud problems is the top 20% of your services are the heavy hitters.
they're expensive.
If you tune those top 20%,
you're going to get 80% of your bill impacted.
That's sort of the 80-20 rule.
Don't optimize the tiny thing that gets fired once a month.
Maybe optimize the thing that gets fired 18 times per user request.
And also question your architecture if that's what it's doing.
Also true.
But the serverless approach tends to kind of peanut butter your code, repose,
services, etc.
and stretch it out. And so you know long you really have that 80-20 rule. If you want to go tune something,
you really have to go and revisit a thousand land this or a thousand different services to go,
you know, change how you're doing a particular thing. So I think there's a sprawl problem
that a lot of people experienced when containers first at the market or finally figuring out
like that sprawl and making things too easy and too easy to manage will translate into making more
services, which creates sprawl. So I think my opinion with this trend, I really like it. As a cloud
economist, I love the idea that people are choosing a technology that is efficient and cost
effective. But I also think as a technologist that's run these systems before, I think you're
going to see people start hitting a very similar sprawl problem that they hit with all this has
happened before. And it doesn't, the rise of vibe coding is going to make this worse instead of
better. Like, oh, you have some bad code? We have a solution to that. It's
called a lot more code. It builds on top of everything. It doesn't get additive. It doesn't
refactor on its own. Let's continue to build, build, build, build, the foundation could be rotten.
Yeah, observability in those sorts of environments becomes a lot harder too. I mean, one of the
upsides of a monolith is that, you know, you instrument it to write to logs in an opinionated way,
and it doesn't matter what somebody does in the future, for the most part. You're going to be able to go
get that information, that observability, and go drill down into it. But when you've got 2,000
landos that were written over a period of five years by, you know, 15 different people.
Coverage is uneven. It's weird. Some of the best environments you see are the most,
some of the best environments you see architecturally are some of the newest, just because people
have gotten into it, not having to learn the early mistakes the rest of us did.
Yeah, again, I strongly agree with that. But I think there's every new trend, every new technology,
there is some downside that we just don't see yet. I haven't seen. I haven't seen.
in the last 10 years in this field, I haven't seen any trend emerge that doesn't have some
sort of payoff later that you just don't know yet. So me personally, if I was at the helm of a very
large enterprise organization, I would be fine being made out of late adopter, but I wouldn't
want to be the tip of the spear on a lot of these new technologies. No, spend your innovation tokens
where it matters on the thing your company actually does. Caleb, I want to thank you for taking the time
to speak with me. If people want to learn more, where's the best?
place for them to find you. Yeah, you can hit me up on LinkedIn. I think it's, let me actually
look up my LinkedIn. I should have this memorized here. It's Caleb R-H-H-H-U-R-D. But yeah, they can
hit me up on LinkedIn. Awesome. Or find you on the trail. Thank you for putting up with me as long as you
have and not murdering me out in the middle of the woods. It's an absolute pleasure. Yeah, you too,
Corey. It's always fun. Caleb Hurd, Cloud Economist here at Duck Bill. I am Corey Quinn. Chief Cloud
here at Duck Bill, and this is Screaming in the Cloud.
If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice,
whereas if you've hated this podcast, please leave a five-star review on your podcast platform
of choice, along with an angry, insulting comment that then will be gathered, printed out,
and hand-delivered to us on the trail by Tom.
