Screaming in the Cloud - The Future of Serverless with Allen Helton

Episode Date: September 15, 2022

About AllenAllen is a cloud architect at Tyler Technologies. He helps modernize government software by creating secure, highly scalable, and fault-tolerant serverless applications.Allen publishes content regularly about serverless concepts and design on his blog - Ready, Set Cloud!Links Referenced:Ready, Set, Cloud blog: https://readysetcloud.ioTyler Technologies: https://www.tylertech.com/Twitter: https://twitter.com/allenheltondevLinked: https://www.linkedin.com/in/allenheltondev/

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. This episode is sponsored in part by our friends at AWS AppConfig. Engineers love to solve and occasionally create problems,

but not when it's an on-call fire drill at four in the morning. Software problems should drive innovation and collaboration, not stress and sleeplessness and threats of violence. That's why so many developers are realizing the value of AWS AppConfig feature flags. Feature flags let developers push code to production, but hide that feature from customers so that the developers can release their feature when it's ready. This practice allows for safe, fast, and convenient software development. You can seamlessly incorporate AppConfig feature flags into your AWS or cloud

environment and ship your features with excitement, not trepidation and fear. To get started, go to snark.cloud slash appconfig. That's snark.cloud slash appconfig. I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on, because serverless means it's still somebody's problem. And a big part of that responsibility is

app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are, finding and fixing vulnerabilities right from the CLI, IDs, repos, and pipelines. Snyk integrates seamlessly with AWS offerings like CodePipeline, EKS, ECR, and more, as well as things you're likely to actually be using. Deploy on AWS, secure with Snyk, learn more at snyk.co.scream. That's s-n-y-k.co. slash scream. Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while, I wind up stumbling into corners of the internet that I previously

had not traveled. Somewhat recently, I wound up having that delightful experience again by discovering ReadySetCloud.io, which has a whole series of, I guess, some people might call it thought leadership. I'm going to call it instead how I view it, which is just amazing opinion pieces on the context of serverless mixed with APIs, mixed with some prognostications about the future. Alan Helton, by day, is a cloud architect at Tyler Technologies, but that's not how I encountered you. First off, Alan, thank you for joining me.

Thank you, Corey. Happy to be here. I was originally pointed toward your work by folks in the AWS Community Builder program, of which we both participate from time to time. And it's one of those, oh, wow, this is amazing. I really wish I'd discovered some of this sooner. And every time I look through your back catalog and I click on a new post, I see things that are either, I really agree with this, or I can't stand this opinion. I want to fight about it. But more often than not, it's one of those recurring moments that I love. Damn, I wish I had written something like this. So first, you're absolutely killing it on the content front.

Thank you, Corey. I appreciate that. The content that I make is really about the stuff that I'm doing at work. It's stuff that I'm passionate about. It's stuff that I spend a decent amount of time on. And really the most important thing about it for me is it's stuff that I'm learning and forming opinions on and wants to share with others. I have to say, when I saw that you were, oh, you work at Tyler Technologies, which sounds for all the world like, oh, it's a relatively small consultancy run by some guy, presumably named Tyler. And, you know, it's a boutique team of maybe 20, 30 people on the outside. Yeah. Then I realized, wait a minute, that's not entirely true. For example, for starters, you're publicly traded and okay, that does change things a little bit.

First off, who are you people? Secondly, what do you do? And third, why have I never heard of you folks until now? Tyler is the largest company that focuses completely on the public sector. We have divisions and products for pretty much everything that you can imagine that's in the public sector. We have software for schools, software for tax and appraisal. We have software for police officers, for courts, everything you can think of that runs the government can and a lot of times is run on Tyler software. We've been around for decades building our expertise in the domain. And the reason you probably haven't heard about us is because you might not have ever been in trouble with the law before. If you if you have been. No, no. I, I learned very early on in the course of my life, which is kind of a surprise to absolutely no one who spent more than 30 seconds with me

that I have remarkably little filter. And if 10 kids were the ones doing something wrong, I'm the one that gets caught. So I spent a lot of time in the principal's office. So just taught me to keep my nose clean. I'm, I those squeaky clean types just because I was always terrified of getting punished because I knew I would get caught. I'm not saying this is the right way to go through life necessarily, but it did have the side benefit of, no, I don't really engage with law enforcement in the going about the course of my life. That's good. That's good. But one exposure that a lot of people get to Tyler is if you look at the bottom of your next traffic ticket it'll probably say tyler technologies on the bottom

there oh so you're really popular in certain circles i'd imagine super popular yes yes and of course you get all the the benefits of writing that code that says if defendant equals alan helton then return i like that you get to have the exception cases built in that no one's ever going to wind up looking into. That's right. Yes. The idea of what you're doing makes an awful lot of sense. There's a tremendous need for a wide variety of technical assistance in the public sector. What surprises me, although I guess it probably shouldn't, is how much of your content is aimed at serverless technologies and API design, which, to my way of thinking, isn't really something that public sector has done a lot with.

Clearly, I'm wrong. Historically, you're not wrong. There's an old saying that government tends to run about 10 years behind on technology. Not even just technology, but all over the board, it run about 10 years behind on technology, not even just technology, but all over the board, it runs about 10 years behind. And until recently, that's really been true. There was a case last year, a situation last year, where one of the state governments, don't remember which one it was, but they were having a crisis because they couldn't find any COBOL developers to come in and maintain their software that runs the state. And it's COBOL. You're not going to find a whole lot of people that have that skill.

A lot of those people are retiring out. And what's happening is that we're getting new people sitting in positions of power and government that want innovation. They know about the cloud and they want to be able to integrate with systems quickly and easily have little to no onboarding time. You know, there are people in power that have grown up with technology and understand that, well, with everything else I can be up and running in five or 10 minutes. Why can't I do this with the software I'm consuming now?

My opinion on it is admittedly conflicted because on the one hand, yeah, I don't think that governments should be running on COBOL software that runs on mainframes that haven't been supported in 25 years. Conversely, I also don't necessarily want them being run like a seed series startup where, well, I wrote this code last night and it's awesome. So often I go to production with it

because I can decide not to do business anymore with Twitter for pets. And I could go on to something else like Petflix or whatever it is I choose to use. I can't easily opt out of my government. The decisions that they make stick, and that is going to have a meaningful impact on my life and everyone else's life who is subject to their jurisdiction. So I guess I don't really know where I believe the proper, I guess, pace of technological adoption should be for governments. Curious to get your thoughts on this. Well, you certainly don't want anything that's bleeding edge. That's one of the things that we kind of draw fine lines around because when we're dealing with government software, we're dealing with usually critically sensitive

information. It's not medical records, but it's your criminal record. And it's things like your social security number. It's things that you can't have leaking out under any circumstances. So the things that we're building on are things that have proven out to be secure and have best practices around security, uptime, reliability, and in a lot of cases as well in maintainability. You know, if there are issues, then let's try to get those turned around as quickly as we can, because we don't want to have any sort of downtime from the software side versus the software vendor side. I want to pivot a little bit to some of the content you've put out, because an awful lot of it seems to be, I think I'll call it variations on a theme. For example, let me just read some recent titles to illustrate my point. Going API first, your first 30 days.

Solutions architect tips. How to design applications for growth. Three things to know before building a multi-tenant serverless app. And the common thread that I see running through all of these things are, these are things that you tend to have extraordinarily strong and vocal opinions about only after dismissing all of them the first time and slapping something together and then sort of being forced to live with the consequences of the choices that you've made. In some cases, you didn't even realize you were making at the time. Are you one of those folks that has the

wisdom to see what's coming down the road? Or did you do what the rest of us do and basically learn all this stuff by getting it hilariously wrong and having to careen into rebound situations as a result? I love that question. I would like to say now I feel like I have the vision to see something like that coming. Historically, no, not at all. Let me talk a little bit about how I got to where I am, because that will shed a lot of context on that question. A few years ago, I was put into a position at Tyler that said, hey, go figure out this cloud thing. Let's figure out what we need to do to move into the cloud safely, securely, quickly, all that rigmarole. And so I did. I got to hand select a team of engineers,

people that I worked with at Tyler over the past few years. And we were basically given free reign to learn. We were an R&D team, 100% R&D for about a year's worth of time, where we were learning about cloud concepts in theory and building little proof of concepts. CI, CD, serverless, APIs, multi-tenancy, a whole bunch of different stuff. NoSQL was another one of the things that we had to learn. And after that year of R&D, we were told,

okay, now go do something with that. Go build this application. And we did, building on our theory, our cursory theory knowledge. And we get pretty close to go live. And then the business says, what do you do in this scenario? What do you do in that scenario?

What do you do here? I update my resume and go work somewhere else. Where's the hard part? It turns out that's not a convincing answer. Right. So we moved quickly. And then I wouldn't say we backpedaled, but we hardened for a long time prior to the go-life with the lessons that we've learned with the eyes of Tyler, the mature enterprise company saying, these are the things that you have to make sure that you take into

consideration in a actual production application. One of the things that I always push, I was a manager for a few years of all these cloud teams. I always push, do it, do it right, do it better. It's kind of like a crawl, walk, run. And if you follow my writing from the beginning, just looking at the titles and reading them kind of like what you were doing, Corey, you'll see that very much. You'll see how I talk about CICD. You'll see me how I talk about authorization.

See me how I talk about multi-tenancy. And I kind of go in waves where maybe a year passes and you see my content revisit some of the topics that I've done in the past. And they're like, no, don't do what I said before. That's not right. The problem when I'm writing all of these things that I've used, for example, my entire newsletter publication pipeline is built on a giant morass of Lambda functions and API gateways. It's microservices driven, kind of. And each microservice is built

almost always with a different framework. Lately, all the new stuff is CDK. I started off with the serverless framework. There are a few other things here and there. And it's like going architecting back in time as I have to make updates to these things from time to time. And the problem with having done all of it myself

is that I already know the answer to what fool designed this. It's, well, you're basically watching me learn what I was doing bit by bit. I'm starting to believe that the right answer on some level is to build an inherent shelf life into some of these things. Great. In five years, you're going to come back and re-architect it now that you know how this stuff actually works rather than patching together 15 blog posts by different authors, not all of whom are talking about the same thing, and hoping for the best.

Yep. That's one of the things that I really like about serverless. I view that as a giant pro of doing serverless, is that when we revisit with the lessons learned, we don't have to refactor everything at once. Like if it was just a big MVC controller out there in the sky, we can refactor one Lambda function at a time if now we're using a new version of the AWS SDK

or we've learned about a new best practice that needs to go in place. It's a, while you're in there, tidy it up, please, kind of deal. I know that the DynamoDB fanatics will absolutely murder me over this one. But one of the reasons that I have multiple Dynamo tables that contain effectively variations on the exact same data is because I want to have the dependency between the two different microservices be the API, not, oh, and under the hood, it's expecting this exact same data structure all the time. It just felt like that was the wrong direction to go in.

That is the justification I use for myself, why I run multiple DynamoDB tables that have the same content. Where do you fall on the idea of data store separation? I'm a big single table design person myself. I really like the idea of being able to store everything in the same table and being able to create queries that can return me multiple different types of entity with one lookup. Now, that being said, one of the issues that we ran into or one of the ambiguous areas when we were getting started with serverless was what does single table design mean when you're talking about microservices? We were wondering, does single table mean one DynamoDB table for an entire application that's composed of 15 microservices? Or is it one table per microservice? And that was ultimately what we ended up going with is a table per microservice. Even if multiple microservices are pushed into the same AWS account, we are still

building that logical construct of a microservice and one table that houses similar entities in the same domain. Something I wish that every service team at AWS would do as a part of their design is draw the architecture of an application that you're planning to build. Great. Now assume that every single resource on that architecture diagram lives in its own distinct AWS account. Because somewhere in some customer, there's going to be an account boundary at every interconnection point along the way. And so many services don't do that, where it's, oh, that thing and the other thing have to be in the same account.

So people have to write their own integration shims. And it makes doing the right thing of putting different services into distinct bounded AWS accounts for security or compliance reasons way harder than I feel like it needs to be. Totally agree with you on that one. That's one of the things that I feel like I'm still learning about is the account level isolation. I'm still kind of early on personally with my opinions and how we're structuring things right now, but I'm very much of a like opinion that deploying multiple things into the same account is going to make it too easy to do something that you shouldn't. And I just try not to inherently trust people in the sense that, oh, this is easy.

I'm just going to cross that boundary real quick. For me, it's also come down to security risk exposure. Like my last tweet in AWS.com Twitter shitposting thread client lives in a distinct AWS account that is separate from the AWS account that has all of our client billing data that lives within it. The idea being that if you find a way to compromise my public-facing Twitter client, great. The blast radius should be constrained to, yay, now you can, I don't know, spin up some cryptocurrency mining in my AWS account and I

get to look like a fool when I beg AWS forgiveness. But that should be the end of it. It shouldn't be a security incident, because I should not have the credit card numbers living right next to the funny internet web thing. That sort of flies in the face of the original guidance that AWS gave at launch, and right around 2008 era best practices were, oh, one customer, one AWS account. And then by 2012, they had changed their perspective. But once you've made a decision to build multiple services in a single account, unwinding and unpacking that becomes an incredibly burdensome thing. It's about the equivalent of doing a cloud migration in some ways.

We went through that. We started off building one application with the intent that it was going to be a siloed application, a one-off essentially. And about a year into it, it's one of those moments of, oh no, what we're building is not actually a one-off. It's a piece to a much larger puzzle. And we had a whole bunch of, unfortunately, tightly coupled things that were in there that we're assuming that resources were going to be in the same AWS account. So we ended up, I don't know, I think we took probably two months, which in the grand scheme of things isn't that long. and decoupling what was possible at the time into multiple AWS accounts, kind of segmented by domain, essentially.

But that's hard. AWS puts it, you know, it's those one-way door decisions. I think this one was a two-way door, but it locked and you could kind of jimmy the lock on the way back out. And you'd repose someone from the lobby to let you back in. Yeah, the biggest problem is not necessarily the one-way door decisions. It's the one-way door decisions that you don't realize you're passing through at the time that you do them.

Which, of course, brings us to a topic near and dear to your heart. And I only recently started to have opinions on this myself. And that is the proper design of APIs, which I'm sure will incense absolutely no one who's listening to this. Like my opinions on APIs start with, well, probably REST is the right answer in this day and age. And people are like, well, I don't know, GraphQL is pretty awesome. Oh, I'm thinking SOAP. And people look at me like I'm a monster from the Black Lagoon of centuries past, the XML land. So my particular brand of strangeness aside, what do you see that people are doing in the world of API design that is the, I guess,

most common or easy to make mistakes that you really wish they would stop doing? If I could boil it down to one word, fundamentalism. Let me unpack that for you. Oh, please. I absolutely want to get a definition on that one. I approach API design from a developer experience point of view. How easy is it for both internal and external integrators to consume and satisfy the business processes that they want to accomplish? And a lot of times, REST guidelines, you know, it's all about entity basis, you know, drill into the appropriate entities

and name your end points with nouns, not verbs. I'm actually very much onto that one, but something that you could easily do, let's say you have a business process that given a fundamentally correct RESTful API design, takes 10 API calls to satisfy. You could, in theory, boil that down to maybe three well-designed endpoints that aren't quote-unquote R restful that make that developer experience significantly

easier and if you were a fundamentalist that option is not even on the table but thinking about it pragmatically from a developer experience point of view that might be the better call so that's one of the things that I know feels like a hot take. Every time I say it, I get a little bit of flack for it, but don't be a fundamentalist when it comes to your API designs. Do something that makes it easier while staying in the guidelines to do what you want. For me, the problem that I've kept smacking into with API design, and honestly, let me be very clear on this, my first real exposure to API design rather than API consumer, which of course I complain about constantly, especially in the context of the AWS inconsistent APIs between services, was when I'm building something out and I'm reading the documentation for API gateway and, oh, this is how you wind up having this stage linked to this thing. And here's the endpoint. Okay, great. So I would just populate and build out a structure or a schema that has the positional parameters I want to use as variables in my function. And that's awesome. And then I realized, oh, I might want to call this a different way. Ah, crap. And sometimes it's easy

just to add a different endpoint. Other times I have to significantly rethink things and I can't shake the feeling that this is an entire discipline that exists that I just haven't had a whole lot of exposure to previously. Yeah, I believe that. One of the things that you could tie a metaphor to or for what I'm saying and kind of what you're saying is AWS SAM, the serverless application model. All it does is basically macros cloud formation resources.

It's just a transform from a template into cloud formation. CDK does the same thing. But what the developers of SAM have done is they've recognized these business processes that people do regularly, and they've made these incredibly easy ways to satisfy those business processes and tie them all together, right? If I want to have a Lambda function that is backed behind a endpoint, an API endpoint, I just have to add four or five

lines of YAML or JSON that says, this is the event trigger. Here's the route, here's the API. And then it goes and does four or five, six different things. There's some engineers that don't like that because sometimes that feels like magic. Sometimes a little bit magic is okay. This episode is sponsored in part by our friends at Sysdig. Sysdig secures your cloud from source to run. They believe, as do I, that DevOps and security are inextricably linked. If you want to learn more about how they view this, check out their blog.

It's definitely worth the read. To learn more about how they are absolutely getting it right from where I sit, visit sysdig.com and tell them that I sent you. That's S-Y-S-D-I-G.com. And my thanks to them for their continued support of this ridiculous nonsense. I feel like one of the benefits I've had with the vast majority of APIs that I've built is that because this is all relatively small scale stuff for what amounts to basically shit posting for the sake of entertainment, I'm really the only consumer of an awful lot of these things. So I get frustrated when I have to backtrack and make changes and teach other microservices that talk to this thing that that has now changed and it's frustrating but i have the capacity to do that it's just it's just work for a period of time i feel like that equation completely shifts when you have published this and it is now out in the world and it's not just users but in many cases paying customers where you can't really make those changes without significant notice. And every time you do,

you're creating work for those customers. So you have to be a lot more judicious about it. Oh yeah. There is a whole lot of governance and practice that goes into production level APIs that people integrate with. They say once you push something out the door into production, that you're going to support it forever. I don't disagree with that. That seems like something that a lot of people don't understand. And that's one of the reasons why I push API first development so hard and all the content that I write is because you need to be intentional about what you're letting out the door.

You need to go in and work not just with the developers, but your product people and your analysts to say, what does this absolutely need to do? And what does it need to do in the future? And you take those things and you work with analysts on specific, you work with the engineers to actually build it out. And you're very intentional about what goes out the door that first time because once it goes out with a mistake you're either going to version it immediately or

you're going to make some people very unhappy when you make a breaking change to something that they immediately started consuming it absolutely feels like that's one of those things that AWS gets astonishingly right. I mean, I had the privilege of interviewing at the time Jeff Barr and then Ariel Kelman, who was their head of marketing, to basically debunk a bunch of old myths. And one thing that they started talking about extensively was the idea that an API is fundamentally a promise to your customers. And when you make a promise, you'd better damn well intend on keeping it. It's why API deprecations from AWS are effectively unique whenever something happens. This is a singular moment in time when they turn off a service or degrade old functionality in favor of new. They can add to it. They can launch a V2 of something and then start to wean people off by calling the old

one classic or whatnot. But if I build something on AWS in 2008 and I wound up sleeping until today and go and try and do the exact same thing and deploy it now, it will almost certainly work exactly as it did back then. Sure, reliability is going to be a lot better and there's a crap ton of features and whatnot that I'm not taking advantage of, but that fundamental ability to do that is awesome. Conversely, it feels like Google Cloud likes to change around a lot of their API stories almost constantly, and it's unplanned work that frustrates the heck out of me when I'm trying to build something stable and lasting on top of it.

I think it goes to show the maturity of these companies as API companies versus just vendors. It's one of the things that I think AWS does. You see the similar dichotomy with Microsoft and Apple. Microsoft's new versions of Windows generally still have functionality within them to support stuff that was written in the 90s for a few use cases.

Whereas Apple's like, oh, your computer's more than 18 months old. Have you tried throwing it away and buying a new one? And oh, it's a new version of Mac OS. So yeah, maybe the last one will get security updates for a year and then get with the times. And I can't shake the feeling that the correct answer is in some way both of those, depending upon who your customer is and what it is you're trying to achieve. If Microsoft adopted the Apple approach, their customers would mutiny,

and rightfully so. The expectation has been set for decades that that isn't what happens. Conversely, if Apple decided, now we're going to support this version of macOS in perpetuity, I don't think a lot of their application developers would quite know what to make of that yeah I think it also comes from a standpoint of you better make it worth their while if you're going to take move their cheese I'm not a Mac user myself but from what I hear for Mac users and this could be rose-colored glasses but is that their stuff works phenomenally well you know when when a new thing comes out it doesn't, absolutely. Whenever I say things like that on this show, I get letters.

Oh yeah, really they'll come up with something that is a colossal pain in the ass on Mac. Like, yeah, try building a system-wide mute key. Yeah, that's just a hotkey away on Windows and here in Mac land. But it makes such beautiful sounds. Why would you want them to be quiet? And it becomes

this back and forth dichotomy there. And you can even explain it to iPhones as well and the Android ecosystem where it's, oh, you're going to support the last couple of versions of iOS. Well, as a developer, I don't want to do that. And Apple's position is, okay, great. Almost half of the mobile users on the planet will be upgrading because they're in the ecosystem. Do you want to still be able to sell things to those people or not? And they're at a point of scale where they get to dictate those terms. On some level, there are benefits to it. On others, it is intensely frustrating.

I don't know what the right answer is on the level of permanence on that level of platform. I only have slightly better ideas around the position of APIs. I will say that when AWS deprecates something, they reach out individually to affected customers on some level. And invariably when they say this is going to be deprecated as of August 31st

or whenever it is, yeah, it is going to slip at least twice in almost every case just because they're not going to turn off a service that is revenue bearing or critical load bearing for customers without massive amounts of notice and outreach. And in some cases, according to rumor,

having engineers reach out to help restructure things so it's not as big of a burden on customers. That's a level of customer focus that I don't think most other companies are capable of matching. I think that comes with the size and the history of Amazon. And one of the things that they're doing right now,

we've used Amazon Cloud Cams for years in my house. We use them as baby monitors. And they- Yeah, I saw this. I did something very similar with Nest. They didn't have the Cloud Cam at the right time that I was looking at it. And they just announced

that they're going to be deprecating. They're throwing them for sale. They're not going to support them anymore, which, ooh, at Amazon, we're not offering this anymore. But you tell the story. What are they offering existing customers? Yeah, so I'm slightly upset about it because I like my cloud cams

and I don't want to have to take them off the wall or wherever they are to replace them with something else. But what they're doing is they gave me, or they gave all the customers about eight months headstart. I think they're going to be taking them offline around Thanksgiving this year, just mid-November. And what they said is, as compensation for you, we're going to send you a Blink Cam, a Blink Mini, for every cloud cam that you have in use. And then we are going to gift you a year subscription to the Pro for Blink.

That's very reasonable for things that were bought years ago. Meanwhile, I feel like, not to be unkind or uncharitable here, but I use Nest Cams. And that's a Google product. I half expect that if they ever get deprecated, I'll find out because Google just turns it off in the middle of the night. And I wake up and I have to read a blog post somewhere that they put an update on Nest Cams. The same way they killed Google Reader once upon a time. That's slightly unfair. But the fact that that joke even lands does say a lot about Google's reputation in this space.

For sure. One last topic I want to talk with you about before we call it a show is that as the time of this recording, you recently had a blog post titled, What Does the Future Hold for Serverless? Summarize that for me. Where do you see this serverless movement, if you'll forgive the term, going? So I'm going to start at the end. I'm going to work back a little bit on what needs to happen for us to get there. I have a feeling that in the future, and I'm going to be vague about how far in the future this is, is that we'll finally have a satisfied promise of all you're going to write in the future is business logic. What does that mean? the right companies, the right feedback at the right time, is we can write code as developers and have that get pushed up into the cloud in a phrase that I know Jeremy Daly likes to say,

infrastructure from code, where it provisions resources in the cloud for you based on your use case. So I've developed an application and it gets pushed up in the cloud at the time of deploying it, optimized resource allocation. Over time, what will happen

with my future vision is when you get production traffic going through, maybe it's spiky, maybe it's consistently at a scale that outperforms the resources that it originally provisioned. We can have monitoring tools that analyze that and pick that out, find the anomalies, find the standard patterns, and adjust that infrastructure that it deployed for you automatically, where it's based on your production traffic for what it created, optimizes it for you, which is something that you can't do on an initial deployment right now. You can put what looks best

on paper, but once you actually get traffic through your application, you realize that, you know, what was on paper might not be correct. You ever notice that whiteboard diagrams never show the reality and they're always aspirational and they miss certain parts? And I used to think that this was the symptom I had from working at small scrappy companies because, you know, at those big tech companies, everything they build is amazing and awesome. I know it because I've seen their conference talks, but I've been a consultant long enough now and for a number of those companies to realize that, nope, everyone's infrastructure is basically a trash fire at any given point in time. And it works

almost in spite of itself rather than because of it. There is no golden path where everything is shiny, new, and beautiful. And that, honestly, I got to say, it was really depressing when I first discovered, like, oh God, even these really smart people who are, honestly, I got to say, was really depressing when I first discovered, like, oh, God, even these really smart people who are so intelligent, they have to have extra brain packs bolted to their chests, don't have the magic answer to all of this. The rest of us are just screwed then. But we find ways to make it work. Yep. There's a quote. I wish I remembered who said it, but it was a military quote where no battle plan survives impact with the enemy,

first contact with the enemy. It's kind of that way with infrastructure diagrams. We can draw it out however we want, and then you turn it on in production. It's like, oh, no, that's not right. I want to mix the metaphors there and say, yeah, no architecture survives your first fight with a customer. Like, right. I don't think that's quite what they're trying to say. It's like, what? You don't attack your customers? What's your customer service line look like? Yeah. It's, I think you're onto something. I think that inherently everything beyond the V1 design of almost anything is an emergent property where this is what we learned about it by running it and putting traffic through it and finding these problems.

And here's how it wound up evolving to account for that. I agree. I don't have anything to add on that. Fair enough. I really want to thank you for taking so much time out of your day to talk about how you view these things. If people want to learn more, where is the best place to find you? Twitter is probably the best place to find me,

alanheltondev. I have that username on all the major social platforms. So if you want to find me on LinkedIn, same thing, alanheltondev. My blog is always open as well. If you have any feedback you'd like to give there, readysetcloud.io. And we will, of course, put links to that in the show notes. Thanks again for spending so much time talking to me. I really appreciate it. Yeah, this was fun. This was a lot of fun. I love talking shop. It shows that it's nice to talk about things I don't spend enough time thinking about. Alan Helton, cloud architect at Tyler Technologies. I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud. If you've

enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that I will reject because it was not written in valid XML. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

This has been a HumblePod production. Stay humble.