Screaming in the Cloud - HEY, We’re Building Better Email with Blake Stoddard

Episode Date: August 20, 2020

About Blake StoddardBlake is Senior System Administrator on Basecamp’s Operations team who spends most of his time working with Kubernetes, and AWS, in some capacity. When he’s not deep i...n YAML, he’s out mountain biking.Links Referenced: Basecamp: https://basecamp.com/Twitter: https://twitter.com/t3rabytesSignal v. Noise blog (author page): https://m.signalvnoise.com/author/blake/Signal v. Noise blog (main site): signalvnoise.com

Transcript
Discussion (0)
Starting point is 00:00:00 Hello, and welcome to Screaming in the Cloud, with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the tools. It'll move your business forward fast. Okay, let's talk about what this really is. It's Visual Basic for Interfaces. Say I needed a tool to, I don't know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas, buttons, checkboxes, tables, etc. Then I can
Starting point is 00:01:03 wire all of those things up to queries with all kinds of different parameters. Post, get, put, delete, etc. It all connects to virtually every database natively, or you can do what I did and build a whole crap ton of Lambda functions, shove them behind some API's gateway, and use that instead. It speaks MySQL, Postgres, Dynamo, not Route 53 in a notable oversight, but nothing's perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces, and I don't know front-end. That's the most important part here. Retool is transformational for those of us who aren't front-end types.
Starting point is 00:01:44 It unlocks a capability I didn't have until I found this product. I honestly haven't been this enthusiastic about a tool for a long time. Sure, they're sponsoring this, but I'm also a customer and a super happy one at that. Learn more and try it for free at retool.com slash lastweekinaws. That's retool.com slash lastweekinaws. That's retool.com slash lastweekinaws and tell them Corey sent you because they are about to be hearing way more from me. Normally, I like to snark about the various sponsors
Starting point is 00:02:16 that sponsor these episodes, but I'm faced with a bit of a challenge because this episode is sponsored in part by A Cloud Guru. They're the company that's sort of famous for teaching the world to cloud, and it's very, very hard to come up with anything meaningfully insulting about them. So I'm not really going to try. They've recently improved their platform significantly, and it brings both the benefits of A Cloud Guru that we all know and love, as well as the recently acquired Linux Academy together. That means that there's now an effective,
Starting point is 00:02:47 hands-on and comprehensive skills development platform for AWS, Azure, Google Cloud, and beyond. Yes, and beyond is doing a lot of heavy lifting right there in that sentence. They have a bunch of new courses and labs that are available. For my purposes, they have a terrific learn-by-doing experience that you absolutely want to take a look at. And they also have business offerings as well under ACG for Business. Check them out. Visit acloudguru.com to learn more. Tell them Corey sent you and wait for them to instinctively flinch. That's acloudguru.com. Welcome to Screaming in the
Starting point is 00:03:26 Cloud. I'm Corey Quinn. I'm joined this week by Blake Stoddard, Senior Site Reliability Engineer at a company that has been in the news a fair bit lately, Basecamp. Blake, welcome to the show. Thanks for having me, Corey. So Basecamp was always this sort of aberration in tech circles. You rather famously did not take VC funding. You were originally called, I believe it was 37 Signals. Campfire was a early Slack-alike, only less awful in some ways than Slack has become. And recently you launched a second product, for lack of a better term, Hey, a controversial email client.
Starting point is 00:04:08 Controversial because it doesn't let people track everyone who uses it to read email while they're sleeping. Totally. That is exactly how it went. So it's been an interesting ride. Most notably where people would have heard about this is in many cases in small publications like the Wall Street Journal and the New more or less went nuclear against Apple, which given that my entire world professionally revolves around kicking a trillion and a half dollar company right in the shins, it really resonates with me. But for my case, I do it out of a labor of love, not out of a higher principle necessarily or a business perspective. Again, in fact, many of my business advisors urge me constantly to stop making as much fun of Amazon as I do. So then I make fun of them until they
Starting point is 00:05:09 leave me alone. What's it been like? What has the experience been from someone who has been more or less starts off with the, yeah, I'm a senior SRE. My entire job here is to keep the servers going and oh, there's our company, the New York Times. It's got to be a trippy experience. It's been fun. Yeah. I feel like Hay, the infrastructure behind Hay has been under my wing for a long time, for probably like nine months to a year. I was really the sole infrastructure engineer working on Hay. So I've really seen it from the beginning. And as we got closer and closer to launch, we did all this load testing. We predicted like, okay, in the first couple months, we want to see, I don't know, X hundred thousand customers.
Starting point is 00:05:46 And we knew the resource amounts needed to meet that. And we expected to get there in, I don't know, maybe six months. That sounded great. And then leading up to the launch and the day of the launch, everything just goes viral and it's in the news here and there.
Starting point is 00:05:58 And then all of a sudden we're looking at traffic graphs and we're seeing like the numbers that we expected to see six months into the launch within like two weeks of the launch. It's been a wild ride to scale this app up and to be able to do it with very little customer-facing effect. Everything has gone to plan, which is interesting to say from an ops perspective. It's all worked great. We're proud of how it is. To be clear, in the interest of full disclosure, I am a Hay customer. I am a huge fan of the idea of what Hay is built around. Specifically, the idea of it calls out and shames tracking pixels, various sketchy things that have,
Starting point is 00:06:34 I'm just going to say it, have infested email for a very long time. And it is terrific seeing a company take a stand against this. Tracking pixels are something that we hold near and dear to heart in a way that we want to extinguish them from the earth. And hey, it helps us do that. Oh, yeah. To be very transparent, the last week in AWS newsletter does have a tracking pixel at the end that tracks opens in aggregate. There's also a custom link fuzzer that I have built on top of this, which tells me in the aggregate, cool, I wound up sending 28 links last week to, I don't know, what is it now? 21,000 people at time of this recording. And I want to know how many people clicked any given link and show me a list of what were the top performing links. What are the top five? I care in the aggregate that some number
Starting point is 00:07:23 of individuals have clicked a link, but I could not possibly care less about which individual person clicked a link. I want to know what's resonating with the audience versus what isn't, in other words. And frankly, no one ever hits reply, so there's no real other good way to get that data. Yeah, and I think last week at AWS does this perfectly, and it sets the bar between the ideal use case of no tracking pixels whatsoever. And the current use case of every email marketing company, including a tracking pixel that's linked back to the user.
Starting point is 00:07:54 What I'm hoping personally is that Hey takes off and launches a bit of a revolution, a bit of a revolution. That sounds like a weird, weird way of min maxing in the same phrase, but I'm hoping that it sparks something where it becomes acceptable to, so our media kit can say, we send this to X thousand people and that's all we've got. But we already have that level of lack of transparency around podcasts, where when we put this podcast out, we have no earthly idea who's going to listen. The single metric we get is, oh, this many downloads. And from there, it's all a great mystery. And that's kind of fun in some ways, because there's no good way to track people. And the only other way to do it is horrifying. For the first three or four months I was doing this podcast, I thought I'd forgotten
Starting point is 00:08:41 to turn the microphone on because I got no feedback. Then I went to a conference and more or less got swarmed by people. I love the podcast. You listen to that? And it became this really interesting journey of discovery for me. Turns out it's easier to hit reply to a newsletter, although almost no one does that either, than it is to, I'm going to pull over while I'm commuting, pull out my phone and yell at whoever is on this podcast right now.
Starting point is 00:09:06 Turns out you have to have a really, really bad take for that to be someone's response. Podcasts are interesting, too, because I feel like I'm not very old. I'm 23 at the time that we're recording this. And back when I was, I don't know, in middle school or high school, I read about podcasts and these cool things that anybody could do. And I really wanted to do one, but the audience just wasn't there. But now we've come around to podcasts are the new wave of things that influence the way people think. And it's been a wild thing to watch change. Back on the tracking pixel thing, I feel like we've started to make a dent in the worldview there.
Starting point is 00:09:39 We were talking to MailChimp recently. Now any new MailChimp campaign doesn't come with tracking enabled by default. Email marketers have started writing blog posts about like, what are we going to do now that Hay is blocking how we're able to get information? And then we've seen other blog posts from companies that say, okay, Hay is blocking our things now. Here's all the sleazy things we're going to do now. And then we get to go back and block all those to make their life even more fun. Oh, I will say, I sometimes look at the feed,
Starting point is 00:10:05 because again, it's when you build a platform, at some point you're sort of interested to see, oh, wow, who's signing up for this stuff? How many people are signing up when I do this this week, or how many sign up when I do Y? I have no idea where these people are coming from, but there's been a pretty steady organic growth of about 100 net new subscribers every week,
Starting point is 00:10:22 almost since launch. And I smile, I nod, And it's like when I'm in an airplane, I don't think too hard about the physics of how this works. Because if you question it, it stops working. That is how I believe these things work. And it's nutty, but I started seeing a bunch of people signing up from hey.com email addresses, which is awesome. I signed up myself from my hey account to see how it went. And there were remarkably few scary, shamey things around what I send, which is kind of awesome. And all in all, it's been a great experience. Now, of course, I have a laundry list of feature enhancements, things that annoy me
Starting point is 00:10:55 about Hay. I mean, it is a piece of software. And let's not kid ourselves, the purpose of software is solely to piss people off for not doing things as they would have them done. Yes, Hay is definitely one of those pieces of software where you have to follow the way that we envision the software being used or you're going to have a bad time. I will also say, since I had not been tracking the development super closely, it's Mbox. I am, as in Mike or Mancy, for those who have watched a certain show, Box, I look at this, and my immediate response when that pop-up showed up was, oh my god, they launched this thing after all this work, and they had an egregious typo at the top of the screen. And then it was, oh, it's not a typo. It's cutesy.
Starting point is 00:11:38 I hate it. Thanks. And then, okay, and now I've just become accustomed to it, mostly. Well, I think our corporate stance is that your inbox is for important things that's what email should be for and so that's the take we have is that it's a box for important mail it's actually gone as far as that we've had customers that have started creating chrome extensions that find every reference to the word inbox in the app and change it to inbox to oh i've got to get me one of those.
Starting point is 00:12:08 Yeah, I like it quite a bit. It feels like it needs a few more features before I start taking it incredibly seriously as a mail client, in that right now it only easily works with a hey.com address. And as trendy as it is these days, and as much as I appreciate the long-term perspective that Basecamp has brought to all of its stuff, it feels like it's only a half step removed from, oh, you can email me at flyingdingusataol.com, where it's, this is tied to an ISP or provider that very well may not be around for the long haul. I was checking the other day, my vanity domain for my personal stuff, sequestered.net, was registered back in 2001. And I have gone through so many life changes, iterative steps forward. At one point, the domain for that lived on burning a post fix in Iraq and downtown LA that I was down there at least once a month fixing things that I'd horribly broken
Starting point is 00:12:56 because it turns out that remote access out of band was not something I'd figured out in those days. All the way now to, it lives a Gmail. I'm not super thrilled with it, but it works well enough for the time being. It's been this iterative process through, but the addresses remain the same. Trusting that whatever happens in the future to the hey.com email domain, it makes me reluctant to give it out to various companies where I'm going to need to continue to have an ongoing relationship from that contact point in perpetuity. Sure. One of the policies that we have at Basecamp is that everything we create will live until the end of the internet. And we've exemplified that, actually. The very first product we released to Daulist, we still run it.
Starting point is 00:13:37 It's running on AWS. We moved it from on-prem, and it's shuffled around through all the different iterations of how we run our infrastructure, but it's still going today. And that's the plan for Hayes. We're going to run it forever because that's the policy. We'll run things until the end of the internet. That's a very AWS-like policy as opposed to GCP, where someone shakes their car keys just off screen and, oop, forget this. I'm going to go chase that fun noise with the shiny thing. What is it? SimpleDB that's been around for, I don't know, since the dinosaurs were here and AWS still runs it? Andy Jassy, CEO of AWS, was on record in an interview with the press as calling it a failed product, but you can still get it. People say, oh, but it's not in the console anymore. Spoiler,
Starting point is 00:14:16 it never was. And I checked the other day, the job posting is currently filled apparently, they still hire for the SimpleDB team, which feels on some level like, wow, I didn't realize you'd hire people directly into it. I assume someone was screwing up. You'd put them on a pip, and that was digital Siberia that you would ship them off to. Oh, that's a great take. That's got to be the saddest team there, just because, sure, everyone is going to insult your products on the internet, but when the CEO of your company does it in a press interview, that can't feel great. No, not at all. And I feel like, on the other hand, I'm weary of putting anything on a Google product because I don't know if six months from now it will still be available. And that brings us to what I really wanted to talk about by having you on the show.
Starting point is 00:14:55 You spoke with the A Cloud Guru folks somewhat recently, and you alluded to some things that I wanted to dive into a bit. Specifically, you've mentioned that you are historically an on-prem shop, but talked about the launch for the infrastructure of Hay running on top of AWS, which is kind of awesome, and using Kubernetes, which is the exact opposite of awesome. So tell me a little bit about what would take you from an on-prem environment where you are currently happily living with, by all accounts, no intent to leave, into launching something on top of a public cloud provider. What's your strategy around that? What's the story? So the current status of our infrastructure is that we are, I guess, technically a hybrid cloud company. We still have on-premise data centers. We have two of them.
Starting point is 00:15:38 We have several racks in those. And in fact, we run several of our large revenue generating apps there still. In fact, we've actually run applications with their front-end compute in a major cloud provider with their database still on-prem because its cost and performance were inhibited to do that in the cloud. So we had a mandate to explore the cloud as an option to see if we can run the same workloads that we run in our own data centers in a cloud providers environment where we can do the same thing at the same or cheaper price with access to additional managed services to allow us to do more with the
Starting point is 00:16:11 same size operations team so it acts as more or less as a capability story slash force multiplier in other words yeah and in fact since we've moved to the cloud we have only grown the team by two so while growing the number of applications that we run by one, so I guess that's not a great ratio. True. But at the same time, depending upon the actual percentages, well, okay, that's a little sketchy at scale, but with small numbers, and you look at the amount of time it takes to do these things versus what the alternative is, that's not bad at all. Not at all. And I think the typical like Silicon Valley-backed thing to do would be, oh, we're going to launch a new product. Let's hire 300 people just because we can.
Starting point is 00:16:49 And in Basecamp's case, that is totally not what happened at all. In what you might be forgiven for mistaking for a blast from the past, today I want to talk about New Relic. They seem to be a relatively legacy monitoring company, and I would have agreed with that assessment up until relatively recently. But they did something a little out there. They reworked everything. They went open source, they made it so you can monitor your whole stack in one place, and most notably, from my perspective, they simplified their pricing into something that is much more affordable for almost everyone. There's even a free tier with one user and 100 gigs per month, totally free. Check it out at newrelic.com.
Starting point is 00:17:31 One thing that I see periodically when you have an on-prem environment that then decides to expand into the cloud for some aspects of it is, how do I put this politely? I guess I don't. You wind up with a VMware model, the payday lender of technical debt, where you're going to just run a bunch of VMs,
Starting point is 00:17:44 but now also in the cloud. You're not really leveraging cloud in that story so much as you are making it look like a version of your data center. You are effectively worsening the cloud environment in order to slightly improve your capability data center side. Not that that's inherently a bad thing, but it's not what I would call cost effective either. No, not at all. That's one of the things that I think should be a prime tenant of any corporation that's looking to move to the cloud. It should not be seen as a way to out-sugate CapEx to OpEx. Because if you're going to the cloud, you should be doing it to gain initial value. In our case, we decided to, let's explore the cloud as an option. The mandate was explicitly, do not lift and shift.
Starting point is 00:18:26 In fact, we had a typo recently internally where we said lift and shift. And I think that's a pretty accurate representation of how some corporate cloud moves go. So when we started looking at the cloud, we knew that we wanted to use containers as a way to orchestrate the way that we run our apps. The things that we run on-premise, we do so on bare metal.
Starting point is 00:18:43 We deploy them with Campus Prano. We use Chef to manage the boxes. We don't use containers on-premise, we do so on bare metal. We deploy them with Campus Prano. We use Chef to manage the boxes. We don't use containers on-premise. But by going to the cloud and being able to use a managed container orchestration service, that gives us a great chance to look at containerizing our apps and running them in containers, not just because it's the cool thing to do, but also because we gain value in being able to bin pack them better and use compute more efficiently than we could if we were just running them on a fleet of T2 nano instances.
Starting point is 00:19:09 So tell me a little bit more about the Kubernetes decision. Is this something that predated your Hay build out? Is that something you've been dabbling with for a long time? I've got to say that Basecamp as a company has seemed relatively immune to hype-driven development in many respects. So seeing that you folks were on Kubernetes was a little bit surprising. Yes, it did predate Hey! Development. When we first moved to the cloud and decided that containers were the way forward for us, we started out using AWS's Elastic Container Service, or ECS, as they prefer to call it. ECS is fine. It works well enough. But our take was that the
Starting point is 00:19:47 service was not gaining features at a good enough pace, and we were running into problems with it being sometimes a very big black box where things would happen and you didn't know why, and your fix was to open a support ticket and hope they responded to you quickly and helpfully. And our experience with support is that neither of those two things happen. Right. You'd shame them publicly and loudly in increasingly public places. Yes. And sometimes it helps a lot. Yeah. Beyond that, when we decided that ECS wasn't the way forward, but we still wanted to use containers, Kubernetes was the thing to do here. And around the same time, we also started looking at leaving AWS in favor of Google Cloud, because if you want to use containers, Google's GKE product is the way to go. I mean, for a project that came out of Google, using their managed version of the product is the way to go.
Starting point is 00:20:34 And in fact, we did do that. Basecamp 2, we actually ran on GKE for several months with minimal issues until Google started having a few bad days a lot. At that point, we decided to move Basecamp 2 back on-prem, but we didn't want to leave Kubernetes as a whole. By this point, we'd already started work on Hay, and Hay was living on GKE 2. But when we moved Basecamp 2 away from GKE and moved it on-prem, we decided that we were going to not use Google at all. And so since we already had the infrastructure, we were able to make use of the flagship feature of Kubernetes and move it to another Kubernetes platform with very little additional infrastructure work.
Starting point is 00:21:10 And from there, we moved to EKS, and it's been living there fine since then. What made you decide to go EKS instead of rolling your own control plane? The price of a managed Kubernetes cluster is less than the price of the number of engineering hours it would take to run Kubernetes on bare metal or on EC2. No, very fair. To be blunt, I wish more people accepted that.
Starting point is 00:21:32 Something that also struck me as interesting about your exploration of this was the idea of using Kubernetes on top of Spot. That's something I've been advocating for from an economic perspective for a long time. But in practice, it feels like you talk about the cloud as, oh, it's elastic. You can scale up when the load increases and scale down when it doesn't need to, which makes everyone feel super good about not doing exactly that.
Starting point is 00:21:58 Everything winds up at the same baseline level of usage. And, oh, we'll get to it next sprint, as if suddenly you're going to stop making poor decisions right after this one. Using Spot requires a bunch of additional thought and processes around getting workloads onto Kubernetes. But once they're there, you are able to reap the benefits of not having committed compute capacity, just sitting around when you don't need it. You're able to use all the instances that AWS offers you all while saving money while doing it. Now, on the other hand, Spot looks great on paper. It looks great from a billing perspective, but I'd argue that Spot is one of the stickiest services that AWS offers
Starting point is 00:22:35 because you can't replicate that on-premise. Google's preemptible instance product only gives you a 30-second window versus Amazon's two-minute window. Amazon's implementation of Spot works really well for us because our workloads, we're able to know, well, we're able to have two different versions. We know that some workloads are okay being terminated with a two-minute warning, and then things that we know aren't okay being terminated with a two-minute warning, we run them on on-demand instances, and they're able to work well. So you said, or implied at least, that Spot is one of those things that is not able to be replicated on-prem.
Starting point is 00:23:06 You're fundamentally not going to get there in the same way. Tell me a little bit more about that. What do you think that the spot market does in terms of lock-in? Do you think that this is something that's going to drive people once they learn to adopt to something like the spot market that makes it in some ways harder to leave AWS than any pure technology buy? No, I don't think that it's a product that causes you to say like, oh, this is absolutely amazing. What will we do without it? I think that it makes you feel really nice inside when you see the cost savings. And when you see the ability that you have to pull from vast resource pools that you can't replicate on
Starting point is 00:23:45 premise. It's neat to start seeing capability stories like that around the things, to be very direct, that people are using in AWS. Because they get on stage, they talk a lot about their ridiculous nonsense. Oh, it's a machine learning musical keyboard. Or hey, it's this ridiculous thing that winds up leveraging 18 different implementations of blockchain. But if you look at what people are actually using slash spending the money on, it's EC2, it's data transfer, it's S3, it's database store, it's disk. It's the boring building blocks that no one wants to talk about in keynotes because it's not interesting or exciting anymore the way that it once was, but it's what the world runs on.
Starting point is 00:24:26 So things like Spot seem like they are aligned with that vision of the future. Totally. And using Spot isn't just a click a button and your workloads will be fine. It requires taking the time to look through them, seeing what can deal with being torn down with a very short amount of notice
Starting point is 00:24:39 and accepting that maybe your workloads aren't good for Spot. Maybe you don't want to use this. But for a lot of workloads, you totally can, especially for front-end web workloads like ours, we have no reason to not use Spot. One of the things that I find compelling is this, something like that does force you to refresh your instances,
Starting point is 00:24:57 or at least have a plan to refresh them at virtually any moment. Whereas in practice, we talk about in many environments, oh, we believe in cattle instead of pets. And then you look about in many environments, oh, we believe in cattle instead of pets. And then you look at their environment like, oh, great. So I can turn any one of these things off? Absolutely. Except for that one, that one, that one, that one, and oh, my God, that one. And it comes down to the what we say versus what we do story. Yeah, totally. And that's actually one of the things that Kubernetes helps us with a ton by being able to use spot instances that we can let things come and go into the auto-scaling groups and actually treat these instances like cattle.
Starting point is 00:25:30 Because Kubernetes can take care of scheduling things on other nodes when they become available. It can take care of what's running now, what needs to be running. We have controllers in the cluster that notice that, oh, we've lost an instance. We have pods that need to be scheduled, but we don't have that capacity. Add it to the cluster. Kubernetes helps us a ton there. So when you take a look across the landscape of what cloud providers offer, what you're able to achieve on-prem, do you think the future is pure public cloud for what the sort of things that you do?
Starting point is 00:25:57 Do you foresee there's going to be Basecamp data centers for the next century or something else entirely? I think hybrid cloud is going to continue to be the thing that we see more and more of. I think Google Cloud is pushing a ton with Anthos. We're doing it ourselves. We've done it before where we run front-end compute on Kubernetes and we keep databases on-prem and we connect to them over Direct Connects and GCP interconnects.
Starting point is 00:26:20 Some things are just not feasible to do in the cloud, whether that's from a cost standpoint, whether it's from a performance standpoint, whether it's from a performance standpoint, whether there's some service that you want to run that isn't offered in a managed form in AWS, or do you want to use it as a managed form? I think hybrid cloud is going to be the end goal. And I think we've even seen companies like Netflix that start out being all in on AWS decide,
Starting point is 00:26:38 oh, this is kind of expensive, and decide to run their own data centers with their own hardware sometimes. So hybrid cloud, I think, will end up being the end goal of what we do. But I think the implementation will be different for everybody. What do you think is currently
Starting point is 00:26:50 the most misunderstood thing about Kubernetes in the larger ecosystem? That you're not required to use it. I like that quite a bit. This is coming from Basecamp. Again, one of your founders was the creator of Ruby on Rails. There's a very anti-trendy JavaScript framework philosophy there that, frankly, I wish the rest of the industry shared.
Starting point is 00:27:36 It's easy to dismiss this as, oh, those are just a few countercultural folks who are trying to stir up trouble. I don't believe that there's a lot to be said for using technologies that have been shown to work, for focusing on parts of the story that are, I guess, more in line with what sensible people with a business interest are concerned about. And it feels like on some level, the number one project everyone's trying to solve for remains their own resume. Especially for a company the size of Basecamp, we won't do things unless we see value coming out of them. We didn't look at the cloud and decide to do it just because it was a cool thing to do. We didn't look at Kubernetes and decide to start using it just because it was a cool thing to do. We started using Kubernetes because it gave us a path to accomplish our end goal, which was making our compute more efficient, being able to run it in ways that we can't do on-prem, and being able to do more with the same number of operations staff members. Kubernetes and public cloud are great for those things. For a product like Hay, if we were running that on-premise, we would have massively over-provisioned
Starting point is 00:28:14 the hardware, spent hundreds of thousands of dollars on a product or on hardware for a product that we can't guarantee is going to perform the way that we think it will. We would have had to hire additional people to be in charge of racking and stacking that year. And that's just not something we have to worry about when we're running on a managed cloud product and when we're using something like Kubernetes. It seems increasingly like solving for business value rather than for hype is taking a renewed focus at a lot of companies. I suspect that the longer this pandemic drags on, the better enterprise tech is going to become just due to lack of executive exposure to ads in airports, which historically seems to have
Starting point is 00:28:52 been driving an awful lot of very strange decision-making. Now we're seeing, well, what is the business value for this type of conversations coming out? And I'm optimistic that this is going to usher in a new era of good decision making. But on balance, these are human beings we're talking about. And, well, we have some track record of showing how that is not true. My hope is that we start seeing executive sponsorship lean more into how does the operations team want to run things? How can they run things efficiently? Let the people who do the work make the informed decisions
Starting point is 00:29:26 about how they want to work, how they can work most efficiently, how they can meet the goals of the business, rather than just handing it down from the top with no discussion to the implementers. If people want to hear more about what you have to say about this and other topics,
Starting point is 00:29:40 where can they find you? I do a lot of ranting about Kubernetes and public cloud on Twitter at T3RABYTES, the word terabytes with an three instead of an E. And I've also been writing more on our company blog, Signal vs. Noise, which is signalvsnoise.com. And we will put links to those in the show notes, of course. Thank you so much for taking the time to speak with me today. I appreciate it. Absolutely. Thanks for having me, Corey. Blake Stoddard, Senior Site Reliability Engineer at Basecamp. I'm cloud economist Corey Quinn,
Starting point is 00:30:12 and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts. Whereas if you hated it, please leave a five-star review on Apple Podcasts and a tracking pixel in the comments. This has been this week's episode of Screaming in the Cloud. You can also find more Corey at screaminginthecloud.com or wherever Fine Snark is sold. This has been a humble pod production stay humble

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.