Screaming in the Cloud - Optimizing for Happiness with Alfonso Cabrera

Episode Date: December 3, 2020

About Alfonso CabreraAlfonso Cabrera is the Director of Platform Engineering at Red Ventures, where he helps manage and optimize the extensive AWS footprint. He also spent time at AWS as a So...lutions Architect and worked as a DevOps Engineer at a few startups. Alfonso enjoys fostering community and has organized DevOpsDays Charlotte for the past 5 years. Outside of work, he tries to stay in shape by playing sports of all kinds, and gets his adrenaline fix by riding motorcycles.Links ReferencedRed VenturesFollow Alfonso on TwitterConnect with Alfonso on LinkedIn

Transcript
Discussion (0)
Starting point is 00:00:00 Hello and welcome to Screaming in the Cloud with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. planet. You know, like VPNs were supposed to do before we all started working from home and the VPNs melted like glaciers. Teleport provides a unified access plane for developers and security professionals seeking to simplify secure access to servers, applications, and data across all of your environments without the bottleneck and management overhead of traditional VPNs. This feels to me like it's a lot like the early days of HashiCorp's Terraform.
Starting point is 00:01:05 My gut tells me this is the sort of thing that's going to transform how people access their cloud services and environments. To learn more, visit goteleport.com. This episode is sponsored by a personal favorite, Retool. Retool allows you to build fully functional tools for your business in hours, not days or weeks. No front-end frameworks to figure out or access controls to manage. Just ship the tools. It'll move your business forward fast. Okay, let's talk about what this really is. It's Visual Basic for Interfaces. Say I needed a tool to, I don't know, assemble a whole bunch of links into a weekly sarcastic newsletter that I send to everyone. I can drag various components onto a canvas. Buttons, checkboxes, tables, etc. Then I can
Starting point is 00:01:52 wire all of those things up to queries with all kinds of different parameters. Post, get, put, delete, etc. It all connects to virtually every database natively, or you can do what I did and build a whole crap ton of lambda functions, shove them behind some APIs gateway, and use that instead. It speaks MySQL, Postgres, Dynamo, not Route 53 in a notable oversight, but nothing's perfect. Any given component then lets me tell it which query to run when I invoke it. Then it lets me wire up all of those disparate APIs into sensible interfaces, and I don't know front-end. That's the most important part here. Retool is transformational for those of us who aren't front-end types.
Starting point is 00:02:33 It unlocks a capability I didn't have until I found this product. I honestly haven't been this enthusiastic about a tool for a long time. Sure, they're sponsoring this, but I'm also a customer and a super happy one at that. Learn more and try it for free at retool.com slash lastweekinaws. That's retool.com slash lastweekinaws and tell them Corey sent you because they are about to be hearing way more from me. Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Alfonso Cabrera, who is currently the Director of Platform Engineering at Red Ventures. Alfonso, welcome to the show. Thanks, Corey. I'm excited to be on. I've listened to the show for quite a
Starting point is 00:03:17 while now. I would suggest it might be time to find a show with, you know, intelligent folks, but then again, that's why I have conversations with other people. I'm the, I basically, we have this idea in storytelling where you have the hero's journey. I do something similar called the moron's journey and I'm the moron. So it works out super well. Every time I talk to someone on the show, I learn something. So you've run platform engineering at Red Ventures. Everyone has ventures. What is a Red Venture and how did it come to become color-coded? Great question. So Red Ventures is a portfolio of influential brands and digital platforms and some partnerships as well. But essentially, we connect millions of people through our audience with expert advice and information. We're in a lot of different
Starting point is 00:04:00 industries. So we have some businesses in home industry, in health, in travel, in finance. Some of the brands you may have heard of, things like Healthline or The Point Sky, if you travel a lot, but also things like Bankrate, MyMove, and creditcards.com. I get nostalgic and sad about that now, because again, I used to live on airline miles more or less. And now it turns out that no one travels anymore. And I guess I never thought I would actually miss traveling, but here we are. I know. I'm in the same boat. I miss it. I'm ready to get back out there and go travel again. So what makes this fun is that I met you a few years back when you were not a director at that point.
Starting point is 00:04:41 You were in a different role at Red Ventures. I forget offhand what that was. Yeah, I started pretty early on were in a different role at Red Ventures. I forget offhand what that was. Yeah, I started pretty early on the cloud ops engineering team at Red Ventures about four and a half years ago now. I was one of the first two folks that was helping the company move to the cloud. And then you quit.
Starting point is 00:04:57 Because, you know, the best thing to do is go work somewhere else, given the opportunity. At least that's always been my philosophy, which is why my resume is a patchwork of tattered, broken dreams. But you went to a small company called AWS and acted as a solutions architect for a while. And then you boomeranged back to Red Ventures. So what was that like? Yeah, for sure. It was a great opportunity to go to AWS and try out kind of a different career path, right? I view kind of solutions architecture there as kind as customer-facing, almost similar to a sales engineer type of role. And I had a lot of fun. I learned a ton of stuff
Starting point is 00:05:29 at AWS. Some of the things I really, really enjoyed and took away from there that probably will stay with me forever is just the writing culture is crazy to see there. I was really fond of that. I think writing brings a lot of clarity of thought instead of compared to a PowerPoint deck, right? For example, where you may have a few words in the slide, but generally you're kind of just talking through things. And when you write things down, whether it's a one-pager or a six-pager or the PR FAQs that they're well-known for,
Starting point is 00:05:56 it just brings a lot more clarity and gets everybody on the same page. Something that's definitely stuck with me and I've continued to do as a habit at Redventures. I love your framing of that. Usually when you ask people, oh, so at this job, what did you take away from it? The honest answer is generally a bunch of office supplies and that's about it. But every time you talk to someone who's left AWS, they talk about, on some level, the lessons
Starting point is 00:06:17 that they learned by working there. How much of, I guess, AWS culture do you find is applicable or portable to other companies? Yeah, actually, it's funny because one of the other things I really enjoyed there is kind of the principles-based decision-making they do. Obviously, they're very vocal about their leadership principles and reference them in day-to-day decisions or in meetings and in different projects they're working on. And that's actually something that I've also seen at Red Ventures. One of the things I love about the culture at Red Ventures is it's very similar. We have belief statements that we discuss often, right?
Starting point is 00:06:51 Like those things are brought up when we're making decisions or when we're starting new projects, we kind of frame them against our belief statements. And it makes a big difference, right? Because it kind of gives some consistency or table stakes to things like to decision making, to projects you work on and to discussions you have and to just the company culture overall. I think it makes a big difference. The challenge I always see is that you have these folks who go and look at, what did Amazon do that made them successful?
Starting point is 00:07:19 And their response is, oh, okay. So they read the leadership principles, pick their three favorites and then try and effectively slap them into their own company culture without anything approaching context. It's, okay, frugality, great. We're going to become super cheap. Two pizza teams. I bought a whole bunch of pizzas for everyone. Why is our code still broken? And they don't realize that there's some secret sauce that only really works at Amazon, such as giving things ridiculous names, but that's a bit of a low blow. It's strange seeing how culture is shaped by a company and leadership principles or their equivalent at other companies seem like outgrowths of what those companies have become. But in your case, I think that talking about what you take away is, I guess, more nuanced than that. It's not just about, well, we did X thing and Y result happened. Instead, you're talking about things like the writing culture. For folks who haven't been obsessively stalking a $2 trillion company for the past four years, explain a little bit more about what the writing culture is like, please. Yeah. So for example, as a solutions architect, you typically are partnered with an account executive, depending on what group you're in. I was part of a Greenfield team.
Starting point is 00:08:25 And there's a lot of writing, right? There are quarterly, essentially business reviews. They call them something different internally. But you typically partner with your account exec and write a six-pager or account territory plan and spend a lot of time and effort getting that right and being really thoughtful about it. And that gets reviewed with the entire area team, including leadership every quarter. And that is the foundation for how you're going to go talk to your customers, how you're going to get different
Starting point is 00:08:48 projects, how you're going to help them succeed. Obviously, it's a very customer-obsessed company. So it's definitely skewed towards how do we help our customers succeed? What can we offer? What can we help with? What programs do we have? And all that is laid out every quarter in a really well written document. They're really notorious for bringing red pens to a lot of these business reviews. And the way it works is we get into a room and everyone will read through the person's document with a red pen or some other way to mark up questions
Starting point is 00:09:14 or thoughts they have about what's written there. D minus, you can do better than this, see me after class. Yep, something like that. But really it's tough that person think through what they're writing and what their plan is, right? We all want to see each other succeed. And that person puts a lot of thought into what they wrote. So we want to help them shape that
Starting point is 00:09:30 from different people's experiences and diverse points of view. So it's really awesome to see and something that I really, really enjoyed and will stay with me for the rest of my career, for sure. So what I always find interesting as well, and I'm not necessarily speaking to your specific situation, but the idea of a person leaving a company to go somewhere else and then boomeranging or coming back to the company that they left. Now, maybe that's because I can't envision doing that sort of thing myself, mostly because, again, I'm not a very good employee.
Starting point is 00:10:01 My resignation letter was at one point carved into my boss's office door and I'm generally not eligible for rehire. But what inspired you to leave and what inspired you to return? Yeah, I think honestly, in hindsight, you know, I enjoyed my experience at AWS, but in my mind, I was kind of optimizing for the wrong things. When I kind of reflect back on it, I was kind of optimizing for obviously like just. When I kind of reflect back on it, I was kind of optimizing for obviously just the prestige of working at AWS and what that entails. And really, as I reflected on when I was thinking about going back to ReVentures, I'd rather just optimize for happiness, right? I think ultimately that's what's going to make me successful in my career is if I do what I enjoy and really just focus on what will make me happy versus other
Starting point is 00:10:45 ancillary things like money or prestige or working at the cool startups in Silicon Valley or things like that. Everybody's going to have a different thing or a different view on what makes them happy and what they want to optimize for, but that's kind of where I ended up. There's something to be said as well for watching companies change. I mean, Red Ventures is, to my understanding, not exactly an ancient company. I mean, they've been around for a while. I've been to your office for a DevOps Days
Starting point is 00:11:14 that you folks hosted a couple of years back. And the office doesn't really do justice. It's a campus, especially for those of us who live in San Francisco. It's one of those, oh, wow, this would cost only several trillion dollars in real estate to make happen. It's phenomenally huge and overwhelming. But to my understanding, it's not a company that's been around since the 1800s.
Starting point is 00:11:32 No. And also, that is definitely one of the things that also was exciting about coming back to Red Ventures is our campus is beautiful. And it's kind of tucked in the suburbs of Charlotte. But yeah, we haven't been around that long. So Red Ventures was founded in 2000. And we've grown significantly since then. Now we're up to like 3,000 employees in 10 cities across the US.
Starting point is 00:11:51 And also we have offices in the UK and Brazil. But like our campus, for me, it'd be hard to see another campus compete with what we have. It's pretty amazing. We have tennis courts, we have pickleball courts, bowling, basketball courts.
Starting point is 00:12:04 Obviously, we've posted you for DevOps State Charlotte in the arena auditorium that we have on the campus as well. So it's really awesome. What I find interesting is despite the fact that they've only been around 20 years, my God, it was 2000, really 20 years ago, you've become a large company. And obviously, cloud wasn't really a thing when you were getting started. So you've migrated into the cloud or are migrating into the cloud. I'm sure you can give much more clarity on that than I can. But it also means that as a big company, it's not quite like a Twitter for pets style migration, where, okay, I'm going to redeploy my single tenant application, and even that's never simple. But okay, it took a few months, and then it was done. That's a lot of inertia, a lot of stuff to move,
Starting point is 00:12:46 a lot of culture change that has to happen. What is that like? And you were gone for that long. Did you see significant progression during your absence between the time you left and the time you returned? Yeah, honestly, we've had explosive growth in our usage of cloud just in the past four and a half years since I first started there.
Starting point is 00:13:04 But at a certain point, we took the stance of anything that we're going to develop that's a new project or a new application, we're just going to go straight to the cloud and build a cloud native and take advantage of the capabilities in AWS. We're not going to deal with hybrid cloud. We don't want to spend our energy and focus on that. Let's just start building the cloud native muscles and learn more about the platform. And then over time, we'll transition to things that are still running in the data center and get those moved to the cloud, which we're actively in the process of migrating
Starting point is 00:13:32 our final apps that are running on-prem. So it's been really exciting to see the growth, but what comes with that is a lot of waste and a lot of learning, right? There are things that we didn't know about AWS and how to optimize costs in the cloud and what the features and capabilities are. And even the things that we did learn know about AWS and how to optimize costs in the cloud and what the features and capabilities are. And even the things that we did learn, those things change.
Starting point is 00:13:49 Right now, serverless is a lot more mainstream than when we first started and kind of changes how we approach building and designing applications. We want to take advantage of the ability to scale to zero and save costs and shut things off when we don't need them. And all those things we've been learning through over the past four years. And I think we're in a really good spot now. People mistakenly believe the cloud just means it runs on someone else's computer.
Starting point is 00:14:10 In practice, it means it runs on money. And the capability story is fantastic, but there's no upper bound the way that there is at a data center where it's, well, we're completely out of capacity. We need to break ground on another one. Sort of limits what the upside cost can potentially be in a short period of time.
Starting point is 00:14:29 There is no upper bound with cloud in the same way. It is not possible via conventional means to fill S3 in a given AWS region. Found out accidentally that it is not possible. Added the fuck to that in the bill. But there's definitely a story of discovering then that if everyone has the option to spin something up, that's great, there needs to be something
Starting point is 00:14:53 that winds up as almost a following function of, that's great, that experiment was good to know, we got results out of it, now what? How do we enforce turning things off when people are done with them? Because in practice, you spin something up, great, you're going to retire before that instance does, left to its own devices.
Starting point is 00:15:09 Yeah, for sure. That's something my team has been a focus for my group over the past few months is, how do we tackle being more efficient in AWS and reducing our waste? We are a pretty large company with a lot of AWS accounts. And there are things that may not look like they're expensive in a single account, but when you scale that across
Starting point is 00:15:27 50, 100, 200, 300 accounts, that really adds up. It's reflected in the bill. So for example, some of those smaller things we've been looking at is do we need four NAT gateways in non-production accounts? No! Sorry, did I say that out loud? So we have account automation that will create accounts. But as part of that, we want to revisit that and say,
Starting point is 00:15:48 maybe we can just get by with a single NAT gateway in non-production accounts. We don't need the same level of redundancy there. What about CloudWatch log groups? Do we have retention policies by default with either our Terraform modules or just the ones that already exist? Are we just wasting money there? Duplicate Cloud cloud trail trails is kind
Starting point is 00:16:05 of a sneaky one, right? If you have an org-wide trail, that's free. But if you create a second trail, you're going to pay for all those events that you're already getting for free. And that adds up when you have hundreds of accounts. So it's a lot of these really small things that, to a single team that's operating in one or two accounts, doesn't look crazy. But when you look at the big picture across the organization, you're spending a lot of money on those things. The first and challenging part, of course, is ensuring you have decent governance, ensuring that there's a relevant way of enforcing good practices, building guardrails in. But then the next question very quickly follows, how do you do that organization-wide in a way that doesn't absolutely suck and or wreck the culture? Yeah, I think I've definitely battled with that.
Starting point is 00:16:51 We've gone through different iterations to figure out what works best, and I think we're still attacking that problem. One of the easy things that we've done is really leveraging what AWS already provides. So in this case, like trusted advisor cost optimization checks, right? It's a really easy place to start where you can go and get those across however many accounts you have and just service that information to engineers. Typically, engineers want to do the right thing and optimize their costs, but they don't know where to start or they're tied up building features or providing business value. So if we can reduce some of that friction there and just go through and aggregate
Starting point is 00:17:24 all the findings that AWS is recommending, we don't have to go do that work to get the recommendations ourselves. We can leverage what they've already built and we just service that to them, right? Maybe we create jury tickets for them. We track those month over month. We show like what the total savings opportunity is since AWS already provides that information. So we're just trying to find where we can plug in and include things together and reduce some of the friction so we can service the savings opportunities to the right people. The challenge, of course,
Starting point is 00:17:51 is you want to be responsible. Hey, who copied that extra petabyte of data that no one seems to need somewhere else? And that's a painful experience that no one really wants to deal with unless they have no choice. But you also want to get out of the way and let people innovate. Because left to their own devices, first, you're going to have those same guardrails firing off.
Starting point is 00:18:12 Well, hang on, you just spun up another instance that's going to cost $7 a month. You better get a director-level approval to do that, which is ludicrous. And it also reinforces some of the bad behaviors that you'll see that make a lot of sense in a physical data center environment that makes zero sense to the cloud. It shouldn't take you six weeks to provision an instance the way it would a physical server. And the heavier the process gets, the less likely people are to turn things off once they're done using them because it's so painful to get it back if they need to test it out again. I've talked to companies where when they're not running workloads on their task cluster,
Starting point is 00:18:48 they'll have it do something like folding at home just so it doesn't show as idle and then accounting yells at them, which I think is just monstrous as far as having to subvert corporate process to do the right thing goes. Yeah, and that's, I think, what we want to avoid, right? Like, if you have too much process and really make it cumbersome for your engineers to do what they need to do for the projects and businesses they're working on, you're just going to lose some of the trust there
Starting point is 00:19:12 and honestly probably have some more attrition. Because engineers want to be able to do what they need to do to move the business forward. And with their roadblocks that don't make sense or are just arbitrary processes, because that's the way we decided was best when we first started using the cloud. Like you have to evolve those things, right? And I'm much more of a fan of the trust but verify model when it comes to both kind of cloud security, but also just,
Starting point is 00:19:33 you know, cost efficiency in the cloud and not necessarily blocking engineers up front. This episode is sponsored in part by our friends at New Relic. If you're like most environments, you probably have an incredibly complicated architecture, which means that monitoring it is going to take a dozen different tools. And then we get into the advanced stuff. We all have been there and know that pain, or will learn it shortly. And New Relic wants to change that. They've designed everything you need in one platform,
Starting point is 00:20:03 with pricing that's simple and straightforward, and that means no more counting hosts. You also can get one user and 100 gigabytes per month totally free. To learn more, visit newrelic.com. Observability made simple. One thing that falls in directly under your wheelhouse, and of course is near and dear to my heart, is managing the AWS costs
Starting point is 00:20:25 and managing the cost aspects of the AWS environment. And that's great. The problem that you have with that start to finish is that every time you talk to a company about how are you managing AWS costs, they always give the same answer. Specifically, terribly. We're sort of embarrassed about it, and therefore we're not going to talk about it. I mean, I fix AWS bills for a living,
Starting point is 00:20:51 and I've never yet found any company that will stand up and say, oh, we've solved this problem. We're doing it super well, and it's awesome. The only folks who do are great. What are you spending every month? 20 whole dollars. Great. That doesn't actually speak to much. And it gets worse when it becomes a purely engineering problem, specifically because you wind up having engineers who lack context. And I don't mean this as an insult to engineers, but you have this problem where an engineer left to their own devices will gladly spend eight weeks trying to golf $200 a month off of their AWS developer environment. They cost more than that when they go for their first coffee
Starting point is 00:21:31 break. There's no business value to optimizing at that scale. Then, of course, you have the $2 million a month thing running in AWS. Yeah, maybe spending a couple of weeks knocking that bill in half makes sense. Yeah, absolutely. You definitely don't want to optimize for things that, in the grand scheme of things, are not that important. They're going to take up too much engineering time for little value in return. There's not a good ROI story there for the business. So you really want to just attack what are the biggest problems. What are the problems we have across all the accounts, across the organization? What are the problems we have structurally? How can we improve the culture and have this flywheel effect of just building a
Starting point is 00:22:08 culture of cost efficiency? Because I think that is what pays off and has a much better ROI. What are the inputs we can put in initially and just get this flywheel effect, right? Part of the problem, I think, is building that flywheel and not getting lost. The challenge that I've seen about costing, and large is that it requires persistent, sustained effort, but it's never a lasting priority for most companies. Usually there's a freak out whenever the AWS bill shows up that lasts a few days, and then something else takes precedence.
Starting point is 00:22:37 But like death and taxes, the AWS bill shows up again at the following month and restarts that process and builds more urgency. But it takes a lot of getting it wrong before companies start to see the value in getting it right. The painful part for me is that if you start off doing a few things that aren't that much additional work, you save yourself months of effort down the road. Yeah. And the other thing is like once you get to a certain size with your account footprint and kind of your bill. It really, really takes
Starting point is 00:23:05 dedicated people that are staying on top of the bill, reviewing the bill, trying to optimize, managing RIs, which is just a nightmare. Thankfully, they have Compute Savings Plan now, but obviously it only covers compute. I think my number one AWS wishlist item is for sure a data equivalent of the Compute Savings Plan because that just makes things so much easier. Trying to optimize ROI coverage and then utilization, it's just a moving target when you're an organization of a certain size and you have so many engineers.
Starting point is 00:23:33 The chargeback or showback model becomes painful then too because you have an account that predicts their usage super well and says to buy exactly what they need but those wind up applying to some other workload who decided to YOLO something into production. How do you attribute that back? Not for blame purposes, but for helping to optimize a footprint or understand a predictive model. Exactly. And the entire process of how do you even validate that they need an RI for that workload?
Starting point is 00:23:56 Is it something they're going to use for a year if you're looking at one of your RIs? There's so much legwork that's manual that has to go in there. It can become a nightmare when you get to a certain size and staying on top of it. It's not really a game that you're going to win at with how they're structured today. And that's definitely one of our biggest pain points that we're still trying to get better at. The hardest part for me is figuring out how do you fit whatever optimization approach you're taking into the company culture without breaking it? Because you don't want a bunch of engineers to generally have to think about this stuff, for the most part. I've been an advocate for a long time of the idea that what most engineers need to know about AWS billing fits on an index card. Sure, you need in-house
Starting point is 00:24:36 expertise past a certain scale. I'm not denying that at all. But that doesn't need to percolate out to every engineer working in the environment. There could be a centralized, almost SRE-style model where the cost optimization experts go around on a workload-by-workload basis and are available on a consult level. That seems to work better than teach every engineer to care about the bill. There's got to be a happy medium somewhere in there, but the hardest part is always culture. Absolutely. You don't want your business engineers spending their time worrying about RIs or optimizing savings plan or what do we need to buy for next month or some of those other minutiae details. That should be offloaded to a central team or made easier through things like savings plan. You want them more focused on application optimization in terms of cost.
Starting point is 00:25:24 What are some of the things that we can tweak? Are we building things the right way from the get-go? Are we looking at serverless architectures, things that can scale to zero in some of those optimizations and not focused on some of the RIs and other workloads? Part of the biggest thing that I think teams need to, I guess, wrap their heads around is it's important to focus on AWS cost control.
Starting point is 00:25:45 At some point, if it's not a problem for you right now, that's cool, we'll check back on you in a quarter or so because it's never a problem that goes away on its own. But at some point, conversely, it's time to stop focusing on it and stop caring because above some certain level of optimization or focus on attribution, learning to let go and have some wiggle room
Starting point is 00:26:04 is always going to be the right answer. And you were mentioning earlier that serverless is an area of interest for you folks. This is also sort of a different problem, particularly when communicating effectively with accounting. Because the idea behind serverless is that your usage is much more optimized, but it is also way more variable
Starting point is 00:26:24 because it's completely beholden to customer workload. Yeah, absolutely. Although I still think in the end, you're definitely better off if your use case fits it, designing almost like an event-driven architecture or just working with serverless technologies because you still have the ability to scale to zero, right? Like you don't have to think about shutting things off when you're not using them or when usage is low or managing infrastructure in your non-production accounts. And I think we're getting better at that. We're moving more towards that direction. But we built a Terraform module to pause resources in non-prod and sandbox accounts because there was a need there, right? Because we still have a lot of applications that are running on EC2 or even
Starting point is 00:27:02 just using things like RDS and Redshift that now you can pause before you couldn't. So we kind of built a Terraform module that deploys a Lambda function that pauses those things on a schedule. So obviously on the weekend, if your team is not working or overnight, you can pause your infrastructure running in your non-production accounts. And we've also added the abilities like send those events to EventBridge so we can track the usage and the stop and start times. And in the past few months since we've kind of deployed that module, we've saved 290,000 hours, resource hours, which is essentially bending time by saving 33 years in
Starting point is 00:27:35 a three-month span. There's something incredible to be said about that. It's fantastic to hear stories like that. The challenge, of course, is explaining to accounting that, oh, so what does your 18-month prediction look like as far as spend goes? The honest answer that doesn't lead to nonsense is usually going to look an awful lot like, well, it's going to depend as a function of user growth and traffic. This doesn't actually answer the question, but it does kick the can over to the BI folks rather than engineering in many cases. Again, culture is always the hardest part. Yeah, I do not envy the folks that have to go and figure out the forecasting long-term with CloudSpend because it's such a hard problem to solve. It's essentially impossible to forecast
Starting point is 00:28:18 when you have variable growth and customer usage and you're growing as a company. It's really hard to track. Yeah, we've started doing that as a consulting option on our side. But invariably, it's always a second level of offering after, first, let's figure out what's in the account and do an optimization pass and understand what's going on in a meaningful, intelligent, intuitive way. That's never easy. And figuring out how to get there is always, I guess, sort of, it's a journey more than it is a destination,
Starting point is 00:28:48 which is usually the sort of thing you say about digital transformations when you're discussing that about a cloud bill. That's what drives accountants to drink. Yep, absolutely. So changing gears slightly, you and I first met a few years back. You're an organizer for DevOps Days Charlotte.
Starting point is 00:29:06 And I have a whole story about the time I attended there. In fact, I've been there a couple of times now. But first, tell me about how you got more or less dragged in to helping to organize the DevOps Days, which if you look it up in the dictionary is sort of the poster child for thankless job. I don't know about that. It's definitely rewarding. But yeah, it was a really interesting story how it kind of came about. I remember attending DevOps Days DC in 2015.
Starting point is 00:29:32 I believe it was the first one. It may have been the second one. And Nathan Harvey is one of the organizers there. And they proposed an open space or a breakout topic on organizing DevOps Days. And I went to that and Nathan Harvey showed up and the only other person who showed up was Jason Hand, who was one of the organizers for DevOps Days. And I went to that, and Nathan Harvey showed up, and the only other person who showed up was Jason Hand, who was one of the organizers
Starting point is 00:29:47 for DevOps Days Denver at the time. So I had them all to myself for 30 minutes, so I kind of just asked them what the process was, what did they learn from it, what were some of the pain points, and I went home from that, and I was like, man, that sounds really, really cool. It's awesome to get a community of folks together that are all interested in this DevOps thing.
Starting point is 00:30:04 And a few months later, that was in June, in November, we started the first ever DevOps Day Charlotte in a really crappy hotel by the airport, not knowing what we were doing. But it was the first time we learned a lot. And this past February, we ended up celebrating the fifth anniversary of DevOps Day Charlotte, which you also were at. That still remains one of my favorite talks I've ever given. And it's probably time I told that backstory in public a bit. So a few years ago, I gave a conference talk on things I'd learned interviewing for jobs, how to handle job interviews,
Starting point is 00:30:38 how to handle salary negotiation, like the usual stuff you'd expect. The problem is, is all of this came from my own experience. So what I accidentally built was how to handle job interviews as a white guy in tech, which is not the most accessible or inclusive talk. So I stopped giving it. And what I really liked about the keynote
Starting point is 00:30:58 was it let me resurrect a highly, highly modified form of that talk that I co-presented with a friend of mine, Sonia Gupta, who before entering and later leaving tech was an attorney, both as a district attorney as well as a defense attorney. So she had a lot of thoughts on how negotiation worked and she made the talk a thousand times better.
Starting point is 00:31:20 It probably remains one of the most impactful talks I've ever given, because usually, let's not kid ourselves, it's dumb jokes about cloud technology that's not going to stand the test of time. But interviewing skills are the sort of things that absolutely will pay dividends for someone's entire career. And I still get outreach over that talk. People discover the video every month and people ask questions. That was a fantastic talk. We had a lot of feedback after that talk that people loved that talk and it was really, really impactful for them and kind of how they think about salary negotiation and job interviewing as a whole. That was an awesome talk. Yeah, I hope to give another talk with her again at some
Starting point is 00:31:53 point. Just absolutely a fantastic experience. I found that on some level, and this is going to sound ridiculous and I don't even care, I'm going for it anyway. I've gotten bored with some of the aspects of giving a talk where I get on stage. I know exactly how the entire talk is going to progress, start to finish. Having someone else on stage makes it a lot more entertaining for me. You can bounce things off of someone. You're never going to give anything approaching the same talk twice just because it goes back and forth so well. And it's really neat seeing that interaction. I've learned a lot because since then, I've started this podcast, among other things,
Starting point is 00:32:30 and I've gotten better at having the ad lib recording style of conversation where you don't really know where it's going to end up. I love that style of talk, and I'd love to see more people doing that. Yeah, I think those are some of my favorite talks, right? Like you can tell it's not necessarily scripted. There's a little bit of improv mixed in, especially whenever you're on stage.
Starting point is 00:32:48 But those are usually pretty rewarding and enjoyable. I don't usually show people behind the scenes, but generally speaking, the way this podcast starts is we talk for five or 10 minutes beforehand just to get a broad outline of, make sure that the bio I have is up to date and figure out, okay, what do you want to make sure we do or don't talk about?
Starting point is 00:33:04 Cool, let's go. But I don't have a giant list of topics I'm staring at of going point to point to point. I do it entirely on an improv basis. That's not because I'm awesome at this. It's because I'm terrible at planning ahead. So you've got to be good at improv when that stuff happens. And more often than not, it leads somewhere pretty uplifting and positive. But it's neat seeing that sort of thing unfold on talks because no two people have the same talk preparation approach. So it really forces you to learn to collaborate with people in a different way. Yeah, and kind of tying into just different talk formats,
Starting point is 00:33:36 one of the other things I really love about DevOps Days is the open space format, which you've attended plenty of them and have experience with. But it's just more of an audience-led style of interaction. People walk away from those initially kind of like, I don't know if I'm going to like that, I don't know if I want to go to those. But typically once they go to a couple, they really enjoy that and it ends up being the favorite part of the conference for them. Because they get to talk to their peers and understand actual war stories, right? What are things that are happening in companies and challenges
Starting point is 00:34:04 people have solved and challenges they're still facing and just talking through those things. It's a lot of fun. Yeah, I think there's a lot of value to it. And frankly, I've stopped speaking at most of the DevOps days for the last year or so, primarily because, let's face it,
Starting point is 00:34:18 I'm trapped at home along with everyone else and there's only so many times I can talk into a microphone. But also in part, it's because I think that DevOps days are a terrific way to get started talking. And I am thrilled to see voices we have not heard before from folks who don't look like me, who are able to get up and start telling their stories. My offer that I make periodically, and it stands, I'll renew it now, is if anyone wants to give a talk but doesn't know how to get started or doesn't think they have anything to say,
Starting point is 00:34:47 please reach out to me. That's one of the things I'm incredibly passionate about is helping people tell stories. It's not as hard as it looks. There's a lot of things you can do to make it easier on yourself, a whole bunch of shortcuts and cheats, as it were. I've done a bunch of tweet threads on this,
Starting point is 00:35:04 but those are always ephemeral. It's probably time for another one. And I'm very interested in helping people get on stage and learn to tell the story, because I love it. And I like watching other people catch the gift as well. Yeah, that is awesome. And honestly, the DevOps and DevOps East community is one of the most inclusive and welcoming communities that I've been a part of. It's really awesome to see people share knowledge and are super supportive, especially for first-time speakers. I've noticed a lot of DevOps Days conference intentionally ask for first-time speakers
Starting point is 00:35:34 because they're going to provide support for them and welcome them. And that's one of the other aspects I really love about being part of DevOps Days. Yeah, I think that's probably a good place to leave it. If people want to learn more about who you are, what you're doing, possibly come and work with you, where can they find you?
Starting point is 00:35:50 Yep, I'm on Twitter at Alphonso underscore C. And we are definitely hiring. If any of these problems sound interesting to you, we're looking for kind of senior platform engineers and also software engineers across the board. So you can find us at redventures.com. And I have to say, based upon conversations I had when I was out there with you folks, I did not get a big company feel, you know, except for the fact that we were on this enormous,
Starting point is 00:36:12 gorgeous campus. But I didn't get a cultural sense of what most people think when they hear enterprise company at large scale. It's not one of those cultures where nothing ever changes and everything is sort of frozen in amber. I know that that tends to be a common perception, but I don't believe that that's true. Yeah, no, you're right. We definitely take pride in acting like a startup, right? That's one of our kind of core principles. I'll take it a step further.
Starting point is 00:36:37 You act like a startup without the incredibly toxic, damaging, and trash fire aspects that people often associate with startup. Fantastic point. Yes. We have some really compelling belief statements that really give me pride about working at Red Ventures. One of those is, you know, we believe we can be the change we want to see in the world, right? And we're a company that believes we have a responsibility to interrupt social injustice and take concrete action to drive change. And you'll see that as well. I'm happy to chat with anybody that's interested in finding out more. Excellent. I will, of course, include links to you in the show notes as well. Thank you so much for taking the time to speak with me today.
Starting point is 00:37:12 I really appreciate it. No problem. Thanks, Corey, for having me on. Alfonso Cabrera, Director of Platform Engineering at Red Ventures. I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review at Apple Podcasts or whatever platform you'd like. Whereas if you've hated this podcast, leave a five-star review anyway. But also be sure to include a comment about how my talks aren't nearly as good as I think they are. This has been this week's episode of Screaming in the Cloud. You can also find more Corey at ScreamingInTheCloud.com or wherever Fine Snark is sold.
Starting point is 00:37:59 This has been a HumblePod production. Stay humble.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.