Screaming in the Cloud - Replay - Keep on Rockin’ in the Server-Free World with Michael Garski

Episode Date: November 21, 2024

On this Screaming in the Cloud Replay, we’re revisiting our conversation with Michael Garski, the director of software engineering at famed electrical guitar manufacturer, Fender. Prior to ...this position, he worked as a principal software architect at Viant, a principal software architect at MySpace, a manager of internet development at Countrywide Financial, and a manager of system architecture at Fandango, among other positions. He also had a four-year stint in the US Navy, working as an engineering laboratory technician. Join Corey and Michael as they talk about how artists are angels and Fender’s job is to give them wings, how Fender has diversified its offerings in recent years, how serverless is a mindset and how Fender approach serverless technology, how Fender’s traffic surged during the pandemic and how everything mostly scaled up without a hitch, the challenges of teaching students to play instruments over the internet, the vendor lock-in boogeyman, and more.Show Highlights(0:00) Introduction(0:42) Dragonfly sponsor read(1:25) How does Michael describe Fender’s work(2:08) Fender’s work to go serverless(4:13) The impact of COVID on Fender(6:19) Explaining Fender Play and how it works on the backend(9:44) Working with MediaConvert(11:30) Experiences with scaling and hitting AWS service limits(12:52) Why Michael prefers working on the customer side(15:33) The Duckbill Group sponsor read(16:15) Frustrations with gateways and third-party apps(19:03) Managing a massive influx of users during COVID(21:13) The vendor lock-in boogeyman(23:19) Cloud costs vs. saving time(24:49) Walking the fine line of criticism as a director(28:09) Enforcing consistency across services(31:52) Where you can find more from MichaelAbout Michael GarskiMichael Garski has worked in the Los Angeles tech industry for over 20 years, across companies including Fandango, Countrywide Home Loans, MySpace, Viant, and is currently at Fender Musical Instruments as the Director of Platform engineering were he leads the devops, data, and api engineering teams. His focus currently is on building the platform to support the consumer facing digital products for Fender. The most prominent application he supports is Fender Play, a web and mobile application that provides video-based instruction for guitar, bass, and ukulele for more than a quarter-million subscribers.LinksLinkedIn: https://www.linkedin.com/in/mgarski/Original Episodehttps://www.lastweekinaws.com/podcast/screaming-in-the-cloud/keep-on-rockin-in-the-server-free-world/SponsorsDragonfly: dragonflydb.ioThe Duckbill Group: duckbillgroup.com 

Transcript
Discussion (0)
Starting point is 00:00:00 not only the creativity, it's also the flexibility in how you solve it and the ability to adapt and evolve as the services evolve. Welcome to Screaming in the Cloud. I'm Corey Quinn. We talk to a lot of people here on this show who are deep in the weeds of SaaS companies or cloud vendors or cloud vendors cosplaying as SaaS companies. Today, we're taking it a bit of a different direction. My guest is Michael Garsky, Director of Platform Engineering at Fender Musical Instruments. They make guitars, among many other things. Michael, thank you for joining me. Oh, thanks for having me on, Corey.
Starting point is 00:00:42 It's an interesting time for the Redis ecosystem with licensing changes, the Valkey fork, and AWS Oh, thanks for having me on, Corey. a much lower cost. If you're running ElastiCache or Redis cloud workloads in AWS, Dragonfly guarantees a minimum 25% reduction in your bill. Listeners of Screaming in the Cloud can snag a $200 credit to try it by heading to dragonflydb.io and use the code SCREAMING after signing up to take advantage of that $200 credit offer. Seriously, what are you waiting for? And my thanks to Dragonfly for sponsoring this episode. One of the things that I really appreciate about what you do as a company is I can at least presumably explain it to someone who is not super deep in technical weeds without 45 minutes of explainer first. The easy answer is, oh, Fender, you folks make guitars. These days, no one just does one thing, I have to imagine.
Starting point is 00:01:48 How do you describe what the company does? Oh, well, to quote Leo Fender, his view was that artists are angels and it's our job to give them wings. So in addition to actually making and developing guitars and amplifiers, we've branched off into consumer-facing products to actually teach people how to play those instruments. You folks have been relatively outspoken about the various things you're doing at different AWS events. I mean, my approach to that tends to be that if AWS is great at making bricks that you can use to build amazing things with, well, great,
Starting point is 00:02:23 can you draw a picture of the house that you can build with this? No, we're going to have a customer come out and talk about that stuff instead. You folks have been focusing on a lot of serverless work and you've been very public about the fact that you are almost entirely serverless driven in terms of architecture, if I'm not mistaken. That is true. Tell me about that.
Starting point is 00:02:41 How did you get there and what brought it about? So I work in the digital division at Fender. We started, let's see, we're coming up on five years I've been there. So what we did was initially we started building services that could run within a container or on an EC2 instance, but we started looking at Lambda functions. We had a need to ingest a product catalog. So the IT team was able to drop us off a product catalog into an S3 bucket. And easiest thing to do then was just trigger a Lambda function and then process that file. And it just kind of snowballed in from there. I think a common problem when people hear serverless is they think, oh, great, more discussions about Lambda functions. And
Starting point is 00:03:23 Lambda is almost getting something of a tarred reputation in some circles because when we can build amazing things with it ourselves, we love it. But when we ask AWS how to wind up integrating two services or about a feature gap, their response is, oh, use a Lambda function for it. It starts to feel like they're using it as spackle, and the spackle has become load-bearing. Do you view serverless as being purely function-driven or is it broader than that? It's much broader than that. Serverless is a mindset where you're looking beyond just
Starting point is 00:03:54 Lambda functions to using a lot of third-party services so that you can actually focus on your core business. We use Zora as a subscription provider for web-based subscriptions. We use Algolia for full text search. We use a variety of other services so that we can just focus on the core business. One thing that's been on everyone's mind somewhat recently has been the idea of dramatic changes as far as user behavior goes. And in the more traditional environments where you see things like EC2 instances or on-premises data centers, back when the pandemic first hit and companies that were very focused on a model of business that aligned directly with people behaving in
Starting point is 00:04:37 certain ways that they suddenly didn't, would see 80% drop-offs or more in their user traffic, but their infrastructure spend just kept hanging out exactly where it was in a straight line. So on some level, it feels like, yes, the whole point of cloud is that it can be elastic, except no one builds it that way for a variety of reasons. When COVID hit, what changed for your business? Change for our business is we launched a program called Playthrough, where we did this about a year ago. We started it. We gave away three months of offender play for free.
Starting point is 00:05:11 It was a single-use code that a user would redeem and no credit card required. And over a period of five days, we saw our traffic increase by more than 10 times. And we had very little changes we needed to make. Everything scaled up. We had no issue with, we used a lot of Lambda functions, DynamoDB, everything just scaled up fine. The only point that became a bottleneck was our Elasticsearch cluster. However, beefing up the nodes and adding a few more nodes resolved that issue immediately. So I'm going to go out on a limb and postulate that you folks saw increased pickup when the lockdowns hit, if for no other reason than,
Starting point is 00:05:51 well, I'm trapped at home and I'm tired of staring at the guitar on the wall. I may as well learn to play it. I would guess I could be way off base on that. No, no, that's very true. Even since then, even after that program has expired, of course, not everyone then converts and sticks around, but many, many did, many more than we thought did stick around. And as best I can tell, that is an app that works in the web, it works on mobile, and it's a video-based instruction tool for guitar, at least, but some other instruments as well. How did that come to be? Did that exist before COVID hit? Has that been something that's been in the works for a while? Or was it, well, we're going to do a two-week sprint and build this thing from scratch? No, we launched that this June. We're coming up on the fourth anniversary since it's been launched. We launched this in summer of 2017. One of the problems I've always found is that it's
Starting point is 00:06:57 challenging to learn to do something that is as, I guess, physical and intricate, etc., as a planned instrument without having someone in the room looking at you and smacking you with a stick whenever you do things that are wrong. Nope, that's a bad habit. If you keep doing that, it's going to hurt you. How do you approach that as a company from a non-interactive perspective
Starting point is 00:07:17 of someone who's going to watch a video and do things and maybe it'll work, maybe it won't, particularly in light of things like, well, the competition is YouTube, which, you know, I'm going to roll the dice and sometimes I'll see a great tutorial. Sometimes I'll see one that I don't realize is teaching me terrible things. And then it's going to recommend some baseless conspiracy theory because YouTube. How do you differentiate that? What makes VendorPlay different? So currently you're right. It's just a video-based instruction app. There's not any way to
Starting point is 00:07:44 provide feedback, direct feedback to students within the web and mobile applications. However, we do have an online community, and our instructors, Fender Play instructors, do an office hours feature, where they'll actually answer questions live and talk to students. We are investigating and kind of doing some really research and some possibly being able to provide that type of feedback to students. We are investigating and kind of doing some early research and some possibly being able to provide that type of feedback to users. But it's a very challenging problem just due to the nature of you're playing an instrument that has multiple strings. You're trying to pick out the chord that they're playing and the timing. But it's something we definitely need to add. There's something to be said as well for the kind of care and attention that you folks wind up
Starting point is 00:08:24 putting into your media, where this is how you finger record. And someone on the YouTube video will do it for two-tenths of a second, and they're filming it with a potato that isn't focused properly and pointing at the wrong part of the guitar. You folks have a high bar for quality on this. Is that done in-house? Do you wind up just going through a bunch of random folks that you just wind up offering a bunch of gift cards to or free guitars to do this? How does the program work in the back end? So we have an in-house curriculum team that puts together the lesson plans to really help people learn in small bite-sized lessons so that it's not too overwhelming at once. And that curriculum then is shot and filmed by an in-house video team that put that together. They upload the data into S3
Starting point is 00:09:07 for the final cut. Then that gets transcoded via media convert, and we serve it up via CloudFront. It's rare to wind up talking to a company that is something of a household name about something that they're doing and hear the AWS services that they're using not trend toward a baseline mean, if I can be so bold. Normally, you'll see some of the case studies like, oh, this is an online bank. What services are they using? Oh, they're using EC2 and S3 and load balancing because did you miss the part where it's a bank? They're not going to use these far future services, but due to regulatory risk, among other things, in many cases. You're using Elemental Media Convert, which is one of those relatively high up the stack
Starting point is 00:09:49 offerings that isn't broadly known. It's one of those services that is focused on specific use cases and specific industry verticals in a way that a baseline primitive service isn't. What does Media Convert do? What it does is it takes the final edit of the video, and we have several different presets so that it will put it into an HLS format with different bit rates so that the user's getting the best quality video
Starting point is 00:10:14 depending on their bandwidth. When I looked into it in the early days when it was first launching, I found that it looked an awful lot like Elastic Transcoder, which is a service that they've had for a while. Only they changed up some of the capabilities. It's obviously far more capable as a service. But they also added something that felt like 15 different billing dimensions to it.
Starting point is 00:10:33 So what is this going to cost me? Well, we're going to run it for a month and find out if we're still in business. And it seemed like it was one of those very difficult to get started with and run experiments with service. Now, obviously, services evolve over time. When you started looking into it, was that experience roughly akin to what you found? Or am I completely and unfairly slandering the product? We actually started out using Elastic Transcoder and then moved over to Media Convert. I believe that was last year. We found it to be a little bit easier to use. And the pricing overall in transcoding the videos for us is really a drop in the bucket as compared
Starting point is 00:11:12 to actually hosting them and serving them up via CloudFront. And when we switched over to media convert, we adjusted our settings to lower the maximum bit rate for a given video. We found that after a certain point, the quality to the user just doesn't really improve, and yet we're paying to serve the larger video. One statistic that I found was that in March of 2020, which I believe we're still in at this point, it's that endless September model applied to March, you wound up seeing over an order of magnitude in traffic increase within five days. And looking at that through a lens of traditional architecture, that means that nobody sleeps a whole heck of a lot. Given that you're in on the serverless story, and you have been
Starting point is 00:11:57 since before that hit, what was that scaling experience like for you? Scaling experience was completely seamless. We use a lot of Lambda, DynamoDB, Kinesis, SNS to glue things together and no problems whatsoever. We just had to bump up our Elasticsearch cluster a bit. That was really the only thing because we saw some latency starting to rise on some of our APIs. Let me ask the uncomfortable question then, because whenever I try to scale things up quickly in a cloud environment, what was your experience with smacking into various AWS service limits as the traffic grew? Initially, we actually requested some service limits increased to make sure we weren't hitting the concurrent Lambda invocation limit. And same thing with Cognito, making sure that we weren't going to hit any limit as far as sign-ins and things like that.
Starting point is 00:12:46 So we're able to just put in requests and they serve us around pretty quick turnaround time on that as well. It really does seem like there's a strong benefit on the serverless space. But I had to double check before we started recording that you do in fact work at Fender because you are a staunch advocate for observability. And usually when someone is that passionate about observability, you can guess that they work at an observability slash monitoring company. It's akin to the idea of someone selling mattresses telling you that mattresses are great and you should have four of them. You're on the customer side of that and still very passionate about it. Where'd that come from? It came from my time years ago when I worked at MySpace,
Starting point is 00:13:29 if anyone can still remember that, working on the search systems there. And as the company started winding down to laying people off and being one of the only people left working on those systems, being able to know and understand them, you just have to. So you have to continue to monitor and find ways to monitor it. And that really ingrained how important instrumentation is and being able to really understand the health of your application as it's running so that you can see like, yes, everything is good. And then like something doesn't look right so that you know where to start looking and you can be alert of a problem. So I tend to view the world in olden terms where monitoring was what we did.
Starting point is 00:14:12 And we use something like Nagios, which was the second worst option out there because everything else felt like it was tied for first. I also take a somewhat regressive view that observability is to monitoring as DevOps is to being a systems administrator. It's the same thing, but by using the more modern terminology, you can charge more for it. I'm going to go out on a limb and guess that you take a somewhat contrarian view to that. Yes, yes, I do. It's about really understanding how your application is running. It's not just looking at, oh, how many HTTP 500s am I serving up per hour? If I hit a threshold for the last hour, it's a lot more than that. It's really being able to really dig in and see what the issue is or what's working really well. And to that end, we rely on two services
Starting point is 00:14:58 for this. We use Honeycomb and Epsilon. Honeycomb kind of acts as our top layer because it gives us the really, really good high cardinality metrics where I can punch in a user ID and I can see all the API traffic that this user has performed, as well as even just like when we launched the playthrough in our traffic rows, the reason we discovered that our latency was dropping was due to a service level objective being triggered in Honeycomb on latency. And we were able to respond to that using that before customers really noticed anything at all. Here at the Duckbill Group, one of the things we do with, you know, my day job is we help negotiate AWS contracts. We just recently crossed $5 billion of contract value negotiated. It solves for fun
Starting point is 00:15:46 problems such as how do you know that your contract that you have with AWS is the best deal you can get? How do you know you're not leaving money on the table? How do you know that you're not doing what I do on this podcast and on Twitter constantly and sticking your foot in your mouth. To learn more, come chat at duckbillgroup.com. Optionally, I will also do podcast voice when we talk about it. Again, that's duckbillgroup.com. As an Epsilon customer myself, I'm always conflicted when I find myself going into their service and using it to figure out what the heck's going on with my giant pile of Lambda functions and API gateways and whatnot wired together, because the experience is uniformly excellent. But I'm also frustrated in that it needs a third party to even begin to allude
Starting point is 00:16:36 to what's going on. It feels on some level like the vendor that is providing this service to me should be reasonably effective at telling me what it's doing and when it's breaking. I understand that how I wish the world is and how it actually is are two radically different things, but does that ever strike you as well? Whether or not AWS should be providing that type of level, that seems like more of a service that you can have competition and other vendors that really specialize and get in the weeds on it. I don't think AWS needs to provide every service you could possibly use for your application. That's not something I'm too concerned about, and I don't really even think it's their
Starting point is 00:17:16 place, frankly. No, no, I understand. The problem I keep running into on some level, whatever I try and diagnose it natively, is I look at CloudWatch and it's difficult to understand that is this, in my case, because again, I'm still early days with a lot of these things. Is it the API gateway that's having the problem? Is it the CloudFront distribution that is tied to that? Is it the Lambda function? Where is the handoff? Trying to understand where in a complicated application the failure is occurring is a challenge. And let's be clear, most of that is a problem of my own making because I didn't have the good sense to instrument this thing in a reliable, repeatable way when I built it.
Starting point is 00:17:52 It feels like everything is tied together with duct tape and bailing wire and spit and a bit of luck. As a counterpoint, the more companies I talk to, the more I realize that, no, no, this is actually how most people feel when they look at things that are working. It's, you know, it's terrible. It's a trash fire, but it makes money, so we're going to roll with it. And there's always on some level a sense of what we've built is very far from the platonic ideal of what we should have built. Does that resonate with you? Or do you take a step back and look at what you've achieved with a perspective of, this is awesome. More people should do it exactly like this. And honestly, if it's that one, I'd love to take a look at what you've built. I think there's always room for us
Starting point is 00:18:28 to improve on what we're doing because we're constantly learning and evolving to improve both even at such a low level of like, okay, how do we lay out the files in our service repository to make the best organization to make sense all the way up to, okay, how are we going to do tracing and what kind of information do we need to get from that so that we can find problems when they occur? We're always looking to learn what others are doing and talking to others in this space. No one will ever be 100% right. There's always room for improvement everywhere. One thing that you folks have done that I think was really interesting and didn't get as much play as I think it really deserved was that, especially in the early days of the pandemic, you wound up seeing that massive
Starting point is 00:19:16 increase due to giving out almost a million free three-month subscriptions to Playthrough. Additionally, you also worked closely with LAUSD, the Los Angeles Unified School District, to add Fender Play to their middle school music programs curriculum to help supplement their remote learning programs. First, was that all in the same timeframe? And two, what has it been like, I guess, working with a organization that is, I guess, on some level, not particularly cloud-first, I would imagine. When I lived in Los Angeles, I never got the sense that LAUSD was full-on serverless, full-on board with cloud, full-on board with remote learning.
Starting point is 00:19:57 And then the pandemic, of course, exacerbates all of that. Yeah, so those were really two different projects. So the playthrough project, that started in March, and we started working with Los Angeles Unified School District last year during their summer school program. Started out with 1,500 students, and we put it together very quickly. Essentially, we used the same three-month codes that we used for the playthrough promotion so that we could set things up very quickly for students and gave out through our nonprofit arm of Fender, the Fender Play Foundation, gave out 1,500 instruments to these students to use during the summer school program. And that program became so successful, we continued on with them in the fall and now in the current semester, and we will be again this summer. I believe there's 7,000 students in the program now. And working with their IT team has actually been quite nice in dealing with partners. You wouldn't think much of like, oh, it's a school district. What do they have? But
Starting point is 00:20:57 as far as like just ease of working with them, we actually hooked into their SAML provider, Incognito, so that LAUSD students could authenticate when they come in through the remote learning systems. And they were great to work with and very helpful and cooperative. One of the arguments that you'll see that comes up against serverless from time to time is that you are now indelibly linked to your provider. You can't take what you've built with all of these services and just move it over to Azure or GCP on a moment's whim. Now, in practice, people who tend to build for that just build everything on top of EC2 and very little else, and then run it entirely in AWS
Starting point is 00:21:37 and never move it to any of those other places. But was there friction with making that, I guess, architectural commitment to a single vendor? Oh, you're bringing up the vendor lock-in boogeyman. Oh, I absolutely am. Most people who bring that, but I bring it up as a straw man so you can attack it. Most people who bring up the vendor lock-in boogeyman, oh, you have to go multi-cloud, are either trying to sell you something that is required if you want to go multi-cloud, or they're a cloud provider themselves who know that
Starting point is 00:22:03 if you go all in on one provider, it will certainly not be theirs. I think if you properly architect your applications with separations of concerns that you could move to say, okay, say Lambda wasn't working out for us anymore and we needed to take our applications and okay, we're going to put them into a container,
Starting point is 00:22:22 but we're going to stay in AWS. Our applications are set up in such a way that Lambda is basically a deployment pattern. We could easily convert those individual function handlers into route handlers with a minimal effort because the business logic and then the underlying data storage are separated. So it would be feasible for us if we wanted to, say,
Starting point is 00:22:44 move to Azure and use Azure functions and whatever comparable service they have to DynamoDB. I'm not too familiar with a lot of their offerings, but that would certainly be possible to do it with obviously some effort. And really, at the end of the day, the resources you have working on the applications are going to cost you much more than any sort of like software licensing or specific savings you're going to get from a cloud vendor. So might as well go ahead and just use those services that they're providing so that you can just focus on the business. has almost universally been that looking at an awful lot of companies and their AWS bills, it is a challenge to find an environment where the resources in the environment cost more than the people who are operating them. In the context of business, AWS bills seem giant and enormous right up until you look at payroll. And then it's, oh, okay. What's counterintuitive for folks who are
Starting point is 00:23:45 learning this, and I fall prey to it myself, is when I'm playing around as a hobbyist trying to build something, I value my time as free because I'm learning as this goes. And then in that context, especially when I was starting out as a student, it was, oh, great. So this winds up costing me $7 a month. Ooh, that's a lot of money. That's my ramen budget. So I'm instead going to wind up spending eight hours avoiding it charging me anything. It's the exact opposite from the direction you want staff that you're paying to work on these things to go in. How do you approach the idea of increasing the cloud cost if it will save time for your team? It's a balance between, okay, do we need to build this ourselves? And then not only build it, you have to operate it and maintain it. Or what is the cost of getting this
Starting point is 00:24:32 third-party service? And that's really what it comes down to in all of them. And do we actually want to spend time working on this piece of infrastructure that these other people are specializing in and do so well? I've got better things I can have people doing than that. Speaking of people, one thing that you talk about, as you self-describe, is that you wind up not writing a whole lot of code anymore, but you're something of a stickler for observability and enforcing consistency between services. So you'll periodically do things like submit a PR to tweak a log message to put your mind at ease was one example that you gave.
Starting point is 00:25:09 Given that you're a director, which is generally manager of manager style approaches, how do you avoid having those PRs come across to your team as either micromanagement or a condemnation of what they've built? Because I get it. When I see something that's easy and small to tweak, I want to go ahead and get it fixed immediately. I don't want to go
Starting point is 00:25:30 back and forth and play those games. I just want it done. But I'm also always weighing that against, I don't want to have people think that I'm judging them somehow for something I'm very much not. That's a very good point. The larger technical decisions on how things are laid out, I generally just try to, I don't insert myself into. I let the team go ahead and make those decisions and lead that direction and let them take the charge on that. And I kind of take the approach of looking at it as more of a guiding and mentoring and teaching to kind of really hone and instill that discipline in really being able to understand what the applications are doing. And as our team is growing, I have less and
Starting point is 00:26:12 less time to even do those things, but I can go through the systems and go, hey, how come we're not tracing this call to the reCAPTCHA servers? Let's add that in there. And I'll just, at this point now, I mainly just write JIRA tickets to have someone else actually do the work. The more I do this, the more I realize that as complicated as the technology is, the people are in many ways far more complicated and, let's be fair here, non-deterministic. Things that work super well on one person one month could work entirely differently a following month or even with the same person or between teams. It's a constant balancing act on some level. And giving people a sense of psychological safety has always been the biggest challenge. The thing that surprised me about management back when I was running ops
Starting point is 00:26:56 teams was the more, I guess, responsibility you accrue as you rise from individual contributor into the management, rise is sort of a wrong term, it's an orthogonal transition, is that you spend a lot more time on the people problems and your ability to directly control or affect change diminishes because you have to do everything via influence. You get a lot more responsibility with a lot less direct power over the outcome in some ways. Does that align with how you see it? Or do I have very strange approaches on management? Which may be true and why I got out of it as fast as I could. No, that is a good point. Because you are having to influence and guide
Starting point is 00:27:34 and sort of more take a higher level view as opposed to really getting into the weeds of like, okay, what methods are we going to put on this interface? How are we going to see architect the internals of an application? Those are details I just really don't have time for anymore. But larger things is to making sure that we're, okay, it's like, you know, what's the performance of this? And overall, is this something that can be adapted? And as the business needs change, and as we change, then as we learn, what can we do to modify it? And just kind of more just things like guiding and mentoring and really taking a higher level view of that. I'm going to selfishly ask about something that I struggle with myself that goes a bit more into the technical area.
Starting point is 00:28:16 But you talk about enforcing consistency across all of your different services. What does that mean? Similar coding styles? Similar instrumentation? Because I look at the things I've built and the microservices that power my internal nonsense, and each one of those is very different than all the rest. So whatever your version of consistency is, I know I'm not doing it, but how do you view it? So there's really two types of consistency. The one I really refer to the most is in observability. So that if you've got a thousand
Starting point is 00:28:44 Lambda functions out there and each one is logging observability, so that if you've got a thousand lambda functions out there and each one is logging things slightly differently, that's just a pain to deal with. Realistically, dealing with a thousand unicorns is a real pain. So through that observability, at least in lambda, we use an internally developed middleware to make sure that the logging is consistent and it's easy enough to use. And then other consistency, like just within projects of how we lay things out, that's something that's consistently evolving. What's the folder structure and how we organize the code? And we've kind of been evolving that over the last three years.
Starting point is 00:29:18 And, you know, within about the last six months, we've come up with like a really good pattern and a template for the future. And it's not much different from what we started out with, but it's a little bit easier to really do comprehend as a new engineer coming in. It makes more sense. I have to ask, and I understand if you don't want to give a particular endorsement in any direction, but do you go through serverless frameworks, SAM CLI, the CDK, using the console and then lying about it? What is the template that you wind up using for that uniformity? Because even internally, I use three or four of those different things. And professional advice,
Starting point is 00:29:54 don't do that. Let's see. So in our development QA production environments, the infrastructure is all managed with Terraform. Each engineer has their own personal AWS account so that they can work on things there. Oh, that makes billing granularity super easy. Oh, yes. You can tell who's got EC2 and says that running up far too long. But for the most part, we'll use serverless framework in that regard to say for the engineer can just deploy into their local environment. Although we are working on ways to reuse the Terraform infrastructure and deploy that. But we have our own build and deployment pipeline that we built using CircleCI, and all of our Lambda functions are in Go. And so having to compile, say, 20 binaries in a service that gets kind of slow, one of our DevOps engineers actually came up with a way to use Lambda to build the Lambdas
Starting point is 00:30:46 so that we can build them all in a distributed parallel fashion during the build process. One thing that I do love about the whole serverless approach, and it is a neat part about Lambda, is no two people ever seem to do it quite the same way. You can tie things together in so many different and exciting ways that it's fun. It's almost like a modern version of playing with Lego. And I know that if Jeff Barr is listening, he just perked up at that. But I love the concept that you can take so many different ways to achieve similar outcomes. And it almost gives a bigger sense of creativity in how you approach problems. Has that been your experience? Oh, definitely. Not only the creativity, it's also the flexibility in how you solve it and the ability to adapt and evolve as services evolve or change or there's new ones are added. functions for customizing behavior of Cognito with the Cognito triggers is, to me, I think,
Starting point is 00:31:46 a perfect way to customize the service to do exactly what you need to do. I want to thank you so much for taking the time to speak with me today. It's always appreciated. If people want to hear more about what you have to say and how you view these things, or even possibly decide to work with you, where can they find you? I'm somewhat active on LinkedIn. LinkedIn is the best place to find me. Please go ahead and connect to me. Tell me you heard me on the podcast here. And yes, we are hiring. We have all within our technical organization from client, web and mobile engineers, data engineers, DevOps, API. We're always hiring.
Starting point is 00:32:28 And if we don't have something right now that fits your experience, let me know that you're interested and I'll put you on the list so that when we do have an opening, we'll reach out right away. And we'll, of course, include links to that in the show notes. Thank you so much for being so generous with your time. I appreciate it.
Starting point is 00:32:41 Thanks for having me on, Corey. It's nice talking to you. Michael Garsky, Director of Platform Engineering at Fender Musical Instruments. I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me that I'm almost certainly doing that chord incorrectly.

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.