Screaming in the Cloud - Replay - Keep on Rockin’ in the Server-Free World with Michael Garski
Episode Date: November 21, 2024On this Screaming in the Cloud Replay, we’re revisiting our conversation with Michael Garski, the director of software engineering at famed electrical guitar manufacturer, Fender. Prior to ...this position, he worked as a principal software architect at Viant, a principal software architect at MySpace, a manager of internet development at Countrywide Financial, and a manager of system architecture at Fandango, among other positions. He also had a four-year stint in the US Navy, working as an engineering laboratory technician. Join Corey and Michael as they talk about how artists are angels and Fender’s job is to give them wings, how Fender has diversified its offerings in recent years, how serverless is a mindset and how Fender approach serverless technology, how Fender’s traffic surged during the pandemic and how everything mostly scaled up without a hitch, the challenges of teaching students to play instruments over the internet, the vendor lock-in boogeyman, and more.Show Highlights(0:00) Introduction(0:42) Dragonfly sponsor read(1:25) How does Michael describe Fender’s work(2:08) Fender’s work to go serverless(4:13) The impact of COVID on Fender(6:19) Explaining Fender Play and how it works on the backend(9:44) Working with MediaConvert(11:30) Experiences with scaling and hitting AWS service limits(12:52) Why Michael prefers working on the customer side(15:33) The Duckbill Group sponsor read(16:15) Frustrations with gateways and third-party apps(19:03) Managing a massive influx of users during COVID(21:13) The vendor lock-in boogeyman(23:19) Cloud costs vs. saving time(24:49) Walking the fine line of criticism as a director(28:09) Enforcing consistency across services(31:52) Where you can find more from MichaelAbout Michael GarskiMichael Garski has worked in the Los Angeles tech industry for over 20 years, across companies including Fandango, Countrywide Home Loans, MySpace, Viant, and is currently at Fender Musical Instruments as the Director of Platform engineering were he leads the devops, data, and api engineering teams. His focus currently is on building the platform to support the consumer facing digital products for Fender. The most prominent application he supports is Fender Play, a web and mobile application that provides video-based instruction for guitar, bass, and ukulele for more than a quarter-million subscribers.LinksLinkedIn: https://www.linkedin.com/in/mgarski/Original Episodehttps://www.lastweekinaws.com/podcast/screaming-in-the-cloud/keep-on-rockin-in-the-server-free-world/SponsorsDragonfly: dragonflydb.ioThe Duckbill Group: duckbillgroup.com
Transcript
Discussion (0)
not only the creativity, it's also the flexibility in how you solve it and
the ability to adapt and evolve as the services evolve.
Welcome to Screaming in the Cloud. I'm Corey Quinn. We talk to a lot of people here on this
show who are deep in the weeds of SaaS companies or cloud vendors or cloud vendors
cosplaying as SaaS companies. Today, we're taking it a bit of a different direction.
My guest is Michael Garsky, Director of Platform Engineering at Fender Musical Instruments. They
make guitars, among many other things. Michael, thank you for joining me.
Oh, thanks for having me on, Corey.
It's an interesting time for the Redis ecosystem with licensing changes, the Valkey fork, and AWS Oh, thanks for having me on, Corey. a much lower cost. If you're running ElastiCache or Redis cloud workloads in AWS, Dragonfly
guarantees a minimum 25% reduction in your bill. Listeners of Screaming in the Cloud can snag a
$200 credit to try it by heading to dragonflydb.io and use the code SCREAMING after signing up to
take advantage of that $200 credit offer. Seriously, what are you waiting for? And my
thanks to Dragonfly for sponsoring this episode. One of the things that I really appreciate about what you do as a company is
I can at least presumably explain it to someone who is not super deep in technical weeds without
45 minutes of explainer first. The easy answer is, oh, Fender, you folks make guitars. These days,
no one just does one thing, I have to imagine.
How do you describe what the company does?
Oh, well, to quote Leo Fender, his view was that artists are angels and it's our job to
give them wings.
So in addition to actually making and developing guitars and amplifiers, we've branched off
into consumer-facing products to actually
teach people how to play those instruments. You folks have been relatively outspoken about the
various things you're doing at different AWS events. I mean, my approach to that tends to be
that if AWS is great at making bricks that you can use to build amazing things with, well, great,
can you draw a picture of the house that you can build with this?
No, we're going to have a customer come out and talk about that stuff instead.
You folks have been focusing on a lot of serverless work
and you've been very public about the fact
that you are almost entirely serverless driven
in terms of architecture, if I'm not mistaken.
That is true.
Tell me about that.
How did you get there and what brought it about?
So I work in the digital division at Fender. We started, let's see, we're coming up on five years
I've been there. So what we did was initially we started building services that could run within
a container or on an EC2 instance, but we started looking at Lambda functions. We had a need to ingest a product
catalog. So the IT team was able to drop us off a product catalog into an S3 bucket. And
easiest thing to do then was just trigger a Lambda function and then process that file.
And it just kind of snowballed in from there. I think a common problem when people hear
serverless is they think, oh, great, more discussions about Lambda functions. And
Lambda is almost getting something of a tarred reputation in some circles
because when we can build amazing things with it ourselves, we love it.
But when we ask AWS how to wind up integrating two services
or about a feature gap, their response is, oh, use a Lambda function for it.
It starts to feel like they're using it as spackle,
and the spackle has become load-bearing.
Do you view serverless as being purely function-driven or is it broader than that?
It's much broader than that. Serverless is a mindset where you're looking beyond just
Lambda functions to using a lot of third-party services so that you can actually focus on your
core business. We use Zora as a subscription provider for web-based subscriptions. We use
Algolia for full text search. We use a variety of other services so that we can just focus on
the core business. One thing that's been on everyone's mind somewhat recently has been the
idea of dramatic changes as far as user behavior goes. And in the more traditional environments
where you see things
like EC2 instances or on-premises data centers, back when the pandemic first hit and companies
that were very focused on a model of business that aligned directly with people behaving in
certain ways that they suddenly didn't, would see 80% drop-offs or more in their user traffic,
but their infrastructure spend just kept hanging out
exactly where it was in a straight line. So on some level, it feels like, yes, the whole point
of cloud is that it can be elastic, except no one builds it that way for a variety of reasons.
When COVID hit, what changed for your business?
Change for our business is we launched a program called Playthrough, where we did this about a year ago.
We started it.
We gave away three months of offender play for free.
It was a single-use code that a user would redeem and no credit card required.
And over a period of five days, we saw our traffic increase by more than 10 times.
And we had very little changes we needed to make.
Everything scaled up. We had no issue with, we used a lot of Lambda functions, DynamoDB,
everything just scaled up fine. The only point that became a bottleneck was our Elasticsearch
cluster. However, beefing up the nodes and adding a few more nodes resolved that issue immediately.
So I'm going to go out on a limb and postulate
that you folks saw increased pickup when the lockdowns hit, if for no other reason than,
well, I'm trapped at home and I'm tired of staring at the guitar on the wall. I may as well learn to
play it. I would guess I could be way off base on that. No, no, that's very true. Even since then,
even after that program has expired, of course, not everyone then converts and sticks around, but many, many did, many more than we thought did stick around. And as best I can tell, that is an app that works in the web, it works on mobile,
and it's a video-based instruction tool for guitar, at least, but some other instruments as well.
How did that come to be? Did that exist before COVID hit? Has that been something that's been
in the works for a while? Or was it, well, we're going to do a two-week sprint and build this thing from scratch?
No, we launched that this June. We're coming up on the fourth anniversary since it's been launched.
We launched this in summer of 2017. One of the problems I've always found is that it's
challenging to learn to do something that is as, I guess, physical and intricate, etc.,
as a planned instrument without having someone in the room looking at you
and smacking you with a stick
whenever you do things that are wrong.
Nope, that's a bad habit.
If you keep doing that, it's going to hurt you.
How do you approach that as a company
from a non-interactive perspective
of someone who's going to watch a video and do things
and maybe it'll work, maybe it won't,
particularly in light of things like,
well, the competition is YouTube, which, you know, I'm going to roll the dice and sometimes I'll see
a great tutorial. Sometimes I'll see one that I don't realize is teaching me terrible things.
And then it's going to recommend some baseless conspiracy theory because YouTube.
How do you differentiate that? What makes VendorPlay different?
So currently you're right. It's just a video-based instruction app. There's not any way to
provide feedback, direct feedback to students within the web and mobile applications.
However, we do have an online community, and our instructors, Fender Play instructors, do an office hours feature,
where they'll actually answer questions live and talk to students.
We are investigating and kind of doing some really research and some possibly being able to provide that type of feedback to students. We are investigating and kind of doing some early research and some possibly being
able to provide that type of feedback to users. But it's a very challenging problem just due to
the nature of you're playing an instrument that has multiple strings. You're trying to pick out
the chord that they're playing and the timing. But it's something we definitely need to add.
There's something to be said as well for the kind of care and attention that you folks wind up
putting into your media, where this is how you finger record.
And someone on the YouTube video will do it for two-tenths of a second, and they're filming it with a potato that isn't focused properly and pointing at the wrong part of the guitar.
You folks have a high bar for quality on this.
Is that done in-house?
Do you wind up just going through a bunch of random folks that you just wind up offering a bunch of gift cards to or free guitars to do this? How does the program work in the back end? So we have an
in-house curriculum team that puts together the lesson plans to really help people learn in small
bite-sized lessons so that it's not too overwhelming at once. And that curriculum then is shot and
filmed by an in-house video team that put that together. They upload the data into S3
for the final cut. Then that gets transcoded via media convert, and we serve it up via CloudFront.
It's rare to wind up talking to a company that is something of a household name
about something that they're doing and hear the AWS services that they're using not trend toward
a baseline mean, if I can be so bold.
Normally, you'll see some of the case studies like, oh, this is an online bank. What services
are they using? Oh, they're using EC2 and S3 and load balancing because did you miss the part where
it's a bank? They're not going to use these far future services, but due to regulatory risk,
among other things, in many cases. You're using Elemental Media Convert, which is one of those relatively high up the stack
offerings that isn't broadly known.
It's one of those services that is focused on specific use cases and specific industry
verticals in a way that a baseline primitive service isn't.
What does Media Convert do?
What it does is it takes the final edit of the video,
and we have several different presets so that it will put it into an HLS format
with different bit rates
so that the user's getting the best quality video
depending on their bandwidth.
When I looked into it in the early days
when it was first launching,
I found that it looked an awful lot like Elastic Transcoder,
which is a service that they've had for a while.
Only they changed up some of the capabilities.
It's obviously far more capable as a service.
But they also added something that felt like 15 different billing dimensions to it.
So what is this going to cost me?
Well, we're going to run it for a month and find out if we're still in business.
And it seemed like it was one of those very difficult to get started with and run experiments with service.
Now, obviously, services evolve over time. When you started looking into it,
was that experience roughly akin to what you found? Or am I completely and unfairly slandering
the product? We actually started out using Elastic Transcoder and then moved over to
Media Convert. I believe that was last year. We found it to be a little bit easier to use.
And the pricing overall in transcoding the videos for us is really a drop in the bucket as compared
to actually hosting them and serving them up via CloudFront. And when we switched over to media
convert, we adjusted our settings to lower the maximum bit rate for a given video. We found that
after a certain point, the quality to
the user just doesn't really improve, and yet we're paying to serve the larger video.
One statistic that I found was that in March of 2020, which I believe we're still in at this
point, it's that endless September model applied to March, you wound up seeing over an order of magnitude in traffic increase within
five days. And looking at that through a lens of traditional architecture, that means that nobody
sleeps a whole heck of a lot. Given that you're in on the serverless story, and you have been
since before that hit, what was that scaling experience like for you? Scaling experience was
completely seamless. We use a lot of Lambda,
DynamoDB, Kinesis, SNS to glue things together and no problems whatsoever. We just had to bump
up our Elasticsearch cluster a bit. That was really the only thing because we saw some latency
starting to rise on some of our APIs. Let me ask the uncomfortable question then, because whenever
I try to scale things up quickly in a cloud environment, what was your experience with smacking into various AWS service limits as the traffic grew?
Initially, we actually requested some service limits increased to make sure we weren't hitting the concurrent Lambda invocation limit.
And same thing with Cognito, making sure that we weren't going to hit any limit as far as sign-ins and things like that.
So we're able to just put in requests and they serve us around pretty quick turnaround time on
that as well. It really does seem like there's a strong benefit on the serverless space. But I had
to double check before we started recording that you do in fact work at Fender because you are a
staunch advocate for observability. And usually
when someone is that passionate about observability, you can guess that they work at an
observability slash monitoring company. It's akin to the idea of someone selling mattresses telling
you that mattresses are great and you should have four of them. You're on the customer side of that
and still very passionate about it. Where'd that come from? It came from my time years ago when I worked at MySpace,
if anyone can still remember that, working on the search systems there. And as the company started
winding down to laying people off and being one of the only people left working on those systems,
being able to know and understand them, you just have to.
So you have to continue to monitor and find ways to monitor it. And that really ingrained
how important instrumentation is and being able to really understand the health of your application
as it's running so that you can see like, yes, everything is good. And then like something
doesn't look right so that you know where to start looking and you can be alert of a problem.
So I tend to view the world in olden terms where monitoring was what we did.
And we use something like Nagios, which was the second worst option out there because everything else felt like it was tied for first.
I also take a somewhat regressive view that observability is to monitoring as DevOps is to being a systems administrator.
It's the same thing, but by using the more modern terminology, you can charge more for it.
I'm going to go out on a limb and guess that you take a somewhat contrarian view to that.
Yes, yes, I do. It's about really understanding how your application is running. It's not just
looking at, oh, how many HTTP 500s am I serving up per hour? If I hit a
threshold for the last hour, it's a lot more than that. It's really being able to really dig in and
see what the issue is or what's working really well. And to that end, we rely on two services
for this. We use Honeycomb and Epsilon. Honeycomb kind of acts as our top layer because it gives us the really,
really good high cardinality metrics where I can punch in a user ID and I can see all the API
traffic that this user has performed, as well as even just like when we launched the playthrough
in our traffic rows, the reason we discovered that our latency was dropping was due to a service
level objective being triggered in Honeycomb on latency.
And we were able to respond to that using that before customers really noticed anything at all.
Here at the Duckbill Group, one of the things we do with, you know, my day job is we help negotiate AWS contracts.
We just recently crossed $5 billion of contract value negotiated. It solves for fun
problems such as how do you know that your contract that you have with AWS is the best deal you can
get? How do you know you're not leaving money on the table? How do you know that you're not doing
what I do on this podcast and on Twitter constantly and sticking your foot in your mouth. To learn more, come chat at
duckbillgroup.com. Optionally, I will also do podcast voice when we talk about it. Again,
that's duckbillgroup.com. As an Epsilon customer myself, I'm always conflicted when I find myself
going into their service and using it to figure out what the heck's going on with my giant pile
of Lambda functions and API gateways and whatnot wired together, because the experience is
uniformly excellent. But I'm also frustrated in that it needs a third party to even begin to allude
to what's going on. It feels on some level like the vendor that is providing this service to me
should be reasonably effective at telling me what it's
doing and when it's breaking. I understand that how I wish the world is and how it actually is
are two radically different things, but does that ever strike you as well? Whether or not AWS should
be providing that type of level, that seems like more of a service that you can have competition
and other vendors that really specialize and get in the
weeds on it. I don't think AWS needs to provide every service you could possibly use for your
application. That's not something I'm too concerned about, and I don't really even think it's their
place, frankly. No, no, I understand. The problem I keep running into on some level, whatever I try
and diagnose it natively, is I look at CloudWatch and it's difficult to understand that is this, in my case, because again, I'm still early days with a lot of these things.
Is it the API gateway that's having the problem?
Is it the CloudFront distribution that is tied to that?
Is it the Lambda function?
Where is the handoff?
Trying to understand where in a complicated application the failure is occurring is a challenge. And let's be clear, most of that is a problem of my own making because I didn't have the
good sense to instrument this thing in a reliable, repeatable way when I built it.
It feels like everything is tied together with duct tape and bailing wire and spit and
a bit of luck.
As a counterpoint, the more companies I talk to, the more I realize that, no, no, this
is actually how most people feel when they look at things that are working. It's, you know, it's terrible. It's a trash fire, but it makes money, so we're going to roll with it.
And there's always on some level a sense of what we've built is very far from the platonic ideal
of what we should have built. Does that resonate with you? Or do you take a step back and look at
what you've achieved with a perspective of, this is awesome. More people should do it exactly like
this. And honestly, if it's that one, I'd love to take a look at what you've built. I think there's always room for us
to improve on what we're doing because we're constantly learning and evolving to improve both
even at such a low level of like, okay, how do we lay out the files in our service repository to
make the best organization to make sense all the way up to,
okay, how are we going to do tracing and what kind of information do we need to get from that
so that we can find problems when they occur? We're always looking to learn what others are doing
and talking to others in this space. No one will ever be 100% right. There's always room
for improvement everywhere. One thing that you folks have done that I think was really interesting and didn't get as much play as I think it really deserved
was that, especially in the early days of the pandemic, you wound up seeing that massive
increase due to giving out almost a million free three-month subscriptions to Playthrough.
Additionally, you also worked closely with
LAUSD, the Los Angeles Unified School District, to add Fender Play to their middle school music
programs curriculum to help supplement their remote learning programs. First, was that all
in the same timeframe? And two, what has it been like, I guess, working with a organization that is, I guess, on some
level, not particularly cloud-first, I would imagine.
When I lived in Los Angeles, I never got the sense that LAUSD was full-on serverless, full-on
board with cloud, full-on board with remote learning.
And then the pandemic, of course, exacerbates all of that.
Yeah, so those were really two different projects.
So the playthrough project, that started in March, and we started working with Los Angeles Unified School District last year during their summer school program. Started out with 1,500 students, and we put it together very quickly. Essentially, we used the same three-month codes that we used for the playthrough promotion so that we could set things up very quickly for students and gave out
through our nonprofit arm of Fender, the Fender Play Foundation, gave out 1,500 instruments to
these students to use during the summer school program. And that program became so successful,
we continued on with them in the fall and now in the current semester, and we will be again this
summer. I believe there's 7,000 students in the program now. And working with their IT team has actually been quite nice in dealing with
partners. You wouldn't think much of like, oh, it's a school district. What do they have? But
as far as like just ease of working with them, we actually hooked into their SAML provider,
Incognito, so that LAUSD students could authenticate
when they come in through the remote learning systems. And they were great to work with and
very helpful and cooperative. One of the arguments that you'll see that comes up against serverless
from time to time is that you are now indelibly linked to your provider. You can't take what
you've built with all of these services and
just move it over to Azure or GCP on a moment's whim. Now, in practice, people who tend to build
for that just build everything on top of EC2 and very little else, and then run it entirely in AWS
and never move it to any of those other places. But was there friction with making that, I guess,
architectural commitment to a single vendor?
Oh, you're bringing up the vendor lock-in boogeyman.
Oh, I absolutely am.
Most people who bring that, but I bring it up as a straw man so you can attack it.
Most people who bring up the vendor lock-in boogeyman,
oh, you have to go multi-cloud, are either trying to sell you something that is required
if you want to go multi-cloud, or they're a cloud provider themselves who know that
if you go all in on one provider,
it will certainly not be theirs.
I think if you properly architect your applications
with separations of concerns
that you could move to say,
okay, say Lambda wasn't working out for us anymore
and we needed to take our applications
and okay, we're going to put them into a container,
but we're going to stay in AWS.
Our applications are set up in such a way
that Lambda is basically a deployment pattern.
We could easily convert those individual function handlers
into route handlers with a minimal effort
because the business logic
and then the underlying data storage are separated.
So it would be feasible for us if we wanted to, say,
move to Azure and use Azure
functions and whatever comparable service they have to DynamoDB. I'm not too familiar with a lot
of their offerings, but that would certainly be possible to do it with obviously some effort.
And really, at the end of the day, the resources you have working on the applications are going to cost you much more than any sort of like software licensing or specific savings you're going to get from a cloud vendor.
So might as well go ahead and just use those services that they're providing so that you can just focus on the business. has almost universally been that looking at an awful lot of companies and their AWS bills,
it is a challenge to find an environment where the resources in the environment cost more than
the people who are operating them. In the context of business, AWS bills seem giant and enormous
right up until you look at payroll. And then it's, oh, okay. What's counterintuitive for folks who are
learning this, and I fall prey to it myself, is when I'm playing around as a hobbyist trying to
build something, I value my time as free because I'm learning as this goes. And then in that
context, especially when I was starting out as a student, it was, oh, great. So this winds up
costing me $7 a month. Ooh, that's a lot of money. That's my ramen budget. So I'm instead going to wind up spending eight hours avoiding it charging me anything. It's the exact opposite
from the direction you want staff that you're paying to work on these things to go in.
How do you approach the idea of increasing the cloud cost if it will save time for your team?
It's a balance between, okay, do we need to build this ourselves? And then
not only build it, you have to operate it and maintain it. Or what is the cost of getting this
third-party service? And that's really what it comes down to in all of them. And do we actually
want to spend time working on this piece of infrastructure that these other people are
specializing in and do so well? I've got better things I can have people doing than that.
Speaking of people, one thing that you talk about, as you self-describe, is that you wind up not
writing a whole lot of code anymore, but you're something of a stickler for observability and
enforcing consistency between services. So you'll periodically do things like submit a PR
to tweak a log message to put your mind at ease
was one example that you gave.
Given that you're a director,
which is generally manager of manager style approaches,
how do you avoid having those PRs
come across to your team as either micromanagement
or a condemnation of what they've built?
Because I get it.
When I see something
that's easy and small to tweak, I want to go ahead and get it fixed immediately. I don't want to go
back and forth and play those games. I just want it done. But I'm also always weighing that against,
I don't want to have people think that I'm judging them somehow for something I'm very much not.
That's a very good point. The larger technical decisions on how
things are laid out, I generally just try to, I don't insert myself into. I let the team go ahead
and make those decisions and lead that direction and let them take the charge on that. And I kind
of take the approach of looking at it as more of a guiding and mentoring and teaching to kind of
really hone and instill that discipline in really
being able to understand what the applications are doing. And as our team is growing, I have less and
less time to even do those things, but I can go through the systems and go, hey, how come we're
not tracing this call to the reCAPTCHA servers? Let's add that in there. And I'll just, at this
point now, I mainly just write JIRA tickets to have someone else actually do the work. The more I do this, the more I realize that as complicated as
the technology is, the people are in many ways far more complicated and, let's be fair here,
non-deterministic. Things that work super well on one person one month could work entirely
differently a following month or even with the same person or between teams. It's a constant balancing act on
some level. And giving people a sense of psychological safety has always been the
biggest challenge. The thing that surprised me about management back when I was running ops
teams was the more, I guess, responsibility you accrue as you rise from individual contributor
into the management, rise is sort of a wrong term, it's an orthogonal transition, is that you spend a lot more time on the people problems and your
ability to directly control or affect change diminishes because you have to do everything
via influence. You get a lot more responsibility with a lot less direct power over the outcome in
some ways. Does that align with how you see it? Or do I have very strange approaches on management?
Which may be true and why I got out of it as fast as I could.
No, that is a good point.
Because you are having to influence and guide
and sort of more take a higher level view
as opposed to really getting into the weeds of like,
okay, what methods are we going to put on this interface?
How are we going to see architect the internals of an application? Those are details I just really don't have time for anymore.
But larger things is to making sure that we're, okay, it's like, you know, what's the performance
of this? And overall, is this something that can be adapted? And as the business needs change,
and as we change, then as we learn, what can we do to modify it? And just kind of more just things like guiding and mentoring and really taking a higher level view of that.
I'm going to selfishly ask about something that I struggle with myself that goes a bit more into the technical area.
But you talk about enforcing consistency across all of your different services.
What does that mean?
Similar coding styles?
Similar instrumentation?
Because I look at the things I've built and the microservices that power my internal nonsense,
and each one of those is very different than all the rest. So whatever your version of consistency
is, I know I'm not doing it, but how do you view it? So there's really two types of consistency.
The one I really refer to the most is in observability. So that if you've got a thousand
Lambda functions out there and each one is logging observability, so that if you've got a thousand lambda functions
out there and each one is logging things slightly differently, that's just a pain to deal with.
Realistically, dealing with a thousand unicorns is a real pain. So through that observability,
at least in lambda, we use an internally developed middleware to make sure that
the logging is consistent and it's easy enough to use. And then other consistency, like just within projects of how we lay things out,
that's something that's consistently evolving.
What's the folder structure and how we organize the code?
And we've kind of been evolving that over the last three years.
And, you know, within about the last six months,
we've come up with like a really good pattern and a template for the future.
And it's not much different from what we started out with, but it's a little bit easier to really do
comprehend as a new engineer coming in. It makes more sense. I have to ask, and I understand if
you don't want to give a particular endorsement in any direction, but do you go through serverless
frameworks, SAM CLI, the CDK, using the console
and then lying about it? What is the template that you wind up using for that uniformity?
Because even internally, I use three or four of those different things. And professional advice,
don't do that. Let's see. So in our development QA production environments,
the infrastructure is all managed with Terraform. Each engineer has their own personal AWS account so that they can work on things there. Oh, that makes billing granularity
super easy. Oh, yes. You can tell who's got EC2 and says that running up far too long.
But for the most part, we'll use serverless framework in that regard to say for the engineer
can just deploy into their local environment. Although we are working on ways to reuse the Terraform infrastructure and deploy that.
But we have our own build and deployment pipeline that we built using CircleCI, and all of our
Lambda functions are in Go. And so having to compile, say, 20 binaries in a service that gets
kind of slow, one of our DevOps engineers actually came up with a way to use Lambda to build the Lambdas
so that we can build them all in a distributed parallel fashion during the build process.
One thing that I do love about the whole serverless approach, and it is a neat part
about Lambda, is no two people ever seem to do it quite the same way. You can tie things together
in so many different and exciting ways that it's fun. It's almost like a modern version of playing with Lego.
And I know that if Jeff Barr is listening, he just perked up at that. But I love the concept
that you can take so many different ways to achieve similar outcomes. And it almost gives
a bigger sense of creativity in how you approach problems. Has that been your experience?
Oh, definitely. Not only the creativity, it's also the flexibility in how you solve it and the ability to adapt and evolve as services evolve or change or there's new ones are added. functions for customizing behavior of Cognito with the Cognito triggers is, to me, I think,
a perfect way to customize the service to do exactly what you need to do.
I want to thank you so much for taking the time to speak with me today. It's always appreciated.
If people want to hear more about what you have to say and how you view these things,
or even possibly decide to work with you, where can they find you?
I'm somewhat active on LinkedIn. LinkedIn is the best place to find me. Please go ahead and
connect to me. Tell me you heard me on the podcast here. And yes, we are hiring. We have
all within our technical organization from client, web and mobile engineers, data engineers, DevOps, API.
We're always hiring.
And if we don't have something right now that fits your experience,
let me know that you're interested
and I'll put you on the list
so that when we do have an opening,
we'll reach out right away.
And we'll, of course, include links to that in the show notes.
Thank you so much for being so generous with your time.
I appreciate it.
Thanks for having me on, Corey.
It's nice talking to you.
Michael Garsky, Director of Platform Engineering at Fender Musical Instruments.
I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this
podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've
hated this podcast, please leave a five-star review on your podcast platform of choice,
along with a comment telling me that I'm almost certainly doing that chord incorrectly.