Screaming in the Cloud - Learnings From A Lifelong Career in Open-Source with Amir Szekely

Episode Date: November 7, 2023

Amir Szekely, Owner at CloudSnorkel, joins Corey on Screaming in the Cloud to discuss how he got his start in the early days of cloud and his solo project, CloudSnorkel. Throughout this conversation, Corey and Amir discuss the importance of being pragmatic when moving to the cloud, and the different approaches they see in developers from the early days of cloud to now. Amir shares what motivates him to develop open-source projects, and why he finds fulfillment in fixing bugs and operating CloudSnorkel as a one-man show. About AmirAmir Szekely is a cloud consultant specializing in deployment automation, AWS CDK, CloudFormation, and CI/CD. His background includes security, virtualization, and Windows development. Amir enjoys creating open-source projects like cdk-github-runners, cdk-turbo-layers, and NSIS.Links Referenced:CloudSnorkel: https://cloudsnorkel.com/lasttootinaws.com: https://lasttootinaws.comcamelcamelcamel.com: https://camelcamelcamel.comgithub.com/cloudsnorkel: https://github.com/cloudsnorkelPersonal website: https://kichik.com

Hello, and welcome to Screaming in the Cloud, with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. Welcome to Screaming in the Cloud. I'm Corey Quinn, and this is an episode that I have been angling for for longer than you might imagine.

My guest today is Amir Sekheli, who's the owner at CloudSnorkel. Amir, thank you for joining me. Thanks for having me, Corey. I love being here. So I've been using one of your open source projects for an embarrassingly long amount of time. And for the longest time, I make the critical mistake of referring to the project itself as CloudSnorkel, because that's the word that shows up in the GitHub project, that I can actually see that jumps out at me. The actual name of the project within your org

is CDK GitHub Runners, if I'm not mistaken. That's real original, right? Exactly. It's like, oh, good, I'll just mention that and suddenly everyone will know what I'm talking about. But ignoring the problems of naming things well, which is a pain that everyone at AWS or who uses it knows far too well, the product is basically magic. Before I wind up basically embarrassing myself by doing a poor job of explaining what it is, how do you think about it? Well, I mean, it's a pretty simple

project, which I think what makes it great as well, it creates GitHub Runners with CDK. That's about it. It's in the name and it just does that. And I really try to make it as simple as possible and kind of learn from other projects that I've seen that are similar and basically learn from my pain points in them. I think the reason I started is because I actually deployed CDK Runners,

sorry, GitHub Runners for one company and I ended up using the Kubernetes one, right? So GitHub themselves, they have two projects they recommend and nudge nudge GitHub, please recommend my project one day as well. They have the Kubernetes controller and they have the Terraform deployer and the specific client that I worked for, they wanted to use Kubernetes. And I tried to deploy it and, Corey, I swear,

I worked three days to deploy the thing, which was crazy to me. And every single step of the way, I had to go and read some documentation, figure out what I did wrong, and apparently the order the documentation was was incorrect. And I had to, to yeah I even opened

tickets and they you know they were actually like it's open source project please contribute and fix the documentation for us at that point I said nah let me create something better with CDK and I decided just to have the simplest setup possible so right, what you end up doing in these projects, you have to set up either secrets or SSM parameters, and you have to prepare the ground, and you have to get your GitHub token and all those things. And that's just annoying. So I decided to create a... So much busy work. So yeah, so much busy work and so much boilerplate and so much figuring out the right way and the right order and just annoying. So I decided to create a setup page.

I thought, what if you can actually install it just like you install any app on GitHub, which is the way it's supposed to be, right? So when you install CDK GitHub Runners, CloudSnorkel, you get a HTML page and you just click a few buttons and you tell it where to install it and it just installs it for you and it sets the secrets and everything and if you want to change a secret you don't have to redeploy you can just change a secret right you have to roll the token over or whatever so it's much much easier to install I feel like I discovered this project through one of the more surreal approaches. I had caused to revisit it a few weeks ago when I was redoing my talk for the CDK Community Day, which has since happened and people liked the talk.

And I mentioned what CloudSnorkel had been doing and how I was using the runners accordingly. So that was what incidentally caused me to pop back up with, hey, I've got some issues here. But we'll get to that. Because once upon a time, I built a Twitter client for creating threads, because shitposting is my love language, and I would sit and create Twitter threads in the middle of live keynote talks. Threading in the native client was always terrible, and I wanted to build something that would help me do that. So I did, and it was up for a while. It's not anymore because I'm not paying $42,000 a month in API costs to some jackass. But it still exists in a form of last toot in AWS.com if you want to create threads on Mastodon.

But after I put this out, some people complained that it was slow, to which my response was, what do you mean? It's super fast for me in San Francisco talking to it hosted in Oregon. But on every round trip from halfway around the world, it became a problem. So I got it into my head that since this thing was fully stateless, other than a Lambda function being fronted by an API gateway, that I should deploy it to every region. It didn't quite fit into a Cloudflare worker

or into one of the edge Lambda functions that AWS has given up on. But okay, how do I deploy something to every region? And the answer is, is with great difficulty. Because it's clear that no one was ever imagining with all those regions that anyone would use all of them. It's imagined that most customers use two or three, but customers are different, so which two or three is going to be widely varied.

So anything halfway sensible about doing deployments like this didn't work out. Again, because this thing was also a Lambda function and an API gateway, it was dirt cheap. So I didn't really want to start spending stupid amounts of money doing deployment infrastructure and the rest. So, okay, how do I do this? Well, GitHub Actions is awesome. It is basically what all of AWS's code offerings wish that they were. Code build is sad, and this was kind of great. The problem is, is once you're out of the free tier, and if you're a bad developer where you do a deploy on every iteration, suddenly it starts costing for what I was doing in every region,

something like a quarter per deploy, which adds up when you're really, really bad at programming. So their matrix jobs are awesome, but I want to do some self-hosted runners. How do I do that? And I want to keep it cheap, so how do I do a self-hosted runner inside of a lambda function, which led me directly to you? And it was nothing short of astonishing. This was a few

years ago. I seem to recall that it used to be a bit less well-architected in terms of its elegance. Did it always use step functions, for example, to wind up orchestrating these things? Yeah, so I do remember that day. We met pretty much basically as a joke

because the Lambda Runner was a joke that I did and I posted it on Twitter and I was half proud of my joke that starts in 10 seconds, right? But yeah, no, I think it always used step functions. I've been kind of in love with Step Functions for the past two years. They just, they're nice.

Oh, they're magic. And AWS is so bad at telling their story. Both of those things are true. Yeah, and the API is not amazing. But like when you get it working and you have to spend some time to get it working, it's really nice because then you have nothing to manage ever.

And they can call APIs directly now so you don't have to even create lambdas. It's pretty cool. And what I love is you wind up deploying this thing to whatever account you want it to live within. What is it, the OIDC? I always get those letters in the wrong direction. OIDC, I think is correct. I think it's OIDC, yeah. Yeah, and it winds up doing this through a secure method as opposed to just, okay, now anyone with access to the project can deploy into your account, which is not ideal. And it just works. It spins up a whole bunch of these Lambda functions

that are using a Docker image as the deployment environment. And yeah, all right, if effectively my CDK deploy, which is what it's doing inside of this thing, doesn't complete within 15 minutes, then it's not going to, and the thing's going to break out. We've solved the halting problem. After 15 minutes, the loop will terminate

the end. But that's never been a problem, even with getting ACM certificates spun up. It completes well within that time limit. And its cost to me is effectively nothing, with one key exception, that you made the choice to use Secrets Manager to wind up storing a lot of the things it cares about instead of Parameter Store. So I think you wind up costing me, I think there's two of those different secrets, so that's 80 cents a month, which I will be demanding in blood one of these days if I ever catch you at reInvent. I'll buy you a beer. There we go. That'll count. That'll buy like several months of that. That works. Well, reInvent, no. The beers there are like $18, so that'll cover me for years. We're set. We'll split it.

Exactly. Problem solved. But I like the elegance of it. I like how clever it is. And I want to be very clear, though, it's not just for shitposting because it's very configurable where, yes, you can use Lambda functions. You can use spot instances. You can use code build containers. You can use Fargate containers. you can use EC2 instances. And it just automatically orchestrates and adds these self-hosted runners to your account, and every build gets a pristine environment as a result. That is no small thing. I love making things configurable. People really appreciate it, I feel. And it gives people kind of a sense of power, but as long as you make that configuration simple enough right or at least the defaults good defaults right then even with that power people still don't shoot themselves in the foot then it still works really well by the way we just added ecs recently which people

really were asking for because it gives you the kind of easy option to have the runner well not the runner but at least the runner infrastructure staying up, right? So you can have auto-scaling group backing the ECS, and then the runner can start up a lot faster. It was actually very important to other people because Lambda, as fast as it is, it's limited. And Fargate, for whatever reason, still to this day takes a minute to start up. Yeah, what's wild to me about this is start to finish, I hit a deploy to the main branch and it sparks the thing up, runs the deploy. Deploy itself takes a little over two minutes. And every time I do this, within three minutes of me pushing the commit,

the deploy is done globally. It is lightning fast. And I know it's easy to lose yourself in the idea of this being a giant shitpost where, oh, who's going to do deployment jobs in Lambda functions? Well, kind of a lot of us for a variety of reasons, some of which might be better than others. In my case, it was just because I was cheap. But the massive parallelization ability to do 20 simultaneous deploys in a matrix configuration that doesn't wind up smacking into rate limits everywhere,

that was kind of great. Yeah, we have seen people use Lambda a lot. It's mostly for, yeah, like you said, small jobs. And the environment that they gave you is kind of limited, so you can actually install packages, right? There is no sudo, and you can't actually install anything unless it's in your temp directory. But still, just being able to run a lot of little jobs,

it's really great, yeah. And you can also make sure that there's a Docker image ready to go with the stuff that you need just by configuring how the build works in the CDK. I will admit I did have a couple of bug reports for you. One was kind of useful where it was not at all clear how to do this on top of a Graviton-based Lambda function. Because yeah, that was back when not everything

really supported ARM architecture super well. And a couple of other times when the documentation was fairly ambiguous from my perspective, where it wasn't at all clear, what was I doing? I spent four hours trying to beat my way through it. I give up, filed an issue, went to get a cup of coffee, came back, and the answer was sitting there waiting for me because I'm not convinced you sleep. Well, I am a vampire. My last name is from the Transylvania area. Excellent. Excellent. By the way, not the first time people tell me that, but anyway.

There's something to be said for any immediate responsiveness. Because one of the reasons I'm always so loathe to go and do a support ticket anywhere is this is going to take weeks. And then someone's going to come back with a, I don't get it, and try and like read the support portfolio to you. No, you went right in to, yeah, it's this. Fix it and your problem goes away. And sure enough, it did. The escalation process that some companies put you through is very frustrating. I mean, lucky for you, Cloud Snorkel is a one-man show and this man loves solving bugs.

Yeah. Do you know of anyone using it for anything that isn't ridiculous and trivial like what I'm using it for? Yeah, I have to think whether or not I can. I mean, OK, we have a bunch of dedicated users, right? The GitHub repo that keep posting bugs and keep posting even patches, right? So you can tell that they're using it. I even have one sponsor, one recurring sponsor on github that uses it it's always nice when people thank you via money yeah yeah it is very validating i think he's using it but i also don't think i can actually say it because i got it from

the github yeah it's always fun that's the beautiful part about open source. You don't know who's using this. You see what other things people are working on. You never know. Is this someone's side project? Is this a Skunkworks thing? Or God forbid, is this inside of every car going forward and no one bothered to tell me about that? That is the magic and mystery of open source.

And you've been doing open source for longer than I have. And I thought I was old. You were originally named in some of the Winamp credits, for God's sake, that media player that really whipped the llama's ass. Oh, yeah. I started real early. I started about when I was 15, I think. I started off with Scal or something or even Perl. And then I decided I have to learn C and I have to learn Windows API. I don't know what possessed me to do that when 32 API is unique. But once I created those applications for myself, right, I think there was, oh my God, dude, do you know this? What is it called? Sherlock in Mac OS, right? And these days

for power toys, there is the equivalent of it called, I don't know, whatever that power bar. That's exactly, that was that. That's a project I created as a kid. I wanted something where I can go to the run menu of Windows when you hit win key R and you can just type something and it will start it up, right? I didn't want to go to the start menu and browse and click things. I want to do everything with the keyboard. So I created something called Blaze Run, which helped you really easily create shortcuts that went into your path,

right? The Windows path. So you can really easily start them from WinKeyR. I don't think that anyone besides me used it, but anyway, that thing needed an installer, right? Because Windows, you got to install things. So I ended up- Yeah, these days on macOS, I use Alfred for that, which is kind of long in the tooth, but there's LaunchBar and a bunch of other stuff for it. What I love is the fact that I can double tap the command key and that just pops up whatever I need it to and tell the computer what to do. It feels like there's an AI play in there somewhere.

People can figure out how to spend 10 minutes on building AI that does something other than lets them fire their customer service staff. Oh my God, please don't fire customer service staff. AI is so bad. Yeah. When I reach out to talk to a human, I really need a human. Yes. Like I'm not calling you because I want to talk to a robot. I know there's a website. Leave me alone. Just give me a person. Yeah. You already failed to solve my problem on your website. It's person. Exactly. Oh my God.

Anyway, so I had to create a, an installer, right? And I found, it was called Ensys. So it was a Nullsoft SuperPIMP installation system. Or in the future, when Justin, the guy who created WinUp and Ensys, tried to tone down a little bit, Nullsoft scriptable installation system. And SuperPIMP is, this is such useless history for you, right? But SuperPIMP is the next generation of PIMP, which is plug-in mini-packager. I remember so many of these days, no one would ever name any project like that, just because it's so off-putting to people with sensibilities. But back then, that was half the stuff that came out.

Oh, you don't like how this thing I built for free in the wee hours when I wasn't working at my fast food job wound up, you know, like how I chose to name it. Well, that's okay. Don't use it. Go build your own. Oh, what how I chose to name it. Well, that's okay. Don't use it. Go build your own. Oh, what? You're using it anyway. That's what I thought.

Yeah. The source code was filled with profanity too. And I didn't care. I really did not care. But some people would complain and open bug reports and patches. And my policy was kind of like, okay, if you're complaining, I'm just going to ignore you. If you're opening a patch, fine. I'm going to accept to accept it sure you guys want to create something that's sensible for everybody sure i mean it's just source code you know whatever so yeah i started working on that

i used it for myself and i joined the forums and this kind of answers also your question of why i respond to things so fast just because of the fun i did the same same when I was 15, right? I started going on the forums. You remember forums? You remember that? Oh yeah, back before they all became terrible and monetized. So, you know, people were using SS2 and they had requests, right?

They wanted, back in the day, what was it? There was only support for 16-bit colors for the icon. So they wanted 32-bit colors and big colors, 32, big icons, sorry. 32 pixels by 32 pixels. Remember 32 pixels? Oh yes. Not well and not happily, but I remember it. Yeah. So I started just, you know, giving people, working on that open source and creating a fork. It wasn't even

called fork back then, but I created like a little fork of myself and I started adding all those features and people were really happy and kind of created like this happy cycle for myself when people were happy i was happy coding and then people were happy by what i was coding and then they were asking for more and they were getting happier the more i responded so it was kind of like a serotonin cycle that made me happy and made everybody happy so it's like a win, win, win, win, win. And that's how I started with open source. And eventually, Ensis, again, that installation system, it got so big, like my fork got so big. And Justin, the guy who works on Winamp and Ensis,

he had other things to deal with. You know, there's a whole history there with AOL. I'm sure you've heard all the funny stories. Oh, yes. In fact, one thing that, you want to talk about weird collisions of things crossing. One of the things I picked up from your bio when you finally got tired of telling me no

and agreed to be on this show was that you're also one of the team who works on camelcamelcamel.com. And I keep forgetting that that's one of those things that most people have no idea exists, but it's very simple. All it does is it tracks Amazon products that you tell it to and alerts you when there's a price drop on the thing that

you're looking at. It's something that is useful. I try and use it for things of substance or hobbies because I feel really pathetic when I get excited emails about a price drop in toilet paper. But you know, it's very handy just to keep an idea for price history where, okay, am I actually being ripped off? Oh, they claim it's their big Amazon deals day. And this is 40% off. Let's see what camel, camel, camel has to say. Oh, surprise. They just jacked the price right beforehand and now not 40% off genius. I love that. It always felt like something that was going to be blown off the radar by Amazon being displeased. But I discovered you folks in 2010. And here you are now,

13 years later, still here. I will say the website looks a lot better now. Yeah, it's a recent change. I actually joined Camel maybe two or three years ago. I wasn't there from the beginning, but I knew the guy who created it, again, as you were saying, from the Winamp days, right? So we were both working in the free... Well, it wasn't Freenode. It was not Freenode. It was a separate IRC server that, again, Justin created for himself. It was called LandElite. I never encountered that one. Yeah, no, it was pretty private. The only people that cared about Winamp and ANSYS ended up joining there. But it was a lot of fun. I met a lot of friends there.

And yeah, I met Daniel Green there as well. And he's the guy that created, along with some other people in there that I think want to remain anonymous, so I'm not going to mention. But they also were on the Camel project. And yeah, I was kind of doing my poor version of shitposting on Twitter about AWS, kind of starting to get some traction and maybe some clients and talk about AWS so people can approach me.

And Daniel approached me out of the blue and he was like, do you just post about AWS on Twitter or do you also do some AWS work? And I was like, I do some AWS work. Yes, as do all of us. It's one of those, well, crap, we're getting called out now. Do you actually know how any of this stuff works? Much to my everlasting shame, yes. Why are you asking? Oh my God, no, I cannot fix your printer. Leave me alone. I don't want to fix your Lambdas. No, but I do actually want to fix your Lambdas. He approached me and he asked if I can help him move Camel, Camel, Camel from their data center to AWS.

So that was a nice big project. So we moved actually all of Camel, Camel, Camel into AWS. And this is how I found myself not only in the Winamp credits, but also in the Camel, Camel, Camel credits page, which has a great picture of me riding a camel. Excellent. But one of the things I've always found has been that when you take an application that has been pre-existing for a while in a data center and then move it into the cloud, you suddenly have to care about things that no one sensible pays any attention to in the land of a data center. Because it's like, what do I care about how much data passes between my application server

and the database? Wait, what do you mean that in this configuration, that's a chargeable data transfer? Oh, dear Lord. And things that you never had to think about optimizing are suddenly things you're very much optimizing. Because let's face it, when it comes to putting things in racks and then running servers, you aren't auto-scaling those things. So everything tends to be running over provision for very good reasons. It's an interesting education. Anything you picked out from that process that you think would be useful for folks to bear in mind if they're staring down the barrel the same thing? Yeah, for sure.

I think in general, right, not just here, but in general, you always want to be pragmatic, right? You don't want to take steps that are huge, right? So the thing we did was not necessarily rewrite everything and change everything to AWS and move everything to Lambda and move everything to Docker. We basically, we did a mini lift and shift, but not exactly lift and shift. We didn't take it as is. We moved to RDS, we moved to ElastiCache, right?

We obviously made use of security groups and Session Connect, and we dropped SSH and we improved the security a lot and we locked everything down, all the permissions and all that kind of stuff, right? But like you said, there's stuff that you start having to pay attention to. In our case, it was less the data transfer, because we have a pretty good CDN. It was more of IOPS. And IOPS specifically for our database. We had a huge database with about one terabyte of data. And a lot of it is that price history that you see, right?

So all those nice little graphs that we create in, we call them charts, that we create in Camo, Camo, Camo of the price history. There's a lot of data behind that. And what we always wanted to do is actually remove that from MySQL, which has been kind of struggling with it even before the move to AWS, but after the move to AWS, but after the

move to AWS, where everything was no longer over-provisioned and we couldn't just buy a few more NVMEs on Amazon for a higher bucks when they were on sale. But we had to pay Amazon. And you know when they're on sale, that's the best part. And we know we get good prices on NVME. But yeah, on AWS, you have to pay for IO1 or something. And that adds up real quick, as you were saying. So part of that move was also to move to something that was a little better for that data structure. And we actually moved just that data, the price history, the price points, from MySQL to DynamoDB, which was a pretty nice little project. Actually, I wrote about it in my blog. There is kind of lessons learned from moving one terabyte

from MySQL to DynamoDB. And I think the biggest lesson was about hidden price of storage in DynamoDB. But before that, I want to talk about what you asked, which was the way that other people should make that move, right? So again, be pragmatic, right? If you Google, how do I move stuff from DynamoDB to MySQL, everybody's always talking about their cool project using Lambda

and how you throttle Lambda and how you get throttle from DynamoDB and how you set it up with an SQS and this and that. You don't need all that. Just fire up an EC2 instance, write some quick code to do it. I used, I think it was Go with some limiter code from Uber, and that was it. And you don't need all those lambdas

and the SQS and the complication. That thing was a one-time thing anyway, so it doesn't need to be super, super-duper serverless, you know? That is almost always the way that it tends to play out that you encounter these weird little things along the way and you see so many things that are tied to this is how architecture absolutely must be done and oh if you you're not a real serverless person if you don't have everything running in lambda and the rest there are times where you spit up an ec2 box write some relatively inefficient code in 10 minutes to just do the thing, and then turn it

off when you're done. Problem solved. But there's such an aversion to that. It's nice to encounter people who are pragmatists more than they are zealots. I mostly learned that lesson, and both Daniel Green and me learned that lesson from the Winamp days because we both written plugins for Winamp and we've been around that area and you can if you took one of those non-pregnantist people right and you had them review the Winamp code right now or even before they would have a million things to say that code was and ANSYS too by the way way. And it was so optimized. It was so not necessarily readable, right? But it worked and it worked amazing. And Justin would, if you think I respond quickly, right?

Justin Frankel, the guy who wrote Winamp, he would release versions of ANSYS and of Winamp, like four versions a day, right? That was before you had CICD systems and GitHub and stuff. That was, which was CVS. You remember CVS? Oh, I've done multiple CVS migrations. One to Git and a couple to Subversion.

So yeah, Subversion. Yep. Done them all. CVS to Subversion to Git. Yep, yep. That was fun. And these days everyone's using Git because we now have a monoculture.

Yeah, yeah. I mean, but Git is nicer than Subversion, for me at least. I've had more fun with it. Talk about damning with faint praise. Faint? Yeah, anything's better than Subversion. Let's be honest here. Realistically, copying a bunch of files and directories to a.back

folder is better than Subversion. Well. At least these days. But back then, it was great. Yeah, I mean, the only thing you had, right? Anyway, achieving great things with not necessarily the right tools, but just pure power of will, that's what I took from the Winamp days. Just the entire world used Winamp. And by the way, the Ansys project that I was working on, right, I always used to joke that every computer in the world ran my code, every Windows computer in the world ran my code just because so many different companies use Ensis

and none of them cared that the code was not very readable, to put it mildly. So many companies founder on those shores where they lose sight of the fact that I can point to basically no companies that died because their code was terrible. An awful lot that died with great looking code, but they didn't nail the business problem. I would be lying if I said that I nailed exactly the business problem on Ansys because

most of the time I would spend there on actually shrinking the stub that was appended to your installer data. So there's a little stub that came, the executable basically that came before your data that was extracted. to your installer data. So there's a little stub that came, the executable basically that came before your data that was extracted. I spent, I want to say, years of my life

just shrinking it down by bytes, by little bytes, just so it stays under 34, 35 kilobytes. It was kind of a, it was a challenge and something that people appreciated, but not necessarily the thing people appreciated the most. I think the features... Well, now we have to do the same thing to make sure something fits into a Lambda deployment package.

The scale changes, the problem changes, but somehow everything sort of rhymes with the history. Oh, yeah. I hope you don't have to disassemble code to do that, though, because that's... I mean, it was fun. It was just a lot. I have to ask, how much work went into building your CDK GitHub runners

as far as getting it to a point of just working out the door? Because I look at that, and it feels like the early versions, yeah, there wasn't a whole bunch of code tied to it, but geez, the iterative, how exactly does this ridiculous step functions API work or whatnot? It feels like I'm looking at weeks of frustration. At least it would have been for me. Yeah. Yeah.

I mean, it wasn't like a day or two. It was definitely, but it was not years either. I've been working on a bank about a year now. Don't quote me on that, but I've put a lot of time into it. So,

you know, like you said, the skeleton code is pretty simple. It's a step function, which as we said, takes a long time to get right. Step functions, they are really nice, but their definition language is not very straightforward.

But beyond that, right, once that part worked, it worked. Then came all the bug reports and all the little corner cases, right? Hell, it's other people's use cases, always is. But that's honestly better than a lot of folks wind up experiencing, where they'll put an open source project up and no one ever knows. So getting users is often one of the biggest barriers to a lot of this stuff. I found countless hidden gems lurking around on GitHub with a very particular search for

something that no one had ever looked at before, as best I can tell. Open source is a tricky thing. There needs to be marketing brought into it. There needs to be storytelling around it. It has to actually, dare I say, solve a problem someone has. I mean, I have many open source projects like that, that I find super useful. I created for myself, but no one knows. I think CDK GitHub Runner is, I'm pretty sure people know about it only because you talked about it on Screaming in the Cloud or your newsletter. And by the way, thank you for telling me that you talked about it last week in the conference, because now I know why there was

a spike all of a sudden. People Googled it. Yeah, I put links to it as well. But it's the, yeah, I use this a lot and it's great. I gave a crappy explanation of how it works, but that's the trick I found between conference talks and dare I say podcast episodes. You give people a glimpse and a hook and tell them where to go to learn more. Otherwise, you're trying to explain every nuance and every intricacy in 45 minutes.

And you can't do that effectively in almost every case. All you're gonna do is drive people away. Make it sound exciting, get them to see the value in it, and then let them go. You have to explain the market fit, right? That's it. Precisely. And I got to say, I somewhat disagree with your, or I have a different view on your say that, you know, open source projects needs marketing and all those things. It depends on what open source is for you, right? I don't create open source projects so they are successful, right?

It's obviously always nicer when they're successful, but, and I do get that cycle of happiness that as I was saying, people create bugs and I have to fix them and stuff, right? But not every open source project needs to be a success. Sometimes it's just fun. No, but I talk about marketing, I'm talking about exactly what we're doing here.

I'm not talking, take out an AdWords campaign or something horrifying like that. You built something that solved a problem for someone. The big problem that worries me about these things is, how do you not lose sleep at night by the fact to solve someone's problem and they don't know that it exists? Because that drives me nuts.

I've lost count of the number of times I've been beating my head against a wall and asked someone like, how would you handle this? Like, oh, well, what's wrong with this project? What do you mean? Well, this project seems to do exactly what you want it to do. And no one has it all stuffed in their head. But yeah, then it seems like open source becomes a little more corporatized and it becomes a lead gen tool for people to wind up selling their SaaS services or managed offerings or the rest. And that feels like increasing corporatization of open source that I'm not a huge fan of.

Yeah. I mean, I'm not going to lie, right? Like part of why I created this or I don't know if it was part of it, but like I had a dream that, you know, I'm going to get, Oh, tons of get up sponsors and everybody's going to use it. And I can retire on an island and just make money out of this right like that's that's always a dream right but it's a dream you know and I think bottom line open source is just a tool and some people use it for like you were saying driving sales into their sass some people like me use it just for fun and some people use it for other things or some people use it for politics things, or some people use it for politics even,

right? There's a lot of politics around open source. I got to tell you a story. Back in the ENSYS days, right? Talking about politics. So this is not even about politics of open source. People made ENSYS a battleground for their politics. We would have translations, right? People could upload their translations. And I, you know, or other people that worked on Ansys, right? We don't speak every language of the world. So there's only so much we can do about figuring out if it's a real translation, if it's good or not. Back in the day, Google Translate didn't exist. Like these days, we check Google Translate, we kind of ask a few questions to make sure they make sense. But back in the day, we did the best that we could. At some point, we got a patch for

Catalan language, and I'm probably mispronouncing it, but the separatist people in Spain, I think. And I didn't know anything about that. I was a young kid and I just didn't know. And I just included it. You know, someone submitted a patch. They worked hard. They want to be part of the open source project. Why not? Sure, I included it.

And then a few weeks later, someone from Spain wanted to change Catalan into Spanish to make sure that it doesn't exist for whatever reason. And then they just started fighting with each other and started making demands of me. Like, you have to do this, you have to do that, you have to delete that, you have to change the name. And I was just so baffled by why would someone fight so much over a translation of an open source project? Like these days, I can kind of get what they were getting at, right? But they were so bad at telling that story that it was just like, so basically, screw you for helping is how it comes across. Yeah. Screw you for helping. You're a pawn now.

Just you're a pawn. Unwittingly, just do what I say and help me in my political cause. I ended up just telling both of them, if you guys can agree on anything, I'm just going to remove both translations. And that's what I ended up just telling both of them, if you guys can agree on anything, I'm just going to remove both translations. And that's what I ended up doing. I just removed both translations. And then a few months later, because we had a release every month, basically, I just added both of them back and I've never heard from them again.

So sort of problem solved. Peace in the Middle East. I don't know. It's kind of wild just to see how often that sort of thing tends to happen. I don't necessarily understand why folks are so opposed to

other people trying to help. I think they feel like there's this loss of control as things are slipping through their fingers, but it's a really unwelcoming approach. One of the things that got me deep into the open source ecosystem surprisingly late in my development was when I started pitching in on the SaltStack project right after it was founded,

where suddenly everything I threw their way was merged, and then Tom Hatch, the guy who founded the project, would immediately fix all the bugs and stuff I'd put in and then push something else immediately thereafter. But it was such a welcoming thing. Instead of nitpicking me to death in the pull request, it just got merged in

and then silently fixed. And I thought it was a classy way to do it. Of course it doesn't scale, and of course it causes other problems. But I envy the simplicity of those days and just the ethos behind that. That's definitely something I've learned in the last few years, I would say. Back in the Ensis days, I was not like that. I nitpicked. I nitpicked a lot. And I can guess why, but it just, you create a patch, in my mind, right? Like, you create a patch, you fix it, right? But these days I get, I've been on the other side as well, right? Like, I create patches for open source projects, and I've seen them just wither away and die.

And then five years later, someone's like, oh, can you fix this line to have one instead of two? And then I'll merge it i'm like i don't care anymore it was five years ago i don't work there anymore i don't need it if you want to do it so i get it these days and these days if someone creates a patch just yesterday someone created a patch to uh format cdk get up runners in vs code and they did it just like a little bit wrong so i just fixed it for them and I approved it and pushed it. It's much better. You don't need to bug people for

most of it. You didn't yell at them for having the temerity to contribute? My voice is so raw because I've been yelling for five days at them, yeah. Exactly. I really want to thank you for taking the time to chat with me about how all this stuff came to be and your own path. If people want to learn more, where's the best place

for them to find you? So I really appreciate you having me and driving all this stuff came to be and your own path. If people want to learn more, where's the best place for them to find you? So I really appreciate you having me and driving all this traffic to my projects. If people want to learn more, they can always go to cloudsnorkel.com. It has all the projects. GitHub.com slash cloudsnorkel has a few more. And then my private blog is kichik.com.

So K-I-C-H-I-K.com. I don't post there as much as I should, but it has some interesting AWS projects from the past few years that I've done. And we will, of course, put links to all of that in the show notes. Thank you so much for taking the time. I really appreciate it. Thank you, Corey. It was really nice meeting you. Amir Sekheli, owner of CloudSnorkel. I'm cloud economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice. Whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment.

Heck, put it on all of the podcast platforms with a step function state machine that you somehow can't quite figure out how the API works. If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duck Bill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duck Bill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.