Screaming in the Cloud - Episode 70: Creating Custom T-Shirts through the Cloud with Ken Collins

Episode Date: July 24, 2019

About Ken CollinsKen Collins is a Staff Engineer at Custom Ink focusing on DevOps and eCommerce architecture with an emphasis on emerging opportunities. Custom Ink is approaching its 20th year in business and is entering its second phase in Cloud adoption where Ken helps an increasing growing engineering team succeed using AWS-first well-architected patterns. Ken lives near Norfolk, VA and organizes the area’s Ruby User Group.Links Referenced: Twitter: @metaskillsCustom Ink

Hello and welcome to Screaming in the Cloud with your host, cloud economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud. for having nearly every feature you could possibly want as a managed service at varying degrees of complexity. Others bias for, hey, we heard there was money in the cloud, and we'd like it if you would give us some of that. DigitalOcean is neither. From my perspective, they bias for simplicity. I wanted to validate that, so I polled a few friends of mine about why they were using

DigitalOcean for a few things, and they pointed out a few things. They said it was very easy and clear to understand what you were doing and what it took to get up and running when you started something with DigitalOcean. That other offerings have a whole bunch of shenanigans with root access and IP addresses and effectively consulting the bones to make those things work together. DigitalOcean makes it simpler. In 60 seconds, they were able to get root access to a Linux box with an IP. That's it. That was a direct quote, except for the part where I took out a bunch of profanity about other cloud providers. The fact that the bill wasn't a whodunit murder mystery was compelling

as well. It's a fixed price offering. You always know what you're going to end up paying in a given month. Best of all, you don't have to spend 12 weeks going to cloud school to understand all their different offerings. They also include monitoring and alerting across the board, and they're not exactly small time. Over 150,000 businesses and three and a half million developers are using them. So give them a try. Visit do.co slash screaming, and they'll give you a free $50 credit to try it out. That's do.co slash screaming. Thanks again to DigitalOcean for their support of Screaming in the Cloud. Welcome to Screaming in the Cloud.

I'm Corey Quinn. I'm joined this week by Ken Collins, who's a staff engineer at Custom Ink. Ken, welcome to the show. Thanks, Corey. It's great to be here. Long time listener, first time caller. Well, it's funny you mentioned that because it turns out that I'm relatively familiar with Custom Ink. For those who were around a year or so ago, well, a little less than that, I did a fundraising t-shirt where I had a picture of US East 1 as a burning dumpster on the front of it, and all proceeds went to benefit St. Jude's Research Center, which benefits kids with cancer, an issue that virtually no one who isn't a monster is on the other side of. And you folks were the house that handled fulfillment and printing of those t-shirts.

So I've kept a loose eye on you folks ever since, and it turns out you have some interesting stories to tell about cloud as well. So this is sort of a fun and interesting combination of something that happened in the real world, as well as something that interacts with my own ridiculous nonsense. Yeah, everybody wears t-shirts. You'd be surprised. I'm known for wearing suits for a living. Under your blazer. So let's start at the beginning here. What is your background?

How did you, I guess, enter into this whole world of cloud-native development? Yeah, and first, thanks for running that campaign and picking that fundraising platform that we do. I was glad to see that you were able to do that successfully and also contribute money to a needy cause. I've been at Custom Ink for about maybe six years now, maybe seven, I think six. And it's a very old company. They've been around for 20 years now, I think.

And it's just it's a it's a great story. Like we've been working on getting our cloud adoption skills up over the past two or three years. That's what my role plays into it. I'm a staff engineer. I help sort of make technology and architecture decisions across disparate teams and functional groups. And I find as a longtime Rubyist, I've sort of been retooling my career,

which I think a lot of Rubyists may end up doing as we learn to program the cloud more. Something that I find interesting is that the streams sort of crossed where I tripped over you, not because of any of the t-shirt stuff, but because you wound up releasing something called Lamby. That's L-A-M-B-Y for those who are sitting and not reading the transcript.

What is that exactly? So Lamby is an open source project that is basically a thin wrapper around your Rails project to help ship your Rails application to AWS Lambda. We'll get into that in a little bit, but I found that just to be a fascinating story in its own right. I've had the privilege of walking around these parts of the Custom Ink office, and you folks are bigger than I think most people would think. It's a t-shirt company that's effectively three people with a screen printing press,

and that's the end of it. It turns out you have multiple floors in a non-trivially sized building. How big are you folks? I think we're about 1,700 employees now. We maybe have about 60 engineers and those are split up mostly between our Fairfax DC office and a product team that runs a company called Represent as well. And maybe about half a dozen web ops people. And it's a nice, large-sized company that's still got some smaller roots. But we definitely have a lot of production facilities. We've got a huge production facility in Texas, Reno. So that's where most of the employees end up sitting is in sales service and production

facilities. And you've been in business for just shy of 20 years at this point. So I'm going to go out on a limb and guess that you weren't a cloud native company, given that there really was no cloud 19 years ago. And if we take a look at that, what does your architecture look like?

Well, we've, let's see. So a lot of these things went back before I joined, but mainly I think CustomMik was one of the first large companies that had a Rails app in production. And it was very, we still deal with that today. It's a big monolith. We have traditionally two monoliths like most companies do where you have a front end and a back end architecture. And after about 19 years, we started breaking that up more. We now have hundreds of applications and services, some of them on AWS Lambda. But most of our stuff looks like EC2. And about four years ago, we finished a major shift from sort of, you know, your co-located facilities and actually moved into AWS.

But it very much looks like a lift and shift, right? We had a lot of employees back in the day, like, say, Seth Vargo or Nathan Harvey, who used to run a lot of our config management. They've since moved on to bigger and better things. But our cloud infrastructure, as we moved it into there, looks very much like a bunch of config management and Rails running on EC2. Seth was a guest on this show in its early days,

and it's always interesting to me to see just how cyclical everything is, how everyone sort of ties back into the same thing. The world is never quite as big as we seem to think that it is. As you've wound up progressing, I mean, back in the heyday of Ruby on Rails, which was, what, I want to say early to mid-naughts, and now it sort of seems to have fallen into something of disfavor. And I understand that's something of a controversial statement, but there's an awful lot of stuff that wasn't written in the last

six weeks that runs powerful companies that do interesting things. These aren't just companies that are doing the Twitter for pets style nonsense, rewriting their stack every month because of something they read on Hacker News. There's validity to writing software that solves business problems. And a lot of that is written in Ruby. How do you square, I guess, working on a stack that is in many cases perceived as old news

because it's stable and doesn't crash every three minutes. So therefore it's not the programming language of the future with having to solve real-world business problems. I mean, you print T-shirts. It's hard to get more tangible than that. Yeah, I think when you say stack, I hear the words Rails and the newer stack I'm looking at is Lambda.

It's easy to get distracted and buy into technologies that may solve problems that are interesting, but don't deliver business value. And I'd like to think that one of the things that I do at custom make is always sort of keep that thumb on the scale when we're doing things that reminds people about what we're eventually doing, right? Customers don't care for running an EC2 for running a Lambda or any other thing

like that. They want a website that works. They want to be able to design on a t-shirt and have it look good. And you may be able to solve some interesting problems in some smaller pieces of architecture, but a lot of times it's, you know, we're exploring React in different ways. Rails is an easy choice, and I think there's going to be a sort of a second coming of Rails after v6 is released and using new technologies like Lambda, what I like to call full stack serverless, to borrow a term, and pushing bigger, larger things into AWS Lambda. So what does that adoption curve look like for you? When you go down the path of building out something that historically has been something of a monolith

and breaking that apart into individual components, first you have the technical piece, and that's an interesting topic in its own right. But I think what resonates a bit more with folks who are looking at that themselves is, what does that cultural transformation start to look like? I think we're going to hit two sides of that. One of the benefits of custom make is that we sort of have this well understood

ley lines of where our architecture and our sort of platform needs to be, right? After so many years of working, it's easy for us to see where this monolith needs to be broken up. Over about the last year, we've done quite a few high profile Lambdas that are just very small microservices. They do one thing very well. Some of them may compose designs on product images. Others may resize clip art when you're working in the lab. But those can be solved and you don't really have to think too much about the architecture. They do their one thing well. I think the thing that we're looking at now

with how to move the mindset into more cloud native, especially around Lambda, not getting into containers yet, is just thinking how much you can actually put inside of that Lambda, what it can actually do, right? So many people will build like, say, a node framework,

like what is one of the popular ones out there i think it was called oh i can't remember but like you could put a node framework that does a web application framework and you could shift it off and it would be small per se some people think it's small but i almost guarantee you that if you think of rails on lambd you might think that that's a it's a big thing and it's a big mind shift to move something that monolithic. And it's not. Rails is quite small when you look at comparison to Node projects.

So I think the mind shift is in two things. Realizing when you're dealing with a microservice and realizing when you're pushing into sort of full stack serverless with Rails and Lambda. And each of those have interesting problems to solve and topics to bring up. So why go in a cloud native direction? I mean, obviously what you've built and been working for you, it's gotten you this far, for lack of a better term. What was the advantage of moving

off of that legacy architecture? Well, I don't think we're going to know for another few years or so. But the bet is that we can better sort of adopt cloud native technologies, the full offering of AWS and ship faster. So many of our applications now are, you know, some people might find this horrific. We're deploying applications through Capistrano in that legacy manner. Shifting to Lambda is a way that you can get inside the ecosystem of AWS. From there, you can sort of branch out and say, okay, now let me go learn about DynamoDB or Route 53 for latency-based routing or these in number of other offers that they have out there and start to make good isolated platform decisions where larger amounts

of our teams can sort of make these decisions and get things out without having to be tied down to our ecosystem, right? Where you don't have to hook things into Capistrano for deploy. In fact, you don't even have to deploy at all. You can start bringing up more modern things like code pipelines and automatic deployments and stuff, green-blue deploys, etc. Gotcha. And that's not necessarily to say that there's a right or wrong path here. It's one of those areas where there's absolutely a capability story of taking advantage of new technological developments. It's one of those areas where there's absolutely a capability story of taking advantage of new technological developments. It's just always interesting to me to see how that manifests at a company that isn't venture-funded,

where engineers generally don't tend to have carte blanche to do whatever they want, and seeing how companies that are doing things here in the real world with a solid business model behind them are thinking about these things. Yeah, we always have the new apps that we have to build or the new services that are sort of tied to business OKRs. And we're generally left free on how to do that from an engineering side. We have good architects that help make decisions for us on what investments we want to make in certain technologies.

But the way to do that is really kind of left up to the team. We even have some groups now working on a new container-based system that might be ECS, that might be Fargate, it might be Kubernetes, who knows. It shouldn't be Kubernetes, but that's okay. Yeah, we don't want to give money to the Greek god of spending money. Exactly. It's always fun to wind up seeing how this stuff tends to come to be.

I'm not entirely sure there's any right or wrong direction as far as how most of that goes. I'm curious about your personal transformation as well. You started off as someone who was a long-time Rubyist, and now you've been transitioning into what looks an awful lot like a cloud-native developer. What's that been like? Well, I think I've sort of been putting it into three phases here. One is learning about.

The other is sort of choosing new tools and then building pipelines around those tools and those technologies. And those kind of run sort of concurrently and asynchronously together, and even sometimes recursively where one dovetails right back into the beginning. For the learning side, that's been really hard, right? I've myself have gone out and got AWS certified from, I believe it's called the developer associate. And being someone who hasn't gone through academic sort of training, right?

I don't have a CS degree. I'm very much self-taught. I was very hesitant to get certified. I kind of feel in some ways, right, certification is like you just learned to learn enough to pass a test, but learning about is really, and working in the tools sort of has more value. But I've ended up finding that certification is a good way to learn about something. It's kind of like why you go to conferences, right? You don't go to conferences to learn things. You go to learn about them.

And learning about AWS is just going to be a full-time job. That certification gets you a kickstart into that and hopefully gets you hooked into always keeping up with AWS offerings, hopefully with things like last week in AWS or any other avenue. But that learning is the first part, it's the key, right? You have to know about all these things in AWS because they're certainly asking you to architect solutions around cobbling all these things together.

And if you don't know the right stones to reach for, then you're not going to do well. I think you just hit on something very important, that the purpose of conferences is not to learn something new, it's to learn about them. I feel like an awful lot of folks go to the big AWS conferences,

and others, with a mind towards attending specific technical sessions to learn specific implementation details. At least for me, I've never found that to make a whole lot of sense. You want to wind up figuring out what those tools are, of course, and you want to see how those things wind up being used in the real world. But especially at a conference where all of the sessions wind up on the internet two days

later, I'm not convinced of the utility of flying halfway across the country to do that. I feel like people don't tend to come into conferences with the right strategy in mind. Maybe it's because when they go to the conferences, their only dialogue is that one-on-one they're getting from the conference speaker. They're not maybe going up and engaging the speaker before or after the session. Maybe they're not having enough hallway conversations throughout the conference about those topics. But I tend to agree, right? But I also like to go and learn about how other people are doing very sort of minute problem solvings and learn if I can fit that in my tool or not. I'd say if you're going

to a conference to figure out if Kubernetes is right for you, then that's probably not the right solution. I think that a lot of people also go the other direction and view it as just an excuse to go party somewhere for a few days. Sure, you can do that, but I don't find it to be the most productive use of anyone's time when we're talking about being around some of the smartest people on the planet when it comes to a particular technology. One of the things we've done at Customink is we sort of pick out these new technologies and how we'd like to work with them. And we sort of have a roadmap of

Keystone projects that help us understand the architecture, right? So we're not going to do a lot of meetings and make decisions about, all right, everybody, is it Fargate or Kubernetes? Let's hands up or hands down, right? We're going to maybe look at a few projects. We're going to spin some up and we're specifically going to call them out as these are sort of exploratory projects so that we're going to learn if we like something or not. And sometimes they may end in, you know, something being set down in stone and a sort of a ways of working, but we're really cognitive about like, Hey, this is something new. This is something we want to try. We don't even know if we know enough about it to make a call or not. Let's just explore that and let a few people go deep into the tea,

right? And just drop down really low, go learn all this stuff. And when they surface back up, we'll sort of share that laterally with the org. I think there's a lot of value in being intentional in the technologies you select rather than just what seems to have the most stars on GitHub or something similar to that. It winds up almost, instead of trying to follow the herd, you're trying to solve for specific technical problems that you have. That said, you can take that too far, and reinventing everything from first principles and going off in your own direction that is completely at odds with the rest of the larger

industry, if nothing else, it makes hiring a lot harder. Yeah. And this is one of the things, you know, so I do like GitHub stars. If you do start the Lambie project, I would love you to death. But I think Lambie, for me as a Rails developer, and I hope this is true for other Rails developers or anybody looking to do Ruby specifically on AWS Lambda, it's just this incredible thing where I can take this ecosystem that I'm familiar with, I can ship it off to AWS Lambda, and it forces me to reconcile the other things that are going on at AWS and how to be more excellent in that ecosystem.

And I don't think there's any better way to learn if something is right for you, right? Like I'm not even worried about containers right now in my career, but I will be in some few months. The one thing that I think that will right now in my career, but I will be in some, you know, a few months. The one thing that I think that will fit together with Rails developers, and I hope they adopt with the Lambdew project is that once you get into this ecosystem and once you have this capacity to sort of ship Rails to Lambda,

you're sort of forced to learn about these other things. And I'm just, I'm having a ball every day learning more about AWS. I think last week, I recently started putting some deep thoughts into how we deploy our applications across multiple regions, not for a sort of disaster recovery, but for availability and redundancy. And I would not have thought of that if I was spinning up a single dyno in Heroku or just putting things onto EC2 and throwing them over the DevOps wall, if you will. Getting to that end, how do you wind up hiring effectively into an environment

where you're starting to do something that is relatively unheard of in the larger industry? By that, I do mean Lambda. Until last reInvent, when they announced a Ruby runtime for Lambda, if you wanted to do this, you were using things like traveling Ruby and having to shove a whole bunch of things into place. And frankly, Rails itself never really seemed like much of a particularly Lambda-friendly environment when everything is built in a monolithic sense where there's startup latencies and the rest. And you've made that work with Lamby. So I guess my twofold question on that. The first is, how do you wind up effectively hiring people to start wrapping their heads

around this who have the experience that you need? And secondly, how do you wind up seeing this evolving over time as best practices among cloud technologies continue to evolve? I think you have to take two approaches. One is you have to hire people that you think have the right ability to learn and solve problems and then just train them into sort of your ways of working, whether that be Ruby or Node or what have you. And the other is maybe start to look a little bit sort of remote for your hires. Traditionally, custom make has been very centric to hiring in the Fairfax and Northern Virginia area.

But we're looking at maybe changing that. I think problem solving is the most important thing a programmer can do. Besides, say, communication or something else. Rails is not so hard. It's easy to train somebody into Ruby and stuff, but that is a, it is a hard problem to solve. Rubyists aren't coming out of the walls anymore. They're just kind of sitting there delivering business value softly in the background. Yeah. And for better or worse, that's probably the right answer. It's a, I think all of us need to deliver business value. And it

turns out that rewriting everything every three weeks is not necessarily the best way to do that for most shops. So as you decide to start down this path of converting what you had running in Rails into a Lambda function series, you wound up having to build your own framework to do this called Lamby. Tell us a little bit more about that, please. Sure. I'd like to think it's only several dozen lines of code, and a lot of it's sort of inspired from AWS themselves on how to get a Sinatra app running. Right now, Lambie basically just acts as a small wrapper

around your code in Lambda to send the handler to a rack-compatible app, which a Rails app is. It doesn't do much more than that. Everything else that the project talks about on GitHub is mainly tooling around using AWS SAM, which is the serverless application model command line interfaces, to package and build the app and deploy it out to production. If anything, it's more deploy scripts than a sort of a framework for Rails. The one thing that I really like about the project is that it actually,

other than the internals of your app and how you use say S3 or DynamoDB or anything like that, the project does not care about your Rails app, right? There's no difference between a Rails app for an EC2 instance or a Rails app for Lambda. It's basically just your Rails app and you're delivering it.

I like that approach a lot and I think it's going to afford us some certain decisions later on in the project's lifecycle when we eventually write wrappers for application load balancers versus API Gateway. But for now, mainly all the gem does is it acts like a rackup file where it mounts your application for the Lambda handler and just sends rack messages to it all day long. How constraining do you find a Lambda environment for the way that Ruby on Rails historically thinks about applications? The only one that I've really hit upon now is I've not really done any database connections. I know that there's the VPC cold start issue. So as soon as you attach a VPC to your Lambda, you're going to sort of

incur a penalty of maybe some 15, 20 seconds of startup time from cold. That's true if you use a Lambda with say Elastic Cache like Memcache as well. So there's those things that we haven't really touched on. A lot of the Rails apps that we've been prototyping in it are sort of isolated image processing. We're taking advantage of DynamoDB versus say Postgres or MySQL. And I have not found that too terribly discouraging, right? Like I'm an RDBMS guy.

I love databases. It's been a hard mind shift to start thinking of DynamoDB, but I'd say it's so database connections. Yeah, I think that historically the challenge I always had was that Rails tried to do everything for you under the hood, and that's great, right up until it wasn't, where it started doing things where I didn't agree with the data model, ActiveRecord had a bunch of issues with databases under the hood,

and effectively getting in and unwinding some of that, here be dragons. It always felt like it was terrific for rapid prototyping as something I did up and running quickly. But as that application grew and scaled, breaking it apart, even upgrading from one version of Rails to the next

was hell. There's really no nice way to put that. And finding ways to modernize that onto something that resembles a serverless stack feels like it would be something almost insurmountable. I wouldn't know where to begin. And yet, you didn't just begin, you did it.

Oh yeah, maybe it's my proficiency in Rails, but I find it incredibly easy. To me, it was an easy decision to go, hey, if I don't have an active record model, let me use the AWS record gem, which is a thin wrapper around DynamoDB. And it even encourages you to mix in the things that active record does. So like out of the box, Rails will give you active model validations, right? And the same patterns are requested in the AWS record gem, which is basically you bring, we'll bring DynamoDB to the table for you. You mix in active model validations and pretty much you just feel at home again.

It does require a little bit of sort of tool sharpening and bringing things to bear on it. But I think the gems out there to get things done, if you're looking at a full stack serverless, which I definitely recommend people do, there's definitely uses for small applications where Rails app just does one thing through a couple gems. The problems are easily solved. If you're doing things like image processing or if you're not moving your

Oracle database over to serverless, then I think it'd be just fine. What's next for the project? What is it that's keeping it back from the next step on its path to greatness? I'd like to do a few regional conf talks about it where we do some workshops. We've done a few internally for Custom Ink where we sort of walk engineers through creating their own Rails app. Very much sort of like a 201 class where you get a little bit past hello world. I think for me, it's basically going to be moving to, what is it,

the application load balancers for the event versus API gateway. I read a recent blog article that said that was a lot of cost overhead. Not a lot. Everything in Lambda is going to be cheap, but you could probably get a little bit better performance and a little bit more cost squeezed out if you switch to the application load balancer. And then other than that, it's really just building a community around it and finding out stories and people talking about how they solve certain problems. People can sort of pull more

resources together and understand, hey, if I had an application that did one small thing, what's the pattern that you might choose from? My Rails application might be using more of the JavaScript stuff. What's the best way to build the assets for that and ship them off to a CDN, et cetera? So how do you see the future of the project evolving now that Ruby is a first-class runtime in the world of Lambda? I think it's going to grow. I think there's going to be a lot of people,

a lot of Rails programmers that are, to date, shifting their careers, doing things like migrating Heroku instances to AWS that are doing this type of consultancy that need to learn more about AWS. And with tools like Lambie out there, they're going to be able to make some really good decisions for them on how certain Rails applications go into AWS Lambda and how it uses more of their ecosystem. I'm excited to learn more about people's stories about what they can share on the Lambie website. We can contribute more blog articles and also technical content for the product website on how other people can sort of take those learnings. And that's really where I'd like to see it go. Just more success stories and more adoption and more documentation. If people want to learn more,

where can they find out about Lambie in general and you specifically? So Lambie is located at lambie.custominctech.com. I'm pretty much meta skills on every other thing. So whether that be Twitter or Xbox or LinkedIn or what have you. Ken, thank you so much for taking the time to speak with me today. I appreciate it. Thanks, Corey. I've had a good time. Ken Collins, staff engineer at Custom Ink. I'm Corey Quinn.

This is Screaming in the Cloud. This has been this week's episode of Screaming in the Cloud. You can also find more Corey at Screaminginthecloud.com or wherever Fine Snark is sold. This has been a HumblePod production. Stay humble. this has been a humble pod production stay humble