The Changelog: Software Development, Open Source - Sidekiq and Ruby (Interview)

Episode Date: June 7, 2013

Adam Stacoviak and Andrew Thorp talk with Mike Perham about sustaining open source, sidekiq, message processing with Ruby, and more.

Welcome back, everyone. This is The Change Log, where members support a blog and podcast that comes with fresh and what's new in open source. You can check out the blog at thechangelog.com slash nothing in our past show. I don't know why I said nothing. Check out the blog at thechangelog.com in our past shows at 5x5.tv slash changelog there's the slash the show's host by myself adams dekoviak and andrew thorpe you're in uh aloha town man what's up hey how's it going yeah we're uh down in waikiki beach in hawaii i think this is the the first changelog that's being recorded out of hawaii yeah that's that is uh that's amazing and speaking of listening listening to The Change, we'll be continuing live every Tuesday at 5 p.m. Central Standard Time right here on 5x5.

And this is episode number 92. And today we're joined by none other than Mike Parham. He's a Rubyist known for gems such as Sidekick. If you haven't used that, you're wrong. Dolly and Lunchie. Welcome to the show, Mike. Thanks for having me, guys. So just to clarify, Mike,

is it Perham or Perham or how do you say that? It's Perham. Perham. Got it. Just like Birmingham, Alabama. Birmingham. Yeah. My favorite person to say Birmingham, Alabama is Brandon Mathis, man. I love the way he says it. He's been on the show before, too, and I don't know if you know him, but he does Octopress. Oh, sure, yeah. Yeah, he says Birmingham the best, in my opinion. Just a little side note. Yeah, I don't know how Alabama folks pronounce it,

but if they pronounce the ham, then they're obviously doing it wrong. Speaking of things just going right and wrong, but good to have you on the show. But I wanted to have you on the show for a long time. You know, we at Pure Trader, we use Sidekick. And as a matter of fact, it was pretty neat when we first started to use it because it was so fast. It made us change things quite a bit. But because it would just run through jobs, it's super easy.

But for those who may not be familiar with who you are, give the listeners an intro to who you are, Mike. Sure. Well, I consider myself a Rubyist at this point. I've been doing Ruby for about seven years. Before that, I was doing Java for almost a decade. But I'm a longtime open source enthusiast and developer. And Ruby just happens to be my tool of choice these days.

So, yeah, I've been working on, you know, the first thing I was really known for in the community was probably Memcache Client. I took that over about five or six years ago and polished it up. And then from there I moved on to DALI because I wanted to sort of write the next generation of Memcache client. You said for a while you've actually been writing software for a long time. I mean, I guess in comparison to comparison maybe some people come on the show my first open source project that i released the source code for was 18 years ago oh and what was that if you don't mind me asking it was a application launcher for windows nt35 wow yeah you go and and and that was back when I didn't know of any open source out there. There was really no Windows source code available for you to use as sort of a reference when you were building things. So all I had was the MSDN documentation.

So I just wrote this little tool and put the source code up on my website. And this was, I was in college at the time and when you're in college, you're doing research projects and sort of everything's open source because you're an academic. There's no commercial aspect to what you're doing. So I put the source code out

for this little Windows application out there. And yeah, just open source and Windows at that time was non-existent. It was all shareware. I haven't always been in the open source community, but that's kind of neat to think about 18 years ago what open source was like and how drastically different the landscape has become with not just GitHub, but just democratizing sharing code and everything from the movement GitHub

that started with sharing code and Git and all that stuff. But it's pretty wild to think about that. You're showing your age, and perhaps I'm showing my age, because GitHub wasn't the first revolution. The first revolution for me was SourceForge.net. They're a popular punching bag these days, but I joined them, I think, within about the first month of them going public. And that was a revolution too, because before then, you had to set up your own CBS server. You distributed tarballs through an

FTP site, generally. Having this software as a service and for free to the community was really revolutionary. It seemed like you really had to want to be a part of the community at that point, right? I mean, you really had to push to get in because the barrier was not quite as low as it is today. Yeah, and I think each source forge lowered the barrier to some extent, and GitHub has lowered it even further. That's just improvements over the lifetime of a community, I guess you could say. Yeah, SourceForge is, like you said, it's kind of the popular thing to be the punching bag. But there's something to be said about a service that's lasted as long as they have. I mean, you'll still see a project every now and then that is on SourceForge. And I think, I don't know if iTerm2 is still on there, but it was on there for the longest

time. So it's still around. It's still definitely kicking, which is, there's something to be said about anything on the web that lasts that long, you know? Yeah. Well, you know, like anything, their heart was certainly in the right place at the start. And, you know, it just, entropy has, you you know taken it down over the years to where uh you know the the ui was was let's just say not great and uh you know there was ads everywhere

and and it was obvious that the user was not the customer we were just eyeballs for their ads and they just happened to be in the developer space and And that's why GitHub sort of ate their lunch. That's a pretty unique little thing you just said there because one of the ways we – I don't know if you're a long-time listener and follower of the change law, but we kind of had a dark period this past year around August to December. And it was just kind of reorganization and whatnot. When we relaunched, we decided not to put ads on the site and a bunch of other decisions. But we tried to, I guess, and if you didn't know this, the changelog is member supported. So if you're listening, you can go to the changelog.com slash membership and sign up to support what we're doing. We have writers that cover all sorts of open source. And I'm sure we've even covered Sidekick at one point in time.

Actually, not recently. We had a nice post from Kelly Martin, one of the guys that works with us at Pure Charity, covered Sidekick. So I think even in there, he was talking about how fast it was. So it's just pretty wild. But you'd mentioned who the customer is, and I think that's pretty neat to look at,

how GitHub has changed their focus on open sources, helping developers be better developers and helping code be more shareable, more forkable, more liberated, I guess is the easiest way to say it. Right. Yeah, it's really kind of revolutionized. Well, again, you've got revolutions every five or ten years it seems like, but they've been certainly the latest revolution. And by all accounts, they're a huge success. So why don't we go ahead and – this is a great conversation. Why don't we go ahead and jump into the meat of the call. And so we kind of want to talk about Sidekick and kind of what it is and not necessarily just how fast it is, but why is it so fast? So I think, Mike, if you don't mind,

I think a good thing to start with is why don't you give us a little bit of a lesson on what message passing and message processing is and how it's been handled in Ruby traditionally and what makes Sidekick different. Sure. Yeah, I mean, I was thinking about this this weekend and how to describe Sidekick. My current best thoughts around how to describe Sidekick is it's a background processing framework. I think of it kind of like Did we lose Mike?

I think so I'm here, I pressed the mute button on my microphone accidentally Sorry about that guys So Sidekick is a framework for background processing kind of like Rails is a framework for web applications

and I try to provide all the tools necessary to build an application that has a non-trivial asynchronous processing component to it. And so that's sort of what shapes the design and the directions that I go with regard to Sidekick features and functionality. When you get right down to it, Sidekick's competition is gyms like Delayed Job and Rescue, Q Classic, those sorts of gyms. But Sidekick is different from those three in that it is explicitly multi-threaded. Instead of running one thread per process

and having each process process one job at a time, you're processing many jobs concurrently. And that's really where it gets its speed. Sidekick is not any faster than Rescue if you change Sidekick's concurrency to be one. But if you're using Sidekick in its default mode, it's going to process 25 jobs concurrently, which means that if you have one rescue process

and one Sidekick process, Sidekick's going to ultimately go 25 times faster, all other things being equal, of course. And that's where it gets its speed, is the concurrency and the multi-threading. So the idea of a message processing, so for anyone who doesn't know, like, if there's something that you want to push off into the background that you don't want to, you know, take time for, let's say, on user create, you want to send an email,

you would push that into a queue to be processed in the background so that it doesn't take, you know, take up time on the front end for the user to wait for those emails to be sent. Correct. Exactly. Anything you want to do asynchronously is a possibility to push into Sidekick. Let me give you an example. You gave the example of an email. That's a great example. Any third-party call that we do here at my full-time job here at the Climb, we try to push that into a Sidekick job so that we don't have explicit third-party network dependencies in our application execution. Sidekick has this full-featured, robust retry mechanism. So if a job fails, Sidekick will actually retry it. And when you've got a third-party network call, the network could be down, that third party could be down for maintenance.

There's any number of reasons why that call could fail. And so having a retry mechanism for asynchronous processing is critical, in my opinion. Yeah, that retry mechanism you talk about is really interesting. And I think if you've maybe come from, you know, rescue or delay job and move to sidekick, which I think a lot of people have, it's kind of, you got to, it could almost be a gotcha. So the idea is that if the job fails, there's a, what do you call it? A in-between retries. A retry queue. But in between retries, the amount of time increases. It's an exponential back off. Right.

So the idea is, what is the amount of time between the first retry? It's 15 seconds. And then between the last and second to last, how long would you wait? I think it's like three days. Right. So the idea is that if you're not, like it gives you a lot of time to fix it, and then it also deals with the case in like you said, network issue or something like that that's going to resolve itself very quickly. And so that's something that we had to get used to up here at Charity because, I mean, I think we maybe jumped the gun a little bit and just started using SideQuick. SideQuick.

Hey, that's a good name. We just started using SideQuick because know the potential speed boost we could get and there were a few gotchas you know that we kind of had to just learn and the retry one uh while it's it's i mean it's a lifesaver in every case it definitely was something we had to make sure we read about to understand what was happening the the other gotcha that I think you've actually documented in the wiki that people have had to deal with is the after commit thing. Can you kind of talk about that a little bit? Yeah, of course. The gotcha is that a lot of people want to perform background processing on a newly created database record. So you create a new user record, and now you want to send them email.

So generally what you do there is you create the user and then fire off a background job with that user's ID to Sidekiq, and then Sidekiq will look up that user and then send them an email. The problem is that Sidekiq is so fast that sometimes that user and then send them an email. The problem is that Sidekick is so fast that sometimes that user creation transaction has not actually committed such that the Sidekick database connection still can't see that user object. And so it'll throw an exception saying no such user exists. And then 15 seconds later, when Sidekick retries that job, now it'll exist and now it'll work. So people find themselves often seeing errors that immediately fix themselves. job into an after commit callback so that you know that the user record has been committed

and is visible to everybody else in the database before the sidekick job is created. Yeah, so basically it's not necessarily a gotcha. It's just, like I said, it's something that you have to just be aware of, and it tends to fix itself for people that never actually address the problem because of the exponential backoff on the retry queue which is pretty cool that exponential backoff seems i mean just from an outsider who doesn't do much of of what you guys are talking about i'm still kind of in the early echelon of uh being a true hacker i suppose but um you said 15 seconds and the next retry is days later right right? No, no, no. It's exponential backoff, which means it does 15 seconds, then 30 seconds, then a minute, and then five minutes.

And so it actually does about 25 retries at ever-increasing delays. After that last retry, if it fails on that last retry, what happens to the job? It actually calls a callback on your worker if it's defined called retries exhausted. And if you don't have that callback, it just discards the job, assuming that it will never succeed. So you can address the problem of it failing 25 times however you want, basically. Exactly. Yeah, that's pretty cool. So I think we talked a little bit about it, but I would love to kind of talk about, so was the main reason that you decided to create this, even though there were solutions like Delay Job and Rescue out there,

was to take advantage of the growing popularity in the multi-threaded Ruby environment? Well, there was a couple of reasons why I wanted to build Sidekick. Performance was certainly probably number one. I worked on Rescue for a customer. Previous to my current job, I was a consultant at Carbon 5, which is a consultancy in San Francisco. And they had a client which was using Rescue,

and they were doing thousands of jobs. And they had, I think, 10 different machines that were just dedicated to running rescue. And they were actually using JRuby, which is the worst of both worlds because the JVM is this big behemoth. And the way you achieve efficiency through the JVM is you run it a lot of threads, but rescue is single threaded and it forks. So now you through the JVM is you run it a lot of threads. But Rescue is single-threaded, and it forks. So now you have these JVM processes which are gigantic, and they're single-threaded. So it's absolutely horrible for efficiency reasons. So what I did is actually I patched Rescue to be multi-threaded, and they went from 10 machines down to one machine because we could leverage threads.

And so, so once I saw that sort of kind of benefit, I realized there had to be a market for some, some improvement here. And so I started building Sidekick. Since then I've, you know, there have been other reasons why I think Sidekick is a great leap forward. I think another big one is the fact that Rescue is rather bare bones in its basic configuration. It doesn't have a lot of extensions. It doesn't have a lot of APIs. It's really just I process jobs on a queue, and that's basically it. Sidekick just has a lot more, like, scheduled jobs.

It has the whole retry system. It's just, and it's got a full meta API so that you can actually query Redis for all the different jobs and queues and workers and what they're doing currently. It's just got a lot more features to it. And so that's another big reason why I felt that with Rescue I had to bring in 10 or 15 different plug-ins just to get a decent full functional system. So you kind of talked about you were working on this through whatever you were doing until your current employment situation. So what was that like, just starting this project when you were working somewhere else? Sorry, go ahead. You're, you're, um, let me correct you. The timeline's actually wrong.

I started Sidekick when I left Carbon 5. I had, uh, I think two weeks downtime when I was moving from San Francisco to Portland. And during those two weeks of downtime, I, I wrote the first version of Sidekick. Basically, I, I didn't, I was basically, you know, I, I, I quit on Friday and on Saturday morning I was like, well, what do I do? Um, nice. Well, that, that multi-threaded rescue thing was pretty awesome and I haven't had a lot of time to build it. Why don't I build it? And so I just started working on it there. So, uh, so yeah, the, the, the impetus, the idea came from working with a client of Carbon 5's, but I actually started it after I had left. Okay, gotcha.

So what was the – so you have this Sidekick, the gem, and you also have Sidekick Pro. When did the idea to fork that into its own service come about? So when I, when I wrote Sidekick the first couple of months, I had a, having been a long time open source person, I'm not a 20 year old guy that can afford to spend his nights and weekends on everything, on stuff, just, just to learn stuff. I have a family and a wife and kid. And so I wanted people to be able to pay me for what I was doing. And so when Sidekit first came out, I gave people the ability to pay for a license. Basically, I released it as LGPL, and then I allowed people to pay me 50 bucks to get a commercial license if their lawyers didn't like it. And this brought in a couple hundred bucks, but at the end of the day, it was chump change compared to the hours I was actually spending

on Sidekick. So at that point, there's a fork in the road and you really have to decide what you want to do here. With a lot of big open source projects, they go the consulting route. Take Ember, for instance, is one currently. And of course, MySQL is a long time one where you have this open source core project, but then you have services and consulting and training around it that also brings in money. And so I had to decide, well, do I want to be a consultant and, and try and drum up business and maybe, maybe hire people and start a company around this or, or, or what do I want to do? And, and when it came down to it, I just didn't want to do that. I enjoy my job right here. I enjoy having free time with my family.

I didn't want to start a startup, a one-man startup to try and build this thing. So I decided to go the product route and actually try to build a premium product on top of the open source foundation. And that's really what Sidekick Pro is. It's a set of functionality that extends the free open source version with some really valuable capabilities.

And that's where I think a lot of open source people who want to spend months and years maintaining projects, they need to spend months and years maintaining projects, you know, they need to, they need to get paid for their time. It's, it's a valuable resource that they're providing to the, to the community. And I don't think there's anything wrong with either accepting, either building some sort of value added product on top of the open source or, you know, asking for like get tips and that sort of thing. So that's, that's been. So that's been kind of an interest of mine over the last few months is how do we make open source not only valuable to young people who are trying to learn and wanting to hack on weekends on small projects,

but also longer term, bigger projects that really entire communities are relying on, like Rails, like Sidekick, like maybe Sinatra, and those sorts of gems. So a lot of people do this kind of a thing like you have Redis and you have Redis to go, which is just providing Redis as a service. But Sidekick Pro is a little different in that you actually enhanced Sidekick and added some functionality, not just providing purely Sidekick as a service in and of itself. So what is the functionality that you added to Pro to make it Pro? Right.

So there's a couple big features that Pro has on top of Sidekick. The first one is this notion of a batch. So you can create a set of jobs which, when all those jobs are complete, you can have callbacks called or you can have notifications sent out. And that is really valuable from a scatter-gather kind of standpoint. If you think about some work being done, you want to, and if you think of Sidekick, Sidekick really tries to be a framework for building concurrency into your application so that you can parallelize a lot of work. But the problem is that by parallelizing things asynchronously, you don't know when anything is done, right?

You want to receive an email, for instance, when your thousand jobs are done, but you can't do that with normal, the base Sidekick. And that's what a batch allows you to express, is you say, I want to create a batch of these thousand jobs, and when all thousand are complete, send me an email or call this method. So that's the first feature that Sidekick Pro gives you. The second feature is reliability. The Sidekick tries to be as reliable as possible, but there are native extensions and Ruby VM bugs

that cause the Ruby VM to simply crash. And there's nothing Sidekiq can do about that except to change the way that it enqueues jobs in Redis. And so Pro offers you an alternative way of enqueuing such that if Sidekiq crashes, that job is still in Redis and it's not lost. Because with the base Sidekiq, when you pop a job off to work on it it's popped off into memory and it's it's gone it's gone from redis so if that job does not if the vm crashes that job is lost so those are those are two of the big features that that people have been buying Pro for.

Right, so it's kind of the best of both worlds. Anyone who is a, you know, not, I wouldn't say anyone, but, you know, most people who are experienced Ruby developers could probably develop Sidekick Pro or, you know, because of the foundation that Sidekick offers. But the amount of time it would take makes that $500 cost to get Sidekick Pro, like you said, seem like chump change. So it's absolutely worth spending that money if you need those utilities. It's funny. I've gotten many people saying that Sidekick Pro is too cheap. And I've gotten a couple people saying that Sidekick Pro is, that's ridiculous. How dare you charge $500 for that thing? But yeah, like you say, when you think about the cost of a

Ruby freelancer, a good freelancer is going to cost you $100, $150 an hour. So you're talking three to five hours of a good developer's time and you get this functionality. And if it solves a problem for you, you know, you pay the money, literally you have it 10 minutes later and, and the problem solved. So, so yeah, I mean, the reality is there's a lot of businesses out there that are willing to pay a couple hundred bucks to make a problem go away immediately. I had, um, a customer at RailsConf who found that his sidekick processes were crashing and he was losing jobs. And I told him, listen, you can continue to do sidekick, but you're going to have to debug why this crash is happening. And time is

of the essence here because you're losing jobs every day when these things crash. The alternative is pay $500 and make the problem go away. It's not the sidekick code that's causing the crashes. It's something in the application and the gems that it's using that's causing it to crash. So that's out of my control. I'm sorry. I feel bad that the process is crashing. I certainly hate it when it happens to me.

But you pay a couple hundred bucks and the problem goes away because you restart Sidekick Pro and the jobs just start processing again. Could I ask a question that maybe everyone else is thinking of this when they're listening to you talk about this, but since Sidekick is open source, and I guess you are the core computer, of course, since you're the creator of it, but is it plausible or is it possible for someone to fork it, add similar functionality to Sidekick Pro, and kind of do that? Like send a pull request. Would you accept that? Would you accept things that mimic or recreate Sidekick pro functionality? Well, that's a great question. Um,

you know, you could fork, you could fork sidekick and as long as you do what's legal under the license, you know, it's, it's fair game. I think there's, there's a difference between what's cool. Of course, don't do this. Anybody. I was going to say there's a difference between what's moral and what's ethical. Yeah. I'm not saying anybody should do that. First of all, I wouldn't say anybody should do that. I'm just wondering if you thought about that, if that's a concern really. I actually did. But I think the amount of time it would take to create the features is non-trivial such that if you want the features, just pay for it. Is your time really worth so little that you're willing to spend 20, 30, 50 hours to rebuild this feature

and then release it to the public? I don't know that that really makes a lot of sense. And like I said, it may be legal under the license, but yeah, at the end of the day, it's just not very cool. I work really hard and spend a lot of hours on base Sidekick giving that away for free. For someone to just sort of copy the features and release it and sort of eat my lunch, it just doesn't seem very friendly. Can we talk about your lunch for a little bit? I'm just curious, since this is 500 bucks a pop, you don't have to give out any exact numbers. I'm just curious how successful this has been for you because it's half a grand. Well, it's been for

sale for, I think, eight months now, and I'm nearing 100 customers. Wow. Nice. So I wanted to ask you actually, you kind of hit on it a little bit, Adam, and I wanted to ask you, has anyone else released Sidekick as a service, be it the pro version or other features or just the base Sidekick or anything? That's an interesting question. And certainly when I was thinking about where to go with Sidekick, doing Sidekick as a service was one of the directions I thought. There's a company out there called Iron.io, I think it's called, that does message processing as a service. And so it definitely is possible to do it. My biggest issue is simply that I have to provide an execution environment for people's worker code. So the Ruby code has to

execute on my sidekick servers. And that means that you have to sandbox Ruby. And you basically have the same problem that Heroku has, which is you have possibly a malicious application running on your server. So there's this whole sandbox that you need to build. And that's non-trivial. And I didn't know how to do that. And I didn't, to be honest, didn't really have a lot of interest in building it. So that sort of deep-sixed that idea. Gotcha.

I wanted to roll back. I meant to ask you this a minute ago. Rubius love semantics, right? And they love to spend tons of hours arguing about what's the right way to do something. What's the Ruby way? What's the Rails way, right? So I wanted to ask this. Before, we were talking about rescue.

So let me get this in real quick. The perform method in rescue was a class method, right? And in Sidekick, you decided to do it as an instance method. I was hoping you could kind of elaborate on why you chose that. Sure. To me, that's a fundamental decision due to the multi-threaded nature of Sidekick. The reality is when people write code,

they're going to use instance variables simply because they're not going to pass method arguments to every single method necessarily in the class that they're using. So when you use instance variables in an instance, you're multi-threaded safe. But when you use instance variables in a class method, you are extremely thread unsafe. And so the reason why I designed it that way is because I'm trying to guide people

to writing Ruby code that is multi-threaded safe and will work well in Sidekick. And so using a class method perform would immediately cause threading problems for almost everyone as far as I'm concerned. So yeah, I had to change that to make people's code safer. Gotcha. So you can assume, being a Rubyist, there was a, as you put it, a fundamental decision as to why it was that, and not that you just willy-nilly chose that. So it's good to know. Yeah, exactly. So being – has there been any interest in the rescue or delayed job or any of those camps to kind of mimic sidekick and go multithreaded? Or have you heard about anything like that? I've heard rumors here and there of Rescue 2.0 being under development

and them wanting to provide sort of pluggable concurrency. You know, you could use a forking model or you could use a threaded model or maybe a hybrid approach. I'm not sure. But I don't know what the latest of that is. As far as I know, I mean, I heard those rumors a year ago and I don't know what the latest of that is. As far as I know, I mean, I heard those rumors a year ago, and I don't know if they've made any progress or what. I certainly haven't heard of, or I don't know what the latest is with regard to Rescue 2. Gotcha. And I haven't kept up with delayed job at all. I think you're with most of the community on that. Well, when I was building Sidekick initially, I certainly trolled through their readmes

looking for features that were cool and stuff that I could sort of liberate and reuse myself in Sidekick. And from delayed job, I took the delay method because I thought that was a great idea. And certainly from rescue, Rescue heavily influenced Sidekick's data formats in Redis and the initial sort of way it worked. Right. For anyone listening, that was not intended to be a shot at Delayed Job.

Just I think that we've kind of – we used Delayed Job. Well, I look at it similar to how we were talking about the Source Forge and other things. Delayed Job was like a lifesaver for us at one point, and they've been around for so long now. But I think the Ruby community loves to go with the hot and fresh and the new, whatever that is. So Sidekick is hot and fresh, but for those who maybe just learning Ruby or just getting started, how does one choose one of these three, delayed, job, rescue, or sidekick? Do you just jump in right into sidekick, or is it so fast that you just can't hold it down? You've got to maybe try something else to keep it slow for a bit, maybe not worry about the after commits and stuff.

Like any sort of software decision, you have to evaluate, you know, what's out there and what's appropriate for you. A lot of people don't want to use Sidekick because they don't want to bring in Redis, for instance. Right. A lot of people choose QClassic because they're already on Heroku. They're already on Postgres. They literally need add nothing except the gem and a couple Ruby classes, and that's it. And that's perfectly okay. Sidekick isn't perfect for everyone. I use Redis because I think it offers an amazing amount of functionality.

You just have to decide what's appropriate for you. QClassic is great if all you want is Postgres. As you begin to learn. Like you mentioned earlier, Andrew, those that are seasoned Rubyists, they have certain things about them, class versus an instance,

those types of things. I feel like even me, as I learn Ruby, and as I get deeper into learning Rails and using it, that I kind of graduate into certain things like, oh, I should use this versus that. And I just wondered what the process might be there. And it's a good point mentioning Postgres and even being able to use it on Heroku and not having to do extra things to utilize Sidekick. Well, and there's something to be said, too.

Kenneth writes on one of our previous shows, talked about the tribal knowledge that these communities develop, right? So Sidekick was birthed out of – that's a weird way to put it. But it was born out of a need, right, that Rescue was not necessarily solving. And so a lot of people who were using Rescue said, oh, this is what I need. So you migrate to Sidekick. So somebody who's just now coming to the community, they might, you know, see these are all my choices and I don't know which one to pick. But somebody who's been doing message processing for the last, you know, eight years has kind of followed the trend as the needs have grown and the solution has been, you know, created in the sense of Sidekick versus Rescue versus delayed job and the different technologies so there there is something to be said about that you know and so when a newcomer i mean i would venture to say that when a newcomer jumps into the community their choice will most often be whatever is recommended to them from the people they ask and so you know i mean for me personally yeah for me

personally like i am gonna typically recommend sidekick because of the trend in multithreading and where you can go with it and the amount of efficiency that's gained from it. And that's just kind of the way it goes. It's that tribal knowledge that the communities develop. Well, since you mentioned it like that, I mean, and we're kind of talking about this, what's the overhead for those that might want to use Sidekick? You'd mentioned the need to utilize Redis and stuff like that. I don't, I'm not sure even, I don't, I know I don't use Redis, but at least not yet. But what is the overhead of adding Redis to your application stack

and getting up and running with that? I think that Redis is a pretty amazing piece of work. Antires is a pretty awesome developer, open source-wise. I mean, very knowledgeable. The community loves to argue about cap theorem and what have you around his work. But at the end of the day, Redis has been amazingly reliable and hasn't never given us a single problem. So I'm a big fan of Redis in general. Adding Redis to your application is pretty darn simple.

I mean, Redis is almost as simple as memcached as far as I'm concerned in terms of setting it up and running it. And then you just point Sidekick to it, and you're done. There's really not a lot of administrative overhead to Redis. And that's on purpose. That's one of the reasons why I chose Redis, is I wouldn't ever think to use something like, for instance, Cassandra as a data store for Sidekick, just because it's so complex to set up. And, you know, it's really designed to run on many machines.

Well, Sidekiq needs to be able to scale from a single person running on one machine to, you know, an application running on a dozen, you know, dozens of machines so uh you know redis kind of fits that bill pretty well so the the i've used lunchy dolly and sidekick all in different times of my life and one thing that's interesting to me and and i maybe you can kind of speak on this a little bit is sidekick has by far been the most popular of the tools you've created, even though all of them have served a need at some level for me. So what do you think it is about Sidekick

that has made it so popular compared to other projects that you've done? Well, when it comes right down to it, Lunchy is just a little command line tool that solves a very, very basic problem. It's not solving big problems that people have every single day in terms of their application and how to build their business. Dolly is basically just another, it's a faster Memcache client, but it basically connects your application to Memcache. There's no way to spin it and make it more than it really is there. Sidekick, on the other hand, is, like I said,

I've tried to make it into this framework for background jobs. A lot of people who are developing Rails apps need to process things asynchronously. I've tried to add as many features and bits of functionality to Sidekick to make it extremely useful, while also maintaining it's as simple as possible to get started and also keep it high performance. Gotcha. So Sidekiq really solves a problem of how do I make my application something that is asynchronous friendly and highly performant.

So it's the size of the problem and the size of the potential audience that's creating the popularity. That makes sense. I think so. I think, yeah. What about Girl Friday? I guess this is kind of built on top of Sidekick. What is Girl Friday and how is it different? The last couple of years

of my Ruby open source have been focused on scalability and performance and certainly Memcached is part of that. I work with Memcached, Client, and Dolly. Another problem that I was solving over and over and over at all the Ruby companies that I was working for was how do you do background work?

How do you do asynchronous processing efficiently? Because I wrote a system for a company called Five Runs that I worked for six or seven years ago. And this background processing thing was called Qbert. And it was called Qbert because it was a queue that was in the database. And guess what? It was basically like delayed job, except before delayed job came out. And so when I moved to my next company after five runs, I wrote this thing called Jobber. And it was a background processing worker that pulled jobs. I forget where it pulled them out of, if it pulled them out of the database or what. But as you can see, there's a trend here. Every company I've been going to, I've been

working on these asynchronous processing systems, and they've always sort of been less than what I needed to build in the long run. In the case of Qbert and Jobber, the database is not the right place to put the queue and they were single threaded, so they weren't terribly efficient. And so when I went to Carbon 5 and worked with this client that had the rescue problem, I saw I'm solving the same problem over and over and over. And so that's sort of where Sidekick came from. Girl Friday was sort of a different stab at solving the same problem. Girl Friday runs inside your Rails process. So it uses threads also, but instead of being a separate process that uses Redis as sort of a data exchange between the processes, Girl Friday was literally

a set of threads running within your Rails process, and your Rails code could just hand jobs to those worker threads, and it would process them in the background. Ultimately, there was some implementation details that I got wrong in Girl Friday that made it a little more painful to maintain than I really wanted. And also I realized that keeping the worker threads in the Rails process wasn't necessarily the right solution. So that's when I sort of focused, took my focus away from Girl Friday and started focusing on a new project that is Sidekick. Gotcha. So I don't want to bring this up and I don't want it to seem awkward or anything, but we had a little back and forth on Twitter a while ago when I mentioned something about the tone and some of your responses to the pull requests can be kind of aggressive. And you actually responded to me.

Well, I didn't actually even mention you, but you responded to it and said it was a definite weakness of yours. And you've tried to be better about that over the last few months. Did you experience flack from the community? And how did you respond to that? And how do you feel like you've kind of grown from that? Yeah, it's, it's something that, that I struggle with and I see other people struggle with. And certainly the internet is something that can turn into a flame fest at the drop of a hat. Um, you know, people, people need to realize, and, and, and I'm one of those people that people have a bad day. Some people are in completely different mindsets and don't see your viewpoint.

And so that's something I struggle with. I try to provide really quick support and try to respond quickly to people's support issues. But sometimes that means that I might give a glib answer or make a joke that is perhaps off tone for really isn't as professional as I should be. So that's something I struggle with, and I think a lot of open source people struggle with that. We're logical people, and so we think our words are going to be taken logically. And oftentimes readers take things emotionally instead. So, that's something that I've tried to tone down and just try to stay as professional as possible when I'm helping people in issues. But yeah, for sure, there's been times where I've let loose a response that I wished I could have taken back 30 seconds later. Yeah, I think the most important thing, I think that everyone can say, you know, everyone can say, you know, things that they don't mean or that aren't the nicest things or aren't the

most professional things to say. But I think the most important thing is the ability to, you know, self-evaluate and determine if you need to adjust that. And I think that's something I respected that you said to me when you said that that was a weakness and something you were trying to be better about. I think the ability to see critically at yourself is a very good thing. So I applaud you for that. Yeah, thanks. You really have to know where your strengths and weaknesses are. And my strength is code.

And so that's what I tend to focus on. And, but part of, part of managing an open source project is dealing with people and interacting with people and helping your customers and helping your users. And, and that's something that, uh, you know, I'm, I'm just not a, I'm not a community support person necessarily, um, by, by a profession or by, um, um, by skill, by skillset, I guess. So I hope the community bears with me and tries to take what I say not necessarily personally or maybe forgives me a little bit if something seems off kilter. It certainly is not. Do you think that these types of situations, though, might suppress some people from releasing certain things to the open source

community because of it's kind of like you know for designers they have dribble you know show and tell there and you know for developers and hackers like us it's you know github is that playground for us where we release our code there we want to share we want to improve you think it helps you or makes you want to suppress some things or others maybe? I definitely think that that's a factor. I mean, I've talked to a lot of people over the years who, when I say, you know, you should blog more, because blogging, it helps you write, it helps you collect your ideas, helps you think. And a lot of people tell me, well, I don't know what I'd say or I don't know how to, you know, I don't really have an opinion that I think people would value. And for sure, I think a lot of younger developers

who might hold me or somebody else in esteem, they're going to be really worried about how the community and their heroes might perceive them. So yeah, I think a lot of open source people are a little hard, thick-skinned, shall we say. And a lot of people with thinner skin, they don't have the, yeah, I wish they had the courage to stand up and just try it. But yeah, sometimes the community can be a little rough. You know, look at, you know, Hacker News. You post an open source project that you want to show off to the community, and 50% will be positive, but that 50% that are negative is rough. Yeah, don't read the comments. Yeah, those are the ones you're going to remember at the end of the day.

You're not going to remember the good ones. You're going to remember the bad ones. And that's unfortunate, but that's a part of life. I wish it wasn't the case. And I've said some stuff before, you know, Mitchell Hashimoto of Vagrant fame, I unwittingly said something negative about some open source project he put out years ago. And he reminded me of it and my jaw dropped. And I couldn't be prouder that he moved on past my negativity and, and went on to, to do, you know, awesome stuff. Um, and it just shows you that, you know,

I try to be positive, but you know, sometimes people are jerks and we're all human, man. You know, we are all human. And in the digital world, when you put words out there, you're right. I mean, cause I can read your joke and I'm like, was he just talking crap about me or was that a joke? And like Andrew at Pure Charity, he's the jokester. So you're never really sure how to take Andrew. But, you know, and that's the case. And it's kind of a bummer. But I think one thing you said that rings true for me is just remind people that, you know, it's like the clerk when you go to get a pack of gum or something like that.

You're not sure if not saying hello to them is going to give them a crappy day and they're going to be a jerk to their wife or to their girlfriend or to their mother. You know, you've got to be nice to everybody. And sometimes you're just not sure what type of day somebody is in. And you may be in the perfect day and they're in you know just a bad place you know you never know right so one of the things that we encourage people to do we we obviously at the changelog we love open source and we try to encourage people to to contribute but you you're right people are a lot of times hesitant to you know start a whole project and contribute all the source code open you know to github or whatever so that everyone can see it and critique it.

So what we try and do is encourage people to jump into other projects and submit pull requests. So on that note, do you have any call to arms that you would like the community to kind of get involved with Sidekick to maybe some features or something you would like to see? Anytime people ask me that, I kind of tell them, if I think of a feature, I'm the type of guy that's just going to go that night and start to implement it. What I'd love to see and what I've tried to provide in Sidekiq

is a framework and APIs for building asynchronous processing. And I just want to see what people can do with it. Because oftentimes the coolest stuff that people come up with is stuff that I just never would have thought of in a million years. And so I've seen 10 or 20 gems built on top of Sidekiq at this point. And I just, I love to hear stories about what people have built on top of Sidekiq. I've got, at RailsConf this year, I talked to a handful of Sidekiq users that were processing over a billion jobs a month with Sidekiq, which I think is awesome.

And so I just love to hear those sorts of stories and love to see kind of what cool things people have built on it. So I don't have any brilliant ideas for people to add to Sidekick, but I'd love to see what you could do with it. So just be creative. As you find a need, solve it. Yeah, and blog about it and let me know about it. I'd love to give you publicity about the cool things that you're building. On that note, if you don't mind me asking, Andrew, I'm curious, Mike, what you thought about Kelly's post on the changelog about Sidekick.

Remind me again. It was something about the three or four caveats to Sidekick. Yeah, he says, earn a Sidekick black belt by breaking a few boards. And his very first point was, is too darn fast. Um, and that's, that's a, that's a great compliment. Um, yeah, you actually, you actually, uh, straightened out some confusion we had on manually retrying failed jobs. Um, was that, I guess this is, you know, was that something that has always been there in the failed job in the retry queue? Yeah, in fact, that's one of the confusions that I sort of have in how people use Sidekiq is I'm not sure if people just don't read

the documentation or what, but I've got a wiki with tons of documentation that goes through each of the big major features that Psychic has and how to use it. And people have wanted to turn off retries. They've built gyms that add different types of failure handling. And I just don't necessarily understand why they're trying to do that. I think of the retry mechanism as awesome as designed, and I use it myself everywhere. So I'm not sure where they're coming from. So I don't know if it's just a matter of they didn't read about Sidekick's own built-in retry mechanism, or if they have some sort of functional need to where they can't retry or they need to do retries manually or something like that.

But yeah, that was one of the confusing points I had about that blog post is Sidekick's retry mechanism is awesome and built in and it works by default. So I wasn't clear why that point needed to be made at all. Yeah, I don't know. I mean, I remember when we started using it um there was it was the idea that we couldn't find failed jobs and i don't know maybe we're just dumb i'm not sure why why it happened but uh maybe it may have arose from once they if we weren't utilizing that um retries exhausted callback they were gone you know who knows now it's been a while but kelly's probably in the back channel too, just like, no, this is why.

Well, the Sidekick retry queue is fully visible in the web UI. So if you have the web UI hooked up and you want to see failed jobs, just click on the retries tab and it's all fully listed right there. And you can run individual

retries manually. Like right now, you can say, if you fix a bug that caused a retry job to fail, you can deploy that code and then go into the retries tab and click on the job and just run it immediately. Or you can wait for the exponential backoff to sort of just naturally rerun the job. Right.

What about a programmer hero, programming hero? Do you have one? Oh, who do I have? Tony Arcieri obviously is, is fundamental to Sidekick's success. He, you know, he's the founder and, and project lead for the Celluloid project. And he's done an amazing job of really improving Ruby's concurrency story. And Sidekick is heavily multithreaded, but the code base really doesn't contain any mutexes at all. And I am a much better person for that. When you build a multi-threaded application with Celluloid, you just don't need to use mutexes,

and by virtue of not having to do locks, your multi-threaded code becomes much easier to reason about and much easier to write and maintain. So he's my hero, and Cellulite has certainly been critical to Sidekick's success. Some other guys and or girls that might be my heroes, Jeremy Kemper I think is an awesome... He's been doing Rails core,

Ruby on Rails maintenance for years now, and every time I meet that guy, he's just doing Rails core, Ruby on Rails maintenance for years now. And every time I meet that guy, he's just incredibly knowledgeable about everything. And yet he just sort of programs in the background and keeps it pretty low profile. But I wish I could be as knowledgeable as him about Ruby in general. I was taking aback a little bit by your some of your background too i didn't realize that in addition to being a rubius you're also a motorcycle racer that's not something often you find on somebody's bio um yeah i i have a i have a sport bike and uh i live about five miles away from a racetrack here in portland so i i So I take my bike to the racetrack about once a month generally,

and I do what's called track days. So I basically pay some money to go around the track as fast as I want all day. Wow, man. So literally all day? Yeah. Basically, you're broken into three groups of fast, medium and slow and you decide which group you go into and then you get 20 minutes an hour and then they just rotate every hour and they do that for eight hours a day. That's going to be pretty wild.

Yeah. It's, it's a lot of fun. I, I hit 170, um, down the front straightaway last time I was there and nearly saw my life flash before my eyes. Wow. So not only is your cycle fast, your psychic is fast as well, or your cycle is fast as well. Exactly. It's really great having you on the show. Thank you so much for taking time out of your day to join us.

I mean, it's really been great for me to hear a lot of this and for the listeners listening out there that want to use Sidekick or are like, what is that Sidekick thing again? I heard Chainsaw say it's really fast, but I'm not really sure about it. So really awesome for you to come on the show and schooling us on Sidekick and message queuing. But for those of you listening, follow Mike on GitHub and Twitter. He's mparam.

This is show number 92. You can find show notes at changelog, or sorry, at 5by5.tv slash changelogs. That's 92. I'm a little off my game today. I'm not really sure why, but let's close this show out.

Say goodbye. See you later. Thanks so much, Mike. Thanks, guys. see you later thanks so much Mike thanks guys