The Changelog: Software Development, Open Source - Capistrano and Burnout (Interview)

Episode Date: October 30, 2013

Adam and Andrew talk with Lee Hambley about some serious subjects such as Capistrano 3.0/2.0, open source burnout, various conversations around deploying, Ruby, respect, handing over the reigns and more. If you hack on open source or run an open source project, you should listen to this episode.

Welcome back, everyone. This is The Change Log. We're a member-supported blog, podcast, and weekly email that covers what's fresh and what's new in open source. Check out the blog at thechangelog.com, our past shows at 5by5.tv slash changelog, and subscribe to our weekly email. It's called the Changelog Weekly. We ship it out every Saturday, and it covers everything that hits our open source radar. You can subscribe at thechangelog.com slash weekly.

This show is hosted by myself, Adam Stachowiak, and also my partner in crime, Andrew Thorpe. Andrew, say hello. Hello. How's it going? I'm perfect. I think we had a joke in a previously recorded version of this that was really awesome that won't get told now, and I'm kind of bummed about that, just so you know. Don't worry.

No one needs to know. No one needs to know. But this is episode 110, and today's show is sponsored by two sponsors today. I'm pretty happy about this. Digital Ocean and TopTal. TopTal is one of two sponsors today. I'm pretty happy about this. Digital Ocean and TopTal. TopTal is one of our latest sponsors. I'm going to tell you a bit more about them later in the show, but real quick, they connect startups, businesses, and organizations to a growing network of elite engineers around the world. So stay tuned. We'll tell you more

about them. But Digital Ocean is a fan favorite.'ve been helping us for for quite some time big supporters of open source big supporters of the changelog but you know them you love them and today i want to tell you about their by the hour pricing plans a cool thing about digital ocean servers is that you can spin up a cloud server by the hour if you only need a server to test an app for a short while the smallest one costs costs just.7 cents per hour. That's right,.7 cents per hour. You can deploy the service for only as long as you need to and not pay any extra. But if this is the first time you're hearing about DigitalOcean, they're a simple cloud hosting provider.

They're dedicated to offering the most intuitive way to spin up a cloud server. You can create a cloud server in 55 seconds, and pricing plans start at bucks per month or by the hour, 0.7 cents per hour. And you get 512 of RAM, 20 gigs of SSD, one CPU, one terabyte transfer. They have data centers in Amsterdam, New York and San Francisco. And you can try them out today for free. That's right, for free. Use the code TheChangelogOctober when you sign up. It's TheChangelELOGOCTOBER.

That'll get you $10 of hosting credit, which is equal to two months free. Spin up your cloud server today at digitalocean.com. And Lee, Lee Hambly is joining us today, or is it Hamly? It's Hambly. Hambly, I was right. There you go. Awesome. Big score.

Lee, you are joining us. You are the maintainer of Capistrano. We've got a fun-filled show lined up, so Andrew, just take us away, bud. Yeah, so once again, we're joined with Lee Hambly, the maintainer of Capistrano, has been maintaining the project for quite some time now. While I think everybody, well, maybe not everybody, a lot of people in open source obviously have heard of Capistrano, if not used it. But I'm sure there are plenty of people out there that have never heard of it or that have not used it.

So, Lee, why don't you give us an introduction to who you are, kind of where you come from, and then a little bit about Capistrano, what it is, and some history. Yeah, so absolutely. My personal kind of background is self-taught jack-of-all-trades developer. Started my early career writing horrible Perl stuff and converting a lot of that stuff to Ruby on Rails, realizing things were nicer on the other side of the fence. And that's when I got stuck into Ruby in general and the very early version of Rails. I think version two was pretty new when I got stuck into Ruby in general and the very early version of Rails. I think version 2 was pretty new when I got started.

Around the same time, we were looking for a way to get these apps deployed. Of course, deploying Rails hurt a lot more than deploying Perl or PHP or some of the other stuff. And back in, what, I guess 2006, 2007 or something, Capistrano was pretty new. Didn't have a great deal of documentation around, and we were really keen on using it. So as we began to use it, I began to write documentation and handbooks and answering questions on mailing lists and everything else. And eventually the original author, Jameis Buck, who worked for 37 Signals, had a kind of breakdown, burnout, I don't know what exactly, but he wanted to spend more time with his family and was looking for someone to take over maintainership.

So that was me as a not very experienced Ruby developer maintaining a kind of infant project back then. And, yeah, Capistrano just kind of snowballed from there. The following version of Rails had it as a default in the gem file. A lot more people got introduced to it from there. And it's always had a very close relationship with Rails. And even in version 4, Capistrano standing right there in the gem file is the, which I guess is unlikely if you're in the Ruby world, it's a Ruby DSL for specifying tasks to run

on one or more remote servers, specifically targeted at doing application-level deployment, not really server provisioning or that kind of stuff, but specifically making a directory on the server for the current release, synchronizing everything with Git, checking it out, linking shared files that shouldn't be checked in like database configs and whatever else. And in the end, restarting everything. It's always been very, very tied to Rails and very, very tied to Ruby. And it's been rewritten recently.

It was released, I don't know, something like 10 weeks ago now, maybe a little bit less. And it's been rewritten from the ground up so that it could be useful for other frameworks, particularly PHP. There's a lot of guys deploying WordPress with Capistrano, which is kind of funny. Yeah.

That is nice. Yeah, that's unique. That's cool to hear. I wasn't aware of that. Maybe we can get into's a little yeah that's unique that's cool to hear i actually i wasn't aware of that maybe we can get to get into that a little bit but um capistrano you mentioned kind of has a had a relatively big change and a lot of stuff came about this i think you know with um the post that you wrote and uh the release announcement came back in june so capistrano version 3 released 10 weeks ago. Why don't you kind of give us a rundown of the major differences? And that was the first major release in about five years, right?

So huge, huge release. And why don't you kind of give us a rundown of what was Capistrano version 3? And why was it five years between the major releases? So one of the big reasons that Capistrano went unchanged for so long was that it basically worked for most people. And there were a lot of situations making small changes to Capistrano itself, where all of a sudden people's deployments were breaking, sites were falling offline. And it was because of very trivial things, such as we changed some shell escapings to fix Unicode issues that were reported

by German users and that broke shell escaping and people relying on having spaces and path names deploying to Windows servers with Sigwin and all kinds of other weird stuff. So it seemed like whichever way we went with version two, we couldn't make everyone happy. And when I say we it's mostly myself and a few people who've come and gone over the years um I don't know whether that's because I'm not very hospitable as a host or whether it's really just not very much fun working on this stuff and um of course when Capistrano was written Rake existed but was very immature in order to write Capistrano, Jamis also had to write NED-SH in order to actually be able to speak the SSH protocol. And he also wrote SCP and SFTP drivers with Pure Ruby. And well, all of that stuff's really complex. And

there was no easy way to change things there was no culture of gem files or bundler or any of this kind of stuff until fairly recently i would say the last 18 months to two years it's become the norm for people to kind of lock their dependencies capistrano starting to be that thing that people put in their gem file now and the kind of whole structure has been put in place by the community which has allowed me to make a big break and say look this has been hurting for way too long and I need to make a big break and actually we were able to cut Capistrano down from thousands of lines of code and now I think it's about I know 1500 maybe and that's because we're leaning on rake we're leaning on a driver called ssh kit which i extracted from it's a slightly nicer library of a net ssh and it's a lot more modular

now it's a lot simpler and one of the biggest changes is that it now no longer ships with any rails assumptions so the rail support has been moved out to a Capistrata Rails gem, and that itself is split into two pieces, migrations and assets, which was a big thing that always frustrated me because I will not use the Rails asset pipeline. Yeah, I've read your laments about the asset pipeline many times now, so that's definitely not something that you try and keep secret. No, I mean, I have to be careful because one of the things that kind of drove my burnout was people being disrespectful towards Capistrano. In the very early days, some guys came out with a tool called Vlad. And Vlad was kind of Git push-based deployment, as far as I remember.

And their tagline was, Vlad sucks less than capistrano and you know it's cool to poke fun at projects which don't really agree with what you're trying to do but uh you know there's years and years and years of work gone into pretty much everything we use in open source and we have to remember that is that part of the the poisonous people sapping your energy, as you have said? Is that part of it or is it something else altogether? I would say I don't want to kind of overstate how difficult it is or understate how difficult it is. But I think Capistrano sits right on the boundary where things get complicated. Rails is very, very, very friendly, very beginner-friendly. I think it's a bit less beginner-friendly

than it has been in the past. But it's still fairly easy for guys to throw together a blog in a couple of days with very little programming experience. And then, of course, everyone says, hey, do you know this thing, Capistrano? You can deploy in minutes. It's super easy.

And then they slam into this incredible learning curve where they have to understand git and they have to understand ssh keys and they also have to understand unix permissions and file mode creation and what the executable bit means in different directories and all of a sudden they don't understand why sudo doesn't work, why stuff is working in SSH and it's not working in Capistrano. And all of this stuff is incredibly complex and stuff that Seasons developers, we learn that stuff bit by bit over years and years and years and years and years.

And I think if you've been on a good run, you've just built your tiny little toy application, you're having a great time, and then you need to deploy it, you hit all of these incredible issues. And the first place you go is GitHub, and you open an issue saying pretty much all of them that we see are, hey, it doesn't work for me. And they never tell you.

And these guys, they don't know that they're not telling you the most important thing, which is that they installed Ruby from Aptitude, and it needs pseudo-access to do anything. And guys who are new to this stuff, they have no idea that that stuff's important. They don't know that they don't know. And we have to kind of triage those guys which i would say is something like

60 or 70 percent of every support query we get is somebody who just didn't know that whatever you can't use rvm or capistrano doesn't behave the same way as an interactive ssh doesn't or i don't know specifically another issue we run into all the time is SSH keys. And people say, hey, well, I can push and pull from GitHub. Why do I need another set of keys for the server? And you can tell them, hey, you don't. You can use agent forwarding. Or you can say, just go on the server, generate another key, put it on GitHub.

You mentioned at one point, and I don't remember where exactly I read it at, but you said the future of deployments was in the platform as a service type of thing. How do you feel? I mean, just kind of hearing what you're saying, I think there's a lot of – I mean, yeah, I've spoken to you obviously now a couple times, and I think that you have a passion for open source and a relatively positive outlook. But you obviously have some things that have kind of burned you. How do you feel about the Herokus of the world that do just the one-step Git deployments and stuff like that for those types of users?

Yeah, I mean, I have to say I really like Heroku. The stuff that they've built is incredible. And the kind of engineering behind that platform is just amazing and i have the feeling that all the guys who work there have an amazing job and they love every minute of it that's just rainbows and unicorns in my head because everything they're doing is the stuff that i wish everybody could do because it just looks cool um but it's another black box and I would say for the most part in our industry, black boxes are totally okay. You know, there's most of us can get by with active record and just treat the database

as a black box and trust that it does what it says it's going to do. And if it doesn't, then we're really in trouble. And i work on kind of bigger applications i'm a consultant day to day and some of my clients have got kind of 350 big incredibly powerful servers and incredibly powerful you know multi-data center failover situation setups and for that kind of stuff heroku is never going to be the correct solution but for you know like the guys from rap genius i think as far as i know it's a fairly typical rails app they're paying heroku something incredible like twenty thousand dollars a month or something was on hacker news and it works for them it's's a black box. They have the money, they can afford it. So it's perfect. And the black box isn't a big problem.

And a black box that just works is much better than a kind of digital ocean. Sorry to pick on your sponsor, but their VMs are great, but you have to know what you're doing with them. And that's kind of where everything falls down. And it's the same with anything. If you're going to use a tool you have to at least understand basically how it works and heroku if you don't

mind i can plug one thing real quick just on their behalf they are doing some really cool stuff around helping the community document that stuff too like they have this community center and they obviously they you know they know that i'm not like sticking up for them but just just so it's clear like they they want to help educate. So they're totally on board with what you're talking about. For some people, you don't know. I went in and I was hacking on a VM with Docker on it, and I messed around and I played with it. But how I got there, because I'm not at DevOps, how I got there was reading a tutorial.

But it was very thoroughly done, and they pay $50 for you to do it. So they do want to help people get to that understanding, the understanding too yeah and i think that's really important and that's um that comes back to kind of why there's been no big issues uh big big changes rather on cap strato over the last few years is that i was focusing kind of wastefully it seems on education and trying to tell like everybody who came along all the hundreds of them, look, this is what you're doing wrong, this is how it works in theory. Yes, I know you think the tilde should be expanded,

but that magic thing that happens when you type tilde and push tab, that's not Linux, that's your shell. And your shell's not there when you're using Capistrano or whatever else. All kinds of weird issues like that. And it really comes down to education. And there will be the guys who have the time and the energy to learn,

and there'll be the guys who don't. And it's kind of time is money or money is time. And if you have the money, I think Heroku is pretty expensive, and it's a black box. Go ahead and use it. And if you really need a tool and you know what you're doing then go ahead read a tutorial use your knowledge anyway and build something around Capistrano I mean it's it's really a tricky tool because we try really really hard to make sure that whatever for example the

RVM integration does its best to figure out which weird way you installed rvm and make sure everything just works but we can't do it perfectly and it's these edges where people have fallen down weird cracks and edge cases that you never imagined those are the vocal minority and they're the guys you hear from they're the guys complaining that whatever. They're the guys complaining that, whatever, they installed RVM via aptitude as well because they read it on a tutorial somewhere. And the tutorial was from 2008. And, you know, they don't know what they're doing wrong. They're following a tutorial that somebody told them about or they found via Google. And they're just trying to ship an app. And unfortunately you very rarely hear from,

from the guys who are being successful and it wasn't at all. Bummer. I mean, they should speak up, you know, hop in there and create a praise issue, right? Yeah. Well you, you, one thing that, and we'll get into this. One thing that I think people tend to do is like, if you look at the issues on like a project like Capistrano, and I think this is the frustrating thing, and maybe you can speak to it, but somebody will experience something. Maybe it's a bug. Maybe it's them not understanding what's going on or a feature that Capistrano just is never going to even attempt to solve for, you know, any number of reasons, right? There's like infinite amounts of possibilities and somebody will come and report it with very little, you know, they give you very little information.

They give you very little, you know, like context and all that. But then like a hundred people will come and comment plus one and that's it. So like it's a ton of pressure for the person that's maintaining this to be like, yeah, this is also affecting me. This is affecting me. And it's like the whole crowd just starts raising their hand, this is affecting me, but nobody's giving you context or information on how to solve it. And, oh, by the way, it's an open source project, so if it's affecting this many people, somebody can step up and attempt to solve it in a way that, you know, follows the contribution guidelines. And to me, when I see as a consumer of open source, when I go to a project and I see something like that, and it's not just Capistrano, it's all over the place, it almost makes me sad because I'm like, well, I wish I had tons of time where I could jump in and try and help solve these problems that aren't affecting me. But with this one project that I

just happened to just be perusing through the issues just to see what's going on. It's like, why don't more people even make an effort or an attempt to try and solve the problem? What do you think? I mean, how do you feel when you see those new issues come in with just tons of plus ones on them? Yeah, I mean, that's, that's been a big driver behind the rewrite actually is prior to the rebuild. Capistrano was incredibly complicated. We had this weird kind of automatic variable lookup with fetch and set with defaults and lambdas and everything else, and that was really complex. We had some really complex concurrency code, and we also had a really complex self-baked DSL, which also was kind of weird. And also internally, most of the stuff inside of Capistrano used load instead of require. And historically there were some good reasons for

that, but it also just made things really weird. So whenever somebody came to contribute, there were basically no tests. The code was incredibly complex, not very well organized by modern standards. I think when it was originally architected, there was no kind of best practices around how to structure a gem, how to test it and everything else. And Jamis did a great job, really. So it was, you know, it's a five or maybe six or seven year old code base. So it's really no wonder that it was ready to be rewritten. And a big driver behind the rewrite was to keep it as small as possible and use stuff which people were familiar with.

So now, as I say, Capistrano is on top of SSH kit and SSH kit lies on top of net SSH, which is the, the low level Ruby driver. And it's kind of awkward because you have to do stream handling and error handling and event stuff. So SSHKit is, you know, you can connect to a server

and run something in one line of code. You don't have to worry how it works, but if you do need to see how it works, the whole of SSHKit is like 700 lines of code. And Capistrano is just the synchronization stuff on top of that so it brings the the default tasks you need for Rails it brings some structure like setup tear down hooks for various places in the deployment and as well as I said earlier Capistrano is also tiny including all the test cases it's like way less than 2,000 lines of code, I think. And so to kind of mitigate this barrier to entry that people have contributing to complex projects, version 3 is really designed to be approachable.

And I don't know if it's a general kind of fear of getting involved in projects, but we haven't seen really the level of adoption in in new contributors that we would have liked we've we've picked up a couple of guys who are doing amazing work on the plugins for rvm rb env and ch ruby and also bundler and rails they're getting tons of pull requests and tons of little fixes and i think that's in part because it's now so much simpler and even ssh kit is getting fixes as well but there's no real big changes being pushed for capistrano um one guy is doing some great work in fact i don't want to name names but everybody who's contributing is doing a great job and the level of contribution has definitely increased but also the number of kind of silly issues has increased as well which is probably a

documentation issue on my part, because there's a lot of people showing up saying, hey, I did this in version two, and now it doesn't work in version three. Yeah, right. It's a new version. We tried our best. Did you maybe consider reading the documentation? And they always say, hey, no, I just expected it to work. So I think... Go to the semantic versioning website. Right. I mean, you're always going to run into those guys. And we've had an incredible number of people coming along saying, hey, why did you rewrite? Version two was working perfectly for me. Now everything's broken.

And you really have to bite your tongue with those guys because, you know, just whatever. Stick with version two, you know? Yeah. O Capistrano was written for you. Don't you remember? Right. No, you talked about – I think you said you brought up the RBM and RBM, CH Ruby and stuff. I've seen – and we'll get into a little bit of the burnout stuff. in this project and just in general with Ruby is the lack of standardization around things like, I mean, you know, the Ruby version management,

stuff like that, Ruby gems, like the, it was a, it's a, it's great work. It's a, you know, kind of, it solved a lot of problems, but it maybe wasn't, you know, it's not very standard and Ruby failed to kind of really lean on like standards in some of these areas. Kind of, can you kind of speak to that a little bit and kind of expand? Yeah, I mean, RBM and RVM are two great examples because one of them was built to be tiny and

do as little as possible. And the other one was built to be as comprehensive and all things to all people as each other. And they both basically fulfill the same job um and again i i'm very keen to stress they're both really cool projects and we we're in contact with the maintainers from both projects to help keep the capistrano integration tight and i think the biggest thing i mean capistrano isn't magic, right? I mean, the basic premise is that you have a release that's timestamped or individual somehow that you can roll back to. Sure, you could do the same thing with Git, but the main thing is the current

symlink, the shared directory and linking everything. So the biggest win from Capistrano is the best practice. And you can have this Capistrano style deploy with Chef or Puppet or Ansible or SaltStack or any of these other tools. But the best thing is we've all pretty much agreed on how you should deploy a Rails app, pretty much. And if you go a level lower and you start to speak about kind of interpreters and Ruby environments and databases and everything else, the people who work at that level have also kind of standardized it. If you use Red Hat, then it's YUM. If you use Ubuntu or Debian, then it's Aptitude. Or, you know, we have solutions for this stuff already in production environments and i think in a lot of small companies or a lot of one-man shops they've written the application they're a developer

they're just not interested in the server and there's an open source project promising to make all the problems go away and so people naturally gravitate towards those things and that's okay but it's not the way the operating system is expecting that to work. It's not the way your shell is expecting that to work. It's kind of a hack because it's relying on scripting, sourcing dot files and environment files. And it's kind of very un-Unix-y. And I know that term means different things to different people, but

yeah, I mean, it's a nice workaround in a lot of ways, and in a lot of ways it's not. I'm actually a big fan of the RVM binary wrappers because that speaks to me a little bit more as a Unix guy. I mean, you're modifying the path, there's a shim executable, which just makes things work.

And you don't really have to worry about shell state, having the right stuff sourced. It's just about the path and everything else is irrelevant. And that seems to me to be the best way to do things. But I've been around servers for like 15 years now. So, you know, I have a lot more history in that area than somebody that's maybe just going through their first getting started with Rails book and then wanting to put something online. So it's difficult because we try and support most of the

same use cases, but you know, we're, we're never going to have 100% coverage. And, in fact, speaking about edge cases, we actually have one guy who's contributed a fix for SSHKit, and he's deploying from Solaris, which is weird anyway, onto a Windows machine. And his Windows machine is running PHP, WordPress. I don't know what web server he's using, but he's running Sigwin specifically so that he can have an SSH server, specifically so that he can deploy with Capistrano. And that's got to be the most far-out use case I've ever had. And it works for him, and that's amazing. It got to work. Yeah.

Yeah. Well, let's take just a minute and pause, give our sponsor, TopTile, a shout-out. They'll be sponsoring the show for the next month. Got to give a huge shout-out. I mean, I got to admit, if no one has heard of TopTal, and it's spelled T-O-P-T-A-L, TopTal, like it's in TopTalent, I had a chance to kind of meet up with our co-founder and CTO, Brandon, and at first I was a little skeptical.

I wasn't sure what to expect. But Brandon helped me understand who they are and what their mission is. And I've got to say these guys are the real deal. They're engineers from top to bottom. They're not non-technical recruiters trying to, for lack of better terms, pimp developers. They're a network of engineers who work with some awesome clients. And I was kind of surprised too, Andrew, because we linked out to them to their engineering blog recently that we had a post explaining

Python implementations that Ian wrote and so they've got an awesome blog to go with it but for those of you out there who are freelancing or for those of you out there who like to test freelancing or even try out a no risk freelance like project while maintaining your full time position

there's a lot of Twitter bios I read that say I do x by day and do x by night and that x by night might be like meteor no js or rails or whatever and if you want to do that that by night thing uh you got to check out top towel you can work on special projects with companies like airbnb rc audio you can do it remotely on a beach andrew i I know you're a fan of that. Or anywhere in the world. I've worked on many beaches. I know. I know you actually have. And many others. They've got a very high

touch, very close relationship with these types of companies. Head to toptow.com slash developer right now. Click on join the best. And when I say join the best, they literally mean join the best. And when I say join the best, they literally mean join the best because they want their clients to work with only the best senior engineers

that are smart, enthusiastic, and driven, not just yes people. Andrew, I know you and I both kind of don't really care for yes people, but they want you to be able to say no if you're working on a project and they're doing something silly. You gotta say, hey, this is a bad decision. They want people to work with to say no if you're working on a project and they're doing something silly. You've got to say, hey, this is a bad decision.

They want people to work with them that are like this. And because they want the best for their clients, they've got a well-thought-out four-stage screening process that begins with something very personal, a Skype conversation face-to-face. The call includes – I'm sorry. From front to back, the entire screening process includes an English-speaking test, a personality test, a timed algorithm test, technical interviews with core TopTile engineers, as well as a test project. But once you've made it through the screening process, the sky's the limit. And if you think you have what it takes, head to TopTile.com slash developer right now. Tell them the change log sent you, and you will be well taken care of. And do me a favor, too. I want to get feedback

from those of you who try this out. If you're going there and you're going to apply, send me an email, adam at thechangelog.com. Let me know what your feedback is and what your experience is because we're really excited about working with TopTal and we want you to have the best experience possible. TopTal.com slash developer

that's T-O-P-T-A-L dot com slash developer to get started and apply. And, Andrew, I know you've got some good questions waiting for Lee, so I'll give it back to you. Yeah, so you talked a little bit about the version management, things like that. When you – a little bit earlier in the show, you brought up vlad uh vlad the deployer i think is what it was called little play on vlad the impaler but um vlad the deployer and their tagline was uh it sucks less than capistrano i remember a few years ago when rbm first came out wayne from rvm this is hard to say rv when rbm came out wayne seguin from rvm kind of tongue twister yeah kind of freaked out a little bit you

know maybe it was due to some back and forth and and it was like you know he'd put all this work into rvm and then when rvm came out a lot of people were saying oh finally because rvm sucked so bad as if like you know like rvm was just it came with ruby and and it was no no it was like robots that built it and no time was put into it. So he got, you know, real upset about that. And there was just some drama that went back and forth when Vlad came out. Sounds like it was kind of the same for you. Did you feel the same way at the time that Wayne did? Maybe like, were there any thoughts in your head, like, ah, screw it. I'm done with Capistrano that, you know, people, whatever, forget it, move on.

Yeah. I mean, that's definitely, um, you know, that thought definitely runs through your head, even if it's only for a split second. There's actually a book that was given to me by a colleague way back when that actually happened called Where's My Cheese? Andrew, if you're out there, probably you remember it. He, well, the author of the book basically talks about how there's kind of two ways to approach the situation and whether it be your cheese, whatever that means in your life, and somebody comes along and moves it or takes it away from you. And you can either react by like upping your game, going and finding something new or just being bitter that somebody took what you had away from you. And I think you mean who moved my cheese, right? Is that the name of it? Yeah, I think you mean Who Moved My Cheese, right? Is that the name of it? Yeah, I think it's Who Moved My Cheese.

Right. It's definitely worth a read. I mean, it's like, what, 20 pages long or something. It's tiny. You absolutely have to read this book. Any person in life has to read this book. Yeah, for sure. And that's a kind of natural reaction is to be bitter that somebody is so disappointed with your tool that they wrote their own.

And then you say, wait, hold on, I'm a hacker. That's pretty much what I do every day, right? I mean, that's why Capistrano exists because whatever bash scripting sucked or Capistrano exists because there was no passenger mod rails back then. Or, you know, there's a bunch of stuff that we replace every day you know people switch to redis because postgresql isn't good enough on key key value store stuff or whatever and it's not a winner takes all game but you do have to show respect for the for the other people in your space i mean it's kind of a big battle It's not even a battle because they solve different use cases. But between Chef and Puppet, you know, DevOps guys love to argue about which is better.

But the point is they both suck in their own special ways and they're both amazing. So it's a really tough one. I do have a particular bone to pick with any project that calls another project out by name and says it sucks less than that other thing. Yeah. But on the other hand – It's the only way to do that. Yeah, exactly. And it's a lack of respect, and that's what really sucks because the guy who's writing the new one has no expectations.

He can write whatever he wants. And in the position of kind of being in the dominant space, you're dealing with people all day, every day, who are using this stuff, who rely on it, companies who are built around it, in the case of early Engine Yard and Capistrano. And most of the work I do on Capistrano now is community support, mailing list support, kind of triaging bug stuff.

I don't write code anymore, which is okay because the project's moving in the right direction and I have some great contributors now. And that's fine. And speaking about RVM, actually, I have to mention it was announced today. I don't know when it was exactly, but it was on Hacker News today that Engine Yard are discontinuing their sponsorship of RVM. And because of that, or at least related to that, the maintainer of RVM is using, I don't know which fundraising self-starting platform it is, but he's trying to raise money to rewrite RVM. And if you use RVM, if you had benefit out of it, then you should definitely support Ed and give him the money. That is new news. Didn't they just drop support for Rubinius? I was thinking that's a couple weeks old. That is news. That's RVM. Michael, a piece was recently shot. That's a bummer.

Michael, if you're listening, we'll do whatever we can to help you out, man. Yeah. Well, you – yeah, I mean, I don't know. I look at something like Vlad and Capistrano specifically. I mean, what did deployment look like before Capistrano, right? I mean, you kind of mentioned it. Like, it sucked to deploy Rails applications early on. And so Capistrano comes out, solves the problem with kind of no map, right? I mean there's just – you're kind of in a cave by yourself trying to figure out how to solve this problem that like everybody is having but nobody has a solution to. So sure, you have your shortcomings and failures and over the years you can kind of learn like – like you said, you get stuck almost. Like this is not the ideal way to solve this problem but so many people are relying on this that we can't just up and pull the rug out from under them and change everything.

So somebody new comes along, and they're like, oh, hey, yeah, like you said, we have no expectations. We can do whatever we want, and we have a map now. We see the solution. We see the problems and the successes, so we can bring in the successes, solve the problems without anyone blaming us and then say, hey, look how much better we are than them. And to me, I think, like you said, that's a lack of respect. But more than that, it's a lack of like even understanding that without the person that you're saying that they suck, without that person, you would either not be able to solve the problem at all, or you would have the exact same growing pains and problems that they had in the first place. Yeah, right. I mean, at least the point is that you have a

solution that sucks, but at least you have a solution, right? And so anyway, it's a whole bigger issue about kind of how people, I think it's a question about quality that we expect from open source, which is just incredible now. I think we have higher demands from open source than we do of commercial software because open source is so good for the most part. And it's something about expectation management and just understanding what's reasonable. As I said, I think the biggest magic about Capistrano isn't the code. It's not the way it's implemented or the cool stuff I did with threading or anything else. It's purely to say, look, this is pretty much, if you're a contractor, if you're a Rails developer,

if something happens and you're the guy who's on call, you probably have at least a clue how it's deployed. Basically, I mean, it's going to be different side to side, but at least you have an idea what crazy something somebody might have done. I mean, there's still going to be those times you log in and somebody left Unicorn running in a Tmux session and you have no idea how to restart stuff, but that's not the norm anymore. And I think five or six years ago, that was the norm. And I'm glad that we're away from that. So I'm really excited about the stuff that's coming out about containerization. And specifically, Docker is amazing. But I think there's always going to be, you know, those pieces of software are going to fill the same void that Heroku is filling. There's going to be probably tens of thousands of people that say,

hooray, containers, now we don't have to use Capistrano anymore. And then they'll remember that they use a Mac, and like C and Docker only runs on Linux. Then they're going to need something to script their build server. So, you know, things will improve. There's always something and right exactly and um that's kind of the pitch with with version three of capistrata is to say look we we basically know how to do this now it's a really tiny tool use it if it works for you don't use it if it doesn't work for you uh and there's much less to learn i mean we've

kind of shrunk the footprint of the whole project and tried to make it this thing that you could just pick up and use without the steep learning curve because it's not going to be the norm for that much longer. Yeah. So I want it, we don't have a ton of time left. I want to kind of get into your specific, uh, burnout. And then we also maybe talk a little bit about harrow um you wrote in the capistrano mailing list a few weeks ago uh about just kind of some displeasure and and and just feeling overwhelmed and honestly just

burnt out which i think i mean you've been doing capistrano, for what would you say about six years now? So, um, I mean, sheesh, I feel like you had a burned out much sooner than that. So, uh, props for holding on for dear life that long, but, um, you put, you posted some stuff, you know, we don't need to get into all the negatives, but you posted some stuff that you just said, basically like Capistrano is a great tool. General purpose. Well, Ruby is a great language. I love the environment, but here are some things that have been troublesome, and one of the big ones was what we talked about before with the issues. But I want to get into one of the comments on that thread specifically,

and I won't mention him, but he basically said, Lee, I feel terrible for never having posted so much as a word of thanks for your effort dedicated to Capistrano over the years. And is that not it in a nutshell? Like, you only hear the negative, right? And these people, and at the end of his post he said, I want you to know that Capistrano has been a cornerstone of the trade that has paid my bills for the last six or seven years.

So everything's worked for him. He's used it. It's worked. He's been pleased with it. Probably if things haven't worked, he's figured out a way around it, figured out how to solve it himself. Overall positive experience. Well, of course, you're not going to hear from this guy. Unfortunately, you'll hear from the person that has a problem getting set up in the first place or has a problem, like you said,

with using, I don't know, using some weird build of Ruby that he got from somewhere else. You'll hear that stuff, the negative. So what do you think, do you think it could have, it could be different for you if, if you would hear kind of both sides of it, like the people that were thrilled about what, what you're doing and very happy with everything and kind of the positive reinforcement rather than only hearing the negative side? Yeah, definitely. And that was, so when I wrote the mailing list post, there was a real chance I was just going to kind of the positive reinforcement rather than only hearing the negative side? Yeah, definitely. And that was, so when I wrote the mailing list post, there was a real chance I was just going to kind of do like why the lucky stiff did a few years ago, just delete everything off GitHub, let the community.

410 gone, yeah. Yeah, right, exactly. Let the community pick up the pieces. They did a great job last time. And somebody would have stepped up to fill the void. And who knows, maybe they'd do a much better job that I was doing. But most of the reason I stayed around was I took a couple of days after that mailing list post and just tried not to do anything and came back to about 120 emails from people

exactly like the quote you just read out. People saying, you know, I'm a deployment consultant. I go around, I earn my living helping people with Capistrano recipes. I don't know, maybe that's a bad thing because it's so broken. But, you know, nice people telling me that they never even considered that it was just, for the most part, one guy with a couple of people helping out from time to time building this tool which is the kind of cornerstone of pretty much every rails deployment in the world i would guess i mean heroku changed that a bit but at least i think we're approaching something like six million downloads that gets skewed a little bit by bundler i don't know if it's really up or down or whatever else but i mean that's that's a lot of people and i would say

up until six months ago i was basically alone 10 months of the year working on this and then there would be someone around for a few weeks helping with a couple of specific issues there were some great contributions for rails from um from nathan Broadbent, specifically around the asset pipeline stuff when that came out for Rails 3. But there's been nobody kind of by my side helping the whole time. And I've been really lucky that

through building Capistrano 3, I've been connected with Tom Clements, who I think he's CTO or lead developer on the beach in the UK. And they use Capistrano for everything. And they always have done. And as soon as he knew that I needed some help, he's now stepped up. And he's basically written kind of 30% or 40% of the new version of Capistrano, which is incredible.

There's also another guy called Ker Shartov. I'm not very good with Russian names, but I hope I pronounced that correctly. And he stepped up and wrote all the integrations for Bundler and RVM and every other weird thing you've ever seen. And he's doing an amazing job of support. So knowing those two guys are there, knowing that I can look in my inbox, see 25 issue notifications from GitHub and ignore all of them because they'll be taken care of. That's an amazing feeling. And I've never had that over the last few years. So you reached out to the community, kind of like almost a distress call, like, hey, I'm drowning here. I think your exact words were, I'm coming apart at the seams. And the response you got was overwhelmingly positive. So you were thinking maybe 410 gone,

but through the response you got you're still here you're still working on this and and sounds like you've gotten some reinvigoration to to keep moving forward and some help to kind of lift you up because i mean we all know like you can't do any of this by yourself for for so long i mean again i'm shocked that you've been doing this by yourself for as long as you have and you're still here at all um but i don't know that that's, that's a open source win. Maybe the lesson learned is to not wait until you're literally coming apart at the seams, or I guess figuratively coming apart at the seams. And a little bit early on in the process,

maybe could have benefited you to reach out and kind of maybe share some of those sentiments earlier, maybe? Yeah, definitely. And I think that's, it's some kind of maybe share some of those sentiments earlier, maybe? Yeah, definitely. And I think that's, it's some kind of stigma. We live in a world where the only people we hear about are the super famous Mark Zuckerberg and the kind of success stories from 37 Signals. And we don't really spare much thought for the guys who put in an incredible amount of work and never get a thank you for anything and that's something that i've changed since my burnout is whenever i'm opening an issue and i'm writing some stuff with the go language right now and i've actually run into three bugs in the go standard library and you know this is written by rob pike and he's programming god and all of these brad fitzpatrick

and the other guys they're amazing amazing. And I kind of don't believe that I've really found bugs. So whenever I'm approaching those guys, I'm like, guys, I'm pretty sure I'm just holding it wrong. But maybe there's maybe a bug. And it really affects the way that I'm communicating with the maintainers of the projects that I rely on. And I'd like to encourage a culture of people asking for help, being open about being real human beings and not trying to provide this image of being the genius programmer who ships everything perfect every time, because it's just not realistic. Yeah. If you, if you talk to every, you know, therapist, psychologist out there,

pretty much everything in life comes down to pride versus humility. And it sounds like – not sounds like, but the truth is a lot of us developers could use a lot more humility and sacrifice a lot more of the pride just for your – not for the – not just for the sake of the person I'm speaking to, but for yourself as well. I mean, you know, if I'm willing to be honest with people and talk about my struggles and development and the things that I don't know, then guess what? I have an opportunity to learn. And if I wear my pride shield proud and I know everything, then guess what? I'm going to fake it and I'm not going to learn.

And I think that, you know, for all of us, we could approach these situations with some humility to really learn and then, you know, potentially make everyone's life better. And I think you said a good point, like, maybe it would, it would be helpful for all the developers when you're opening issues in open source to, to read the contribution guidelines, to read the issue guidelines, to, you know, to, to lead with a positive, right? You're thankful that, that this person is even going to look at your issue. I mean, we rewrite with like the expectation of like, Hey, uh, the reason, like you're, you're going to fix this because it's your fault and you screwed this up. So fix it. And it's like, maybe if we approached it more from a, like, Hey, everything you're

doing is incredible. Um, here's something that's happening. I'm not sure if maybe I'm doing something stupid or, you know, maybe it's lack of understanding, but anything I can do to help you out, let me know. I mean, I feel like if that was the general mindset with issues, it probably would solve a lot of the burnout problem. Yeah, definitely. And I've been, well, actually kind of stupid. I ran into a couple of bugs as well with Go Language,

with the Postgres driver, and I've been communicating with those guys about two specific issues, and I ran ran into a third issue and I've been working with them for basically a week trying to figure out what we were, what was, what was going wrong. And we couldn't reproduce it and we couldn't reproduce it. We couldn't reproduce it. And it turned out to be some weird deadlocking issue because I had a recursive Jason marshalling thing, which was a really stupid typo on my part. And I mailed those guys and said, look, this is really embarrassing, but I completely screwed up.

And those guys were cool about it. They said, hey, look, great. Well, thanks for the bug reports on the other two things. We're going to take care of those sometime. And, you know, hey, at least we know we didn't screw up. It's your fault.

Yeah, yeah. I like that. And I think I'm very lucky to have had the good dialogue with those guys. But I think the assumption that open source software is perfect and therefore it should work is kind of unrealistic. But we've gone in that direction because everything has gotten to be so good. And it's upped the game for the rest of us. It's great that we have to up our games. It's great that guys like Tom and Kerr and I, we have to write great software.

It pushes us, it makes us grow, but it also puts a lot of pressure on us. Right. Well, we're running out of time. Real quick, give us the pitch for Harrow. How do you say that, and what is it, and what we should be looking for? Yeah. So this is based, well, based on the fact that I didn't write Capistrano originally,

I've never really felt like I earned the right to commercialize it or find a way to commercialize it. And a lot of the feedback I had after the burnout was, you know, find a way, find a way to make this sustainable, find a way to work on it full time. And an idea that's been brewing in my mind for a long time is something some people may remember Webistrano. It's a very old, completely obsolete now version of Capistrano that you could run on the web. So it didn't integrate with your app or anything, but it was a website you could host somewhere, click a button, and Capistrano would do its thing. And so I've been working on this together with Tom, and Harrow is going to be a hosted Capistrano for Teams, but also will provide build environments. So you can think of it somewhere between Jenkins or Travis CI, where it will have an up-to-date list of whatever's going on in your

repository. You'll be able to see your branches, your staging environments. It will help managing SSH keys for your users and your team members, or maybe some contractors you have. It will also take care of all the audit logging and everything else. And you'll be able to schedule deploys. There's some really cool stuff about kind of team SSH debugging sessions right in the browser over WebSockets. It's an attempt at making everything that is cool about GitHub, Travis and Jenkins, all of that online collaborative stuff, bringing it to deployment. And firmly in the knowledge that this Capistrano style deployment eventually will be replaced by containerization harrow will run any rake task any build script you have so it will also be perfect for people who want to script the building of debian files or rpms or even scripting containers or whatever else and i think that's

important i want to keep it very affordable and very cheap. I'm probably going to follow the Travis model where you get kind of a crazy number of stuff for free up front and probably most people never see a bill. But for any companies who are getting value out of it, then I want it to be worth it and I want it to make Capistrano sustainable. And it's a kind of promise from my side of the community that Capistrano will always remain kind of priority one MIT licensed open source and will never be proprietary. Of course, I couldn't make it that way, even if I wanted to. But I'm really hoping that Harrow will be successful. We have about 500 people signed up for the alpha already, which is kind of overwhelming.

But I'm excited. It's working pretty well for us and amazing timing is that we were having real hard times with the kind of rpc for repository access checking and webhooks and everything we have to be really careful with people's ssh keys and security tokens so we were doing everything in sandboxed virtual machines and well frankly that sucked um Sorry to everybody who's involved with VirtualBox. But then Docker came along and Docker has made everything amazing. So that's another classic open source win story. Awesome. So Harrow, is it, how do you pronounce it, Harrow or Harrow? It's Harrow. I think there's a school in England called Harrow, a private school,

but it's also a farming implement. It's the thing you use after you plow the fields to get it ready for planting. Plus, it sounds cool. So you've plowed the fields with Capistrano and you're ready for planting. Right, something like that. Well, you're going to be harvesting the money out of their wallets. That's not what's written in the business plan, but yeah. Yeah.

So the website is harrow, H-A-R-R-O-W dot I-O. You can sign up for the alpha and keep an eye on it. It's going to be pretty cool, I think. Yep. And we're totally open to feedback. I mean, it's pretty early. We have all of the structure in place, but if people want to tell us how they want to do this stuff,

then we're really open to making the workflows really flexible. Awesome. So once again, that's h-a-r-r-o-w dot i-o. So we're running out of time here for our new listeners. We ask the same questions at the end of every show. So for the sake of brevity, I won't explain it. And let's ask them now, Lee, for a call to arms, something you'd like to see the community kind of help out with, with Capistrano. Yeah. So first of all, on a wider topic, just respect your open source guys, thank them from time to time, fewer plus ones and GitHub issues. And when it comes to Capistrano, just tell us what doesn't work for you, but try and give us enough information to help you.

That would be my number one request. Gotcha. So be precise, give context with your issues when you open them. Definitely, because very often we can close them straight away and tell you exactly what you did wrong, and otherwise it can take a few days bouncing back and forward to find out that, you know, whatever, you forgot to put a gem in your gem file so right cool um if you weren't doing this i don't know what this is but maybe hero capistrano uh maybe some of your consulting what would you be doing instead oh i don't know i think if it wasn't for open source i probably

would have thrown my life away and i'd be like sitting on a couch somewhere with no job playing video games. So I, in other words, living the dream. Yeah. Right. I mean, I owe my entire career to open source, um, through getting involved at a young age, I have no formal university education or something. So yeah, without open source in general, I don't know what I would do, which is probably why I'm still doing open source. Cool. Uh, programmer hero, somebody that's influenced you greatly. Uh, James Buck, definitely. I know that's kind of soppy to say that because I took over his project, but, uh, yeah, right. Uh, took over and it's all mine now, but,. But I think he got out at the right time. He got out for the right reasons. I think when I look back at the situation he must have been in and the way he must have been feeling, he made the right decision against what's a very strong pull from the community to stay and stay responsible and stay answerable to the community. And he was very bold and standing up against that and going prioritizing his family and his health, which I think is important. Other than that, every guy who ever wrote any line of software I ever used, because again,

we wouldn't be doing what we're doing without you. And it's amazing. It's amazing that we can build the stuff we build on top of the shoulders of giants on hundreds of millions of lines of code in countless languages and it's incredible that it all works together you mentioned the the health part with with jamis and you know that's that's much respect too because those are seriously hard decisions when you absolutely love your craft and it doesn't mean that you don't love your family equally but it's you know it's like that's your passion it's kind of your art you know it's you know coding isn't and developing isn't just hacking it's you know

it's an art form right and to to give that up so he must love his family tons which is super awesome and very respectable too, right? Yeah. And I think for a lot of people, especially myself, I mean, I count myself first as programmer, hacker, engineer. And to admit publicly that I'm struggling to hold everything together with the thing by which I define myself, that's tricky. And actually doing it was the best thing I ever did. And the sense of kind of weight off your shoulders is incredible. And I think we all need to, you know, we're human beings. We're sons and daughters and husbands and boyfriends and girlfriends first.

And the stuff we do open source, it's all digital. It's all ephemeral. If we just cease to exist one day, it would continue or not, and the world would go on. And it's important to keep the perspective. Yeah, I'm glad you mentioned that too because we're going to link off of that in the show notes so anyone can kind of catch feel the angst that you needed to get it out, but it was painful. So I can only imagine the deliberation you must have been going through when you were writing this. I might just drop this project and what that means for you and the community and what kind of feedback you might get. And I mean obviously you came on the show to talk about some of the details around that, so we won't go back into it. But that will be linked out in the show notes. But Lee, I want to thank you for coming on the show to talk about some of the the details around that so we won't go back into it but that will be linked out in the in the show notes but lee i want to thank you for for coming on the show man we really appreciate you just um sticking in there taking the time to come on the show and

share your story and your history and and uh lift james up who's a developer programming hero for you and just um all that you do and sharing. I know sometimes it's not always the easiest thing to do, but the fact that you get up and do it every day and that you love it is awesome, and we certainly appreciate you allowing us to stand up on your shoulders and the code that you've helped lead get written. So definitely want to thank you for joining us. And, Andrew, awesome show, bro. Thank you so much for doing this.

And you, the listeners, for listening to this show. We absolutely could not do this without you. And to mention a couple others, DigitalOcean and TopTile. They're supporting the show, making it possible with DigitalOcean. I want to remind you to take advantage of that $10 hosting credit. And to do so, you can use the code THECHANGELOGOCTOBER. That's THECHANGELOGOCTOBER. I know it's about to be november so if you're listening to this in november it's probably going to still work and if it doesn't hit up support so don't worry about that they do send

stickers around the world so no matter where you're at whether you're in australia if you're in europe or you're here in the US email Barry at DigitalOcean.com he'll ship you some stickers to decorate your laptop and thanks to TopTow for being an awesome new sponsor and they will be sponsoring the next few shows so certainly appreciate their support join the

TopTow worldwide network and work with some awesome people anywhere in the world beach wherever right Andrew I mean beach is kind of the best place to be but TopTow.com slash developer to apply and anywhere in the world, beach, wherever, right, Andrew? I mean, beach is kind of the best place to be, but toptow.com slash developer to apply. And we mentioned

their engineering blog. If you haven't caught up with this yet, go check it out. It's toptow.com slash blog. We were recently, we recently featured them on the changelog.

And I think in the last week's weekly, we had a spot in there for them too. So if you're a subscriber to weekly, we thank you for that as well. And if you're not, go to changelog and i think in the last week's weekly we had a spot in there for them too so if you're a subscriber to weekly uh we thank you for that as well and if you're not go to changelog the changelog.com weekly and subscribe today but awesome show guys let's say goodbye see you later goodbye Goodbye. you