The Changelog: Software Development, Open Source - Inspeqtor and OSS Products (Interview)

Episode Date: November 11, 2014

Adam and Jerod talk with Mike Perham about his new project Inspeqtor and his approach to better application infrastructure monitoring.

Welcome back everyone. This is the changelog and I'm your host Adams Dekovic. This is episode 130. Jared and I have talked to Mike Parham. He's back again, just like Sidekick. He's got Inspector now, another open source slash pro version of software out there for you to use. This one is application infrastructure monitoring reimagined. It's called Inspector.

Now, let me talk about that. We also talked about other fun ways he's making money as an open source developer. So, great conversation there. We've got some awesome sponsors for this show, Codeship, Hired.com, and DigitalOcean. We'll tell you a bit more about Hired.com and DigitalOcean later in the show, but our friends at CodeShip is a hosted continuous deployment service that just works. You can easily set up continuous integration for your application in just a few steps and automatically deploy all your code when your tests pass.

CodeShip has great support for lots of languages, test frameworks, as well as notification services. They integrate with GitHub and Bitbucket and can deploy your code to cloud services like Heroku, AWS, Nogitsu, Google App Engine, or even your own servers. The setup takes just three minutes. Get started today with their free plan and make sure you use our code, the changelogpodcast. Again, that code is the changelogpodcast, and you will get a 20% discount for three months on any plane you choose. Head to codeship.io and now onto the show.

Welcome back, everyone. We're joined here with a guest that's no stranger to the show, Mike Parham. And Mike, you're also not a stranger to the changelog. We've covered all sorts of stuff that you've done over the years, everything from Dolly to Sidekick to Launchy. Launchy, Lunchy, I'm not sure if I... It's Launchy. Launchy.

I think we pronounced... It's spelled Lunchy, though, right? Yeah, it is. Yeah. There's already a gym called Launchy, so I couldn't squat on that name. Right. So I'm here.

Mike's here. Jared's here. Jared, say hello. Hey. We've been waiting to, I think, maybe about a month or two to have this conversation. You've been working on something brand new. So maybe the best way to start would be to just tell us what's going on, what you do brand new. Sure.

For the last four months, I've been working on a top-secret project, which I just announced yesterday. And it's called Inspector. It's my fresh take on application infrastructure monitoring. That is to say, all the moving parts of a server-side application. A little tool which allows you to monitor all those moving parts to make sure everything is healthy. And if anything looks out of the ordinary, Inspector will immediately send out alerts to wherever you have configured to say, hey, this doesn't look right.

So it's really helpful in terms of just, you don't have to watch a dashboard all day or something like that. You just watch your inbox, and if something shows up, then you know to investigate. So Inspector comes into a space that has some offerings. I think you're no stranger to that at Sidekick, of course. Famously was Rescue done right or done better. Inspector has competition with Monit, with God, with – what's the Python one? Supervisor D? Yeah, so there's a few others.

Blue Pill was the one I was thinking of. So there's other players in this space. What's Inspector's unique take, and what's the value add there? Right. So if you know anything about Monit, it's pretty obvious that Monit is the main influence in the way Inspector works. I've used Monit for the last five years,

and I've just always been frustrated with some of its quirks. And so I really wanted to build, for the last few years, I really wanted to build something that was like sort of a Monit++ or my take on Monit. And so as I really got serious about doing this a few months ago, I really started looking at the monet feature set and just sort of what did I want in terms of, you know, my previous job was director of infrastructure, director of technical operations, whatever you want to call it. But basically if the site had problems, I was the guy whose head was on the line.

So this kind of tool was critical for my job. I had to know when something was acting up. So yeah, I went through Monit's feature set and pared down all the different features that I thought were not useful from an application monitoring point of view and added new features that I thought were not useful from an application monitoring point of view and added new features that I thought were critical that Monit didn't have. And so that's what became Inspector.

Inspector makes some interesting choices in terms of its design that some people aren't going to necessarily like. For instance? Well, it doesn't monitor initd, legacy initd services. Really? And that's by choice. Part of application monitoring is I'm trying to guide people on how to build a reliable application.

Inspector is going to tell you if something is not not working reliably but it also wants to guide you to build better applications and part of that is using a proper modern init system like upstart like systemd like run it and so uh initd one of the problems with init net D is there's no sort of central demon that you can query for the status of a process or a service. And everything's kind of rolled into this just big, hairy bash script in Etsy and NetD. And so when I realized this a couple of months ago, I realized that this was a sort of a decision point in the design here that was going to be fundamental to the way Inspector works. Do I want to support this legacy ball of mud, or do I want to make a hard stop here and ask people, hey, or guide people or try to educate people about why this is a problem? And, and it's not that hard to learn the new,

the new systems and, and thereby get a more reliable setup for your application components. So that's, that's kind of what I did. I was going to say the, so just to kind of give the listeners who listen to this to kind of a, maybe an outline of what we think this show might be about this time uh in traditional fashion of the changelog we want to go deep we want to figure out all the details of inspector but i think a neat part of this transition for you which you had

said this you didn't want to um you had to make a choice of uh supporting that ball of what was your word mud um so you had to make this choice of doing that or something else. And episode 92, we had you on talking about Sidekick. You had some success with Sidekick Pro. So for the listener's sake, what we want to do is we want to talk deeply, obviously, about the insides and the innards, so to speak, of Inspector and what you did there, but specifically you've seemed to hit this, the nail on the head, so to speak, of success when it comes to delivering open source the right way, but also making a living because you've got your wife, you've got a beautiful little boy, a furry cat to take care of. It's not just Mike, everybody. Mike's Mike, but he's got family,

so you got to make money. And you found this really cool way to do this. So can you take us through some of the journey to kind of get to making the choice of supporting that ball of mud or not? Sure. Yeah. I mean, at this point in my career, I love writing open source, but I'm also I also had to make the decision that I'm not a charity, that I'm an experienced engineer. And part of that experience is that I write. Ideally, I have the experience to make well-designed software, reliable software. So I have to – I've decided to have a business model where I have both an open source product and a commercial product on top of that, that open source core. And that commercial product is closed source.

And, you know, of course, there's some people that really don't like this model, but I think by and large, everybody understands that people have to make a living, and I can either work for a corporation that is paying me a salary to work on full-time open source, or I can do it on my own. There's a lot of different business models you can have, of course. Another popular one is to have services. So you do consulting for your product. I think the Sensu guys do that.

They do some – Sensu, I think, is some sort of monitoring system. It's a very complex monitoring system built on top of RabbitMQ, if I recall. Similar to React, too. They do the same thing where they have a commercial-based version and they have support. It's a model that is kind of it has its pros and cons too. Yeah, exactly.

Flexibility maybe would be one of them where it's a corporation, multiple people, and again, you seem like you're not a lone rogue agent, but you're, you know, you like doing things the Mike way. Well, and I've, I've specifically tried to avoid the type of business where I would need to build this giant thing and have dozens of employees and take on venture funding and all this kind of stuff. I'm more focused on smaller focused tools like Sidekick, like Inspector,

that I can create a commercial product for. I can support it as a single person and make enough money to provide for my family. So yeah, Contributed Systems is one person, bootstraps, no funding at all. That's awesome. Yeah, for sure. So when you first made this decision back with Sidekick, was it at conception that you said, I'm going to have Sidekick and Sidekick Pro, or did you start off saying, I'm going to do a threaded version of Rescue,

and then it got popular, and then you thought, oh, I could turn this into a living? When I first started out, it was more of just a vague notion of, well, I'm starting my 10th open source project here, and I'm going to work on this for another year, and there go my nights and weekends. So how can I actually make some money for this so that i've justified to my wife um so that's why i initially uh just had sidekick it was it was just the open source product i actually sold commercial licenses for sidekick and that did not bring in a ton of money. It brought in like $1,500 or something like that over the course of six months. But it didn't bring in nearly enough to justify my time. When you took all the hours that I was spending on Sidekick,

I was making minimum wage in terms of selling licenses. Right, hours to dollars, yeah. Right, so that's when I said, okay, I've got this Sidekick thing. It's moderately successful at this point. We're six months in. Why don't I do a commercial product on top of it and sell that for ten times as much money? And to see if I can get people not just to pay because their lawyers tell them, but because they want to buy actual useful functionality.

And so that's when I started, I spent about a month building a sidekick pro and, uh, and then started selling it. And, you know, it, it, it ramped up slowly, but surely, uh, when I, when I first threw it out there and announced it, I had no idea if anybody would buy this thing. You know, it's a Ruby gem. People are used to just saying gem install and not having to put in a credit card. But sales were slow at first, but they've ramped up to the point now where I've got a good income that provides for my family just based on those sales alone. So I want to touch on, before we move on, Jared, I want to touch on one thing that the listeners might be thinking about. In episode 92, we talked to you about this very topic here, but one question that came

up that you don't have to go back into, but I just want to at least touch on it quickly, which is, you know, what's stopping somebody from, since it's open source, taking Sidekick Pro features and putting them in the open source version? That's sort of a hurdle you had to get over. How do you prevent that? Just a quick note on that for this show. I don't prevent it, and I it in their own version of Sidekiq, there's legally nothing stopping them from doing that. Sidekiq itself is LGPL. They can fork it, and they can add a feature to it as long as that feature remains open source in their fork.

They can do whatever they want with it. The thing that people are paying for is long-term support. They're paying for a roadmap. They're paying to know that someone is constantly going to be ensuring that Rails 4.2 is going to work with it, that Rails 5 is going to work with it, that Ruby 2.2 is going to work with it, that they're also paying for taste, that I, as a project dictator, I have the good taste to know which features are good, which features will add instability to the product, you know, that sort of thing.

So they're paying for the experience and the oversight of the project to continue. So, yeah, there's nothing legally stopping people from doing that. You know, just the same thing with Inspector. If people want to fork Inspector, it's GPL. They can add their own feature, which copies it. But, again, I think I'm here for the long run. I've got a product which is paying me to support this for the next few years. If they just fork it and add a feature,

are they going to maintain it for the next two years? Right. That's a good point. I'm constantly going to be adding new features to the open source and the commercial version. Are they going to be constantly pulling in those upstream changes? Businesses don't want to deal with that hassle. They just want to buy something that they know will work and will be there for the next, you know, in years that they can count on. Well, even the dev UX too, like, you know, a fellow developer who would use your open source version, but once their business, you know, they might be fine with using, let's say sidekick open source on their personal projects,

but for their, you know, work they do at their day job or whatever they're doing, they want something that has that support. So they might use the pro version at work. So your customers are still like me and Jared, you know, and the listeners of the show, but they just happen to work somewhere else. And they, and you're right though. I think that was a really good point of, I don't think you said it like you did in 92. So maybe you perfected your language around it. Cause that sounded so much better than, well, not so much better, like in a bad way, but

like it sounded really good. It was a good point to make that, you know, you're, they're paying for the roadmap. They're paying for your taste and they're paying for your taste, and they're paying for this support along the way, not just like day-to-day support like helping with an issue, but like that Rails 4.2 is going to work and other versions and legacy and that kind of stuff. So you weren't sure if people were going to buy this though, but recently you released a post where you said, uh, some of your numbers, which was quite gracious, uh, exposing those. I know that's kind of a private thing for a lot of people, but I think in the post you say why, and I think it's super helpful for us to see that sidekick pro sales, what you said for the last three months of 2012 were, uh, 7,500 bucks in 2013, they totaled 85 grand. And this year sales should top 175,000. In 2013, they totaled $85,000. And this year, sales should top $175,000.

Those are pretty good numbers. Yes. Congratulations on that. Especially since they are a subscription. It's no longer a one-time fee. Right. You charge $750 a year for the Sidekick Pro? Yeah.

So ideally, that's essentially what is paying for Inspector. $150 a year for the Sidekick Pro? Yeah, so ideally that is a, I mean, that's essentially what is paying for Inspector, right? Is that reoccurring income that I know is going to be there so that I can do things like work for four months on a brand new product that I have no idea if anybody's going to buy. Yeah, how much time do you have to continue to work on Sidekick? Sidekick generally takes 10 to 20 hours a week right now So significant It's significant, I'm answering a lot of emails

People still put in issues all the time Although typically 90% of those issues Are some sort of application issue And then I'm constantly on Stack Overflow. If someone posts a Sidekick tagged question, I'm usually answering it within 24 hours. So yeah, it's 90% support at this point. I did just add a feature to Sidekick Pro,

which I'm going to be rolling out in the next version. So I am still doing a little bit of feature work, but for the most part, it's mostly support at this point. So you're pretty happy with your sales. Interesting that you decided then to say, okay, I'm going to start this new thing, same model. I mean, that makes a lot of sense. But, you know, at your current rates,

if we're doing our math right, maybe that's 200, 250 customers. You know, perhaps you could just focus on maybe that's 200, 250 customers. Perhaps you could just focus on turning that into 1,000 customers. Focus on Sidekick Pro, which is obviously a winner as far as being viable in the market. What made you decide, I'm going to add a second product? Diversification. It's that ball of mud.

You wanted to fix the problem. Think about finances and risk. You always talk about diversifying your investments. Right. You have to diversify your time and your investments. What I've done over the last two or three years is invest a lot in the Ruby community

and invest a lot in Sidekick. However, if you take a step back and look at the general tech world, Ruby is 2%, 3% of the tech world. If you want a wider customer base, you've got to go with a more generic product. And that's exactly what Inspector is. Inspector is useful to anybody using Linux.

It doesn't care if you've got a Python app, a PHP app, you know, a Haskell app if you're a neckbeard. It's diversification in the sense that if something better than Sidekiq comes along tomorrow, then uh-oh, what am I going to do? Well, now I've got two different products, which have slightly overlapping audiences, but the Venn diagram is still significantly different.

That is, there's a huge new open territory for me to find customers in now. This is a prime place, too, because, I mean, new open territory for me to, to find customers in now. This is a prime place too, because it's, I mean, you'd said earlier in the show that this is inspired by to a degree from Monit, you know, so there's some inspiration there. You also talked about the ball of mud that you got sick of dealing with. So obviously there's something, some, some competitors in the space that weren't cutting the, you know, cutting the cheese, so to speak. I don't know if that's the right way to say that or not.

Cutting the mustard. Cutting the mustard. Cutting the mustard. There you go. My bad. My bad, y'all. It's probably my Texas ways or something, just seeing the off-color thing. Cutting the brisket.

Yeah. There you go. Cutting the brisket. That's a better way to say it for Texas style. So, I mean, obviously there's something happening there and you, like you said earlier, they're paying for taste. So you have taste and why not do it better? And in fact, a lot of the Linux lower level open source monitoring tools are, they're either a decade old, so they've got a lot of accumulated cruft or they're just,

I don't know. I mean, I hate to use the term over and over. I don't know if it's pejorative or not, but they're very neck beard oriented. They're just not easy to use. They're very unfriendly. They're very tech heavy. Um, you know, I was looking, someone brought up, uh, collected as a possible competitor and I looked at collected and, and its syntax is just terrible. It's just hard to use. It's ugly. Yeah, I mean, I don't know if Linux people care about this stuff, but I do. And I want things to be easy to use, so I do my utmost to pare things down to the bare minimum to get sort of a zen, you the KISS principle in action, keep it as simple as humanly possible.

And so that's why when you look at inspectors' configuration files, they look stupid simple. But they're the bare minimum I needed to achieve what I wanted to achieve. And, you know, I hope people appreciate that. Let's pause the show for a minute. Give a shout out to a sponsor, DigitalOcean, simple cloud hosting built for developers.

In 55 seconds, you'll have a cloud server with full root access, and it just doesn't get any easier than that. Pricing plan started only five bucks a month for half a gram, 20 gigs of ssd drive space one cpu and one terabyte of transfer that's a lot for five bucks a month digital ocean also has data centers all across the world new york san francisco amsterdam singapore and their newest region london you can easily migrate your data between those regions,

making your data always closest to your users. Use the promo code changelognovember in lowercase. It's important that you use lowercase. changelognovember to get a $10 hosting credit when you sign up. Head to digitalocean.com right now to get started. And back to the show. So I see some patterns here um of you taking a thing and like you said inspector is kind of monet plus plus sidekick was kind of rescue plus plus in the same post where you gave your numbers which everybody should probably go out and

read that it's a great post we'll put in the show notes for sure so check out the show notes yeah in there you actually give these repeatable steps that you've taken. Obviously we can kind of see the pattern, but could you walk through the steps you took as far as how to come to a successful open source slash commercial product? And then perhaps as a follow-up, give examples of things that you haven't taken on that maybe somebody else could.

Step one is to find a tool that is non-trivial and important to your current system or workflow. That goes right back to if you write a tiny hundred line gem, nobody's going to care. It has to be non-trivial. It has to be something that will take a lot of time. It has to be something that is worthy of someone's consideration to use, to outsource, or to actually buy from you. So yeah, and ideally, like you say, you want to find something that is a bit painful to use. Maybe it's overly complex or has a lot of features that you don't want. Think about Microsoft Word and look at all the

text editors out there that are just, there's no competition for Word and Word's not competition for them. Word is just this giant set of features of which you use maybe 5% of those features. So is there a market for a Word processor that is just much, much simpler. And I would argue something like a Markdown editor is a perfect example where you take the essence of Word, which is writing a formatted document, but kind of twisting it so that it's much simpler, but is non-trivial to author, and is something that people would pay for. So it's sort of a different take on a word processor. So step two is plan out how you can make it better. Simplify it.

So you're taking Microsoft Word and you're going down and you're saying, I don't need all these toolbars. I don't need hundreds of ribbon commands or menu commands or whatever. People just want to write nicely formatted documents. So you're discarding that superfluous functionality and you're adding your own useful functionality. At this point, I like to think about how am I going to divide the functionality?

How am I going to make a business model out of this thing? You've got to draw a line where you tell customers here's what's available to you for free but if you want more, you've got to draw a line where you tell customers, here's what's available to you for free. But if you want more, you've got to pay for it. And I think, like I said, 90% of people understand that and that this is your full-time job. And so therefore, that's just a line you have to draw. And then you're going to build the thing and see what happens. You've got to evangelize it. You've got to market it.

You've got to support it. It's not just code. Open source is a process. Software is a process. It's not just a bunch of bits that you pound out in Vim over a couple weeks. So you've got to build that thing, and then you've got to support it and evangelize it over the course of months and see how that goes. And that was sort of the sidekick model.

And then once it takes off, you build the commercial version of it and start selling it to your open source user base. And there will be a small percentage of people that will upgrade. And so who knows how much money that's going to be. That could be beer money or that could be enough to make a living money. But that's where you need to start tweaking your pricing. You need to start tweaking the functionality. There's no right answer here, but you'll, you'll need to experiment. But those are the, those are the five steps. Okay. So a few examples, obviously you chose background jobs and then you chose monitoring.

Are there any other pain points that you see out there that, you know, you'd, you'd take on if you didn't have, you know, two successful projects you're already doing? When I was writing this blog post, I was actually thinking, oh, maybe I should give them an example of something that has room for exactly this business model. But you didn't. I didn't because I came up with one idea, but I wasn't sure. I wanted people to sort of think on their own about it. The one that I came up with was HTML to PDF conversion.

There's a ton of services out there that do that. And literally every single business wants this tool. And literally no open source people want it. So what that means is that you've got a very business friendly, very commercial friendly possible product. The one in the Ruby space that I'm familiar with is called Wicked PDF.

It's a gem that wraps the WebKit HTML to PDF binary. And it's old and crufty, and I'm not sure how well it's supported, but that's definitely a tool that if somebody was kind of more in the PDF space, maybe they knew WebKit better than I do. It's something that people might consider doing. It's like I said, every business I've worked at in the last five years has wanted to use that tool for some reason or another. And generally, they would have no problem paying, you know, 25 bucks a month or whatever for a tool which does that. And then you multiply that by a thousand businesses

that need it. Now you've got $25,000 a month in reoccurring income. Yeah. There's, um, there's a formula that goes 30, uh, 30 by 500. And I didn't, I didn't make that up. That's Amy Hoy's thing and Alex Hillman's thing. Um, but where if you can get 500 customers to give you 30 bucks a month you will have a business that makes roughly 150 000 a year you know so if you kind of break it down like you have to these achievable uh yet still hard you know it's not like it got any easier but they become more achievable once you break it down to these five steps you you've kind of given here um and i think your model has legs like obviously it's got legs because you got $125K in the bank that proves that it works. Well, yeah, exactly.

And the wonderful thing about software in general is that you can do it in your spare time. You know, you can do it in nights. You can do it in weekends as long as your employer is somewhat friendly to you, sort of moonlighting, as long as your employment contract doesn't have issues there. Of course, legally, you'll want to verify that. But you want to make sure you're in the clear long term. But, you know, I wrote Sidekick Pro and have been working full time for the climb for the last two years, right? And the last two and a half years, I was working on Sidekick and Sidekick Pro

and bringing in, I was bringing in $100,000 through Sidekick while also having a full-time job making six figures. So you can build all this stuff without much investment, you know, next to no investment aside from some time.

You know, time is a luxury to a lot of people. But if you've got that, you can invest that time into your own possible future career. Be patient, too. I mean, it seems like you've been sitting on some patience, honestly. You didn't seem like you were in a rush to jump ship. The moment you had success with Psychic Pro, basically basically you weren't like, Oh, I'm out. You know, it, so can you talk maybe a little bit about that? And I think maybe Jared's got some other questions. I don't want to stomp on your questions, Jared, but can you talk a bit about the, the patience aspect of, of what you've done?

Sure. Yeah. I mean, uh, having a nice, a nice salary makes it much harder to determine when am I going to jump ship here. You know, my salary is a nice, steady, flat stream of income. And then my Sidekick Pro income was constantly sort of slowly but surely rising up. And there was an inflection point where the two actually met. And I was making as much or more every month from psychic pro than I was my salary. That's when I started saying, okay, how long do I hold on here and continue to draw a salary before I just say, I'm going to do this full time. It worked out to about six months, about in January, about in January this year, when the two sort of met. And I said, well, you know, if this thing keeps going, there's no reason I need to be working

full time at all for somebody else when I can be doing my own thing. And so, you know, a couple months ago, the business came to me and said, we've got some opportunities here. And one of those opportunities was for me to leave with a very nice severance package. And I elected that. And in doing so, that severance package effectively subsidized the building of Inspector. So, yeah, it's worked out really well. And if you're patient, you can time this stuff so that it works out the best for you. So I think one of those other hard decisions, I mean, I'm looking at this as like a viable thing to possibly do. And I think another place where it's difficult, obviously, this is a business

decision is like, where do you actually draw the line in the sand for pro features versus the open source features? You've done this twice now, and you probably felt it out with Sidekick, and I think you may be a little more confident with Inspector. Can you speak generally, and then we'll get into Inspector details after that? Yeah, that's a common question. Almost literally the first question everybody asks me. There's no easy answer. What I've done with Inspector is tried to say, okay, what's a quote-unquote enterprise feature? What is a team feature? What I've tried to do with Inspector is make the open source functionality be the features that

an individual would want if they were a hobbyist and just sort of built their own server-side application without a team. Inspector Pro, on the other hand, has a bunch of functionality so that you can route alerts to different people. You can set owners of different components. So Bob owns the database, but Mike owns the background processing system, and Ted over here owns Nginx or Apache or the app server or whatever. And so that way, if any of these components misbehave, the alerts are routed to the team members that know them best. So that's one approach that I've taken is

again, what's an enterprise feature, what's a team type feature. And that's really all I have in terms of advice. It's not an easy question to answer. And it's just something that you have to judge for yourself. I've got about five or six different features that i want to add to inspector right now and it's really tough trying to figure out all right which of these should go into pro and be sort of locked away from the majority of users that's that's really that's a really painful decision to make because i want to give all the functionality to everybody but But I know that that's just not a viable solution.

Yeah, I'm curious how that affects your open source contributions when not just, okay, that, people actually getting involved in your open source projects, but then also the kinds of pull requests that you'll actually accept, and then you decide, wow, that's a great feature, thanks, I'm going to put it in my pro product. You can't do that, right? If somebody is submitting a PR to me,

I see that as their code, and when I pull it in, it's licensed to me, sort me based on the contribution guidelines, but I would not take somebody's code and then just make it a pro feature. That's immoral or unethical, in my opinion. Agreed. What if you're thinking about implementing that, and then somebody does it for you? I guess you just tell them, I'm going to build it myself?

Yeah, I mean, that's sort of the discussion that needs to be had is, is there a common ground that we can reach here? Maybe there's some subset of the functionality that we can put into open source that's still useful, but I have a different vision for the way this feature is going to work. Or, you know, do I just close it out right and say, no, I'm going to put this,

this is the type of thing that properly belongs in Pro. I don't want to do that, but, you know, that's the worst possible outcome, in my opinion. Something that comes to mind in that regard is what Twitter did with their API. They kind of said to API developers, like, don't hang out in these areas. These are danger areas. These areas are okay for you to hang out in.

We won't stomp on you. But these areas are kind of areas we're heading towards or things we're doing differently. And they kind of like, to a degree, somewhat roadmapped what was safe and what wasn't safe. Yeah, I mean, that's a great analogy. They've had to walk a really fine line because they're not a pipe. They're not just purely a pipe for tweets to flow down.

You know, they also want to control the glass, the way that people see tweets, so that they control the ads that people see. And that really has hit their third-party client ecosystem pretty hard. And so in your case, third-party clients are PRs, contributions, open source developers kind of helping you sustain the open source side, but at the same time, keep it progressing, keep it moving forward. Yeah, I mean, there's always a tension there um where you have something free and then something paid on top of that um

nobody begrudges twitter for having to make a living i know that i myself i would prefer to pay twitter uh you know a dollar a month you know a dollar a month is i would be happy to pay and if app.net and if they take their uh their to pay. And if they take their users, right? Exactly. They take their 100 million users and charge them a dollar a month. You know, you've got 100 million a month coming in. That pays for a lot of office space in San Francisco. But, yeah, I also understand that a social network is based on the size of the network.

And the vast majority of people don't want to have to pay for something if they can just see ads instead, which is unfortunate. Yeah, that 100 million users drops down to 350,000 or something like that. And now your network is not as valuable as it once was. Exactly, exactly. Let's pause the show for a minute, give a shout out to a sponsor. Hired.com is sponsoring the show this week and the URL you need to go to is hired.com slash changelogpodcast. Again, hired.com slash changelogpodcast.

And when you go there, they're going to give you a, they're going to double the signing bonus that they give you if you accept a job on hired.com from $2,000 to $4,000. Every week on Hired, thousands of tech companies in San Francisco, New York, Seattle, and L.A. bid on hiring awesome developers, providing the salary and equity up front. Some of their most in-demand jobs are web and mobile developers, DevOps, UI, UX, and even some product managers. The average developer gets about five to 15 offers with equity, with salary, all that up front. And even if you're not looking for a job,

you might know someone who is, you can refer them to Hired and get an awesome bonus as well if they accept the job. And the amount of that one is $1,337 total lead. So go to Hired.com slash ChangeLawPodcast and get hired. So thinking about contributions, I was just looking at Sidekick here as we're talking. And 713 forks, 242 contributors over its lifespan. I would say that the model, maybe some of that was before you had the Pro version, but it seems like there was no barrier for people wanting to hop in and help out there. So that's good.

Were you concerned about that, especially with Inspector? I mean, it's only been out for a day, but one fork so far. My main concern with Inspector is just the fact that it's using Go, so that it's a relatively new language. So, you know, Monit is written in C and uses, I think they're on Bitbucket. They're just kind of in a different ecosystem, so I'm not sure if people find Bitbucket easy to contribute to or not. I know that I've never really used Bitbucket before,

but hopefully it's on GitHub, it's written in Go, it's easier for people to contribute than something like Monit. But also keep in mind that Inspector is different from Sidekick in that it's not something that tightly integrates with your application. Sidekick is a framework, right? People are interacting with Sidekick APIs, their code is are interacting with Sidekiq APIs. Their code is running within the Sidekiq process. So you've got a lot more moving parts interacting with your app code. And so I think it's natural for people to interact with Sidekiq a lot more. see as much contribution and as much activity around people contributing to Inspector.

Inspector is kind of a black box where you install it, you set it up to monitor your components, and then that's it. You know, we'll see what happens. That could be completely wrong, but that's kind of my feel for it so far. Cool. So let's talk about Inspector then. So you said that you had Monit was a tool that you used. Not super happy with it, but useful. You decided to dig in, see how you could make Monit better.

And you said two things. First of all, removing features that you don't need. And then secondly, adding in some stuff that is more modern or that you think that you do need. So you may reiterate a little bit, but could you enumerate a few on either side of what you've done? Sure. So the first one was removing functionality. Right, which Anit D, you mentioned, is a big piece of that.

Yeah. What I did with Inspector is sort of make the decision that Inspector will not start and stop processes directly. So Monit and God and Blue Pill, they all have a way to start a process, stop a process, to set the user that it runs at as, to set the group that it runs at as, to set the group that it runs as, all this boilerplate to start and stop processes.

And what I realized is that's the job of your init system. All of these, the machine that you're using exists to run your application. Your application components are the most important thing running on that machine. And the most reliable way you can ensure that your components are running is to integrate them with your operating system's init system. And that in Ubuntu is upstart. In CoreOS or CentOS 7, that is systemd. And future Ubuntus are going to be using SystemD also. So Inspector defers the start and stop of processes to the init system

to upstart SystemD, run it, and launch D in OS X. So what I'm trying to guide people to do is to integrate their application components into their init system so that they have something reliable that is always there to start and stop these things. You know, Inspector itself, people shouldn't necessarily rely on to ensure that the components are started. You know, Inspector can crash. But the one thing that can't crash on your operating system is your init system. If it crashes, the machine crashes. So yeah, if you want your components to be up,

you want to integrate them with that init system. So when I made this decision, I realized that cutting out the starting and stopping of processes is a big bulk of the configuration. You know, every monet recipe, every god recipe has four or five lines devoted

to how do I start this thing, how do I stop it, what user does it run as, all this kind of stuff. So that dramatically simplified in Spectre because I don't have to deal with that. A couple other features that Monit had, for instance, are things like monitoring files and directories to make sure that they have the correct permissions, to make sure that they have the correct SHA so that the file contents haven't changed.

That to me is not something I've ever seen anybody ever use. Having been an application engineer using Monit to monitor the various daemons, it doesn't make any sense to me to monitor file SHAs and directory permissions and that sort of thing. Seems like more of a security concern than a monitoring concern. Correct. If you want something like that, you're going to be using either a read-only file system or you're going to be using some sort of IDS intrusion detection system.

So yeah, that seemed like kind of a poor man's security thing and really no reason for it. So that's another example of a feature that I just completely lopped off and had no interest in rebuilding. So what about the installation story? I mean, as I said in email, I'm a Monit user, have been for a long time. I'm a Debian, usually. I can just apt-get install Monit.

Inspector, written in Go, we can talk about that as well. Go kind of has this great story around dropping a binary somewhere. How easy is it to get Inspector on your machine, maybe speak to the open source and the pro versions? I'm sad to say that Inspector is two times as heavyweight as Monit. You have to run two commands, not one command. Oh, man, that's twice as many. Yeah, Inspector, I would imagine, in the future, It's not one command. Oh, man. That's twice as many. Yeah.

Inspector, I would imagine, in the future, will be integrated directly into the various operating systems package repositories. But right now, it being brand new, I have to distribute it myself. Can you do that with your pro version, too, though, down the road? Or not? Yes, I do. The pro version is ready for people to buy. It's available.

I run my own package repo that I control through basic auth, just who can access it. And so when you buy Inspector Pro, you get instructions on how to set up the repo access. And then from there, it's just apt-get inspector pro. So yeah, inspector itself is distributed through this great service called Package Cloud. And they provide sort of package distribution in the cloud, as the name might indicate. And so you have to run one command to set up their repo on your machine. And then from there, it's just apt-get install inspector. Not too bad. Not too bad.

It's as simple as I could possibly make it. But yeah, I mean, I worked for probably a month to get Debian and RPM distribution working. It is ridiculous how hard that stuff is to get working. Was this your first big Go project? Yep. Yeah, in fact, the reason why it took me four months of full-time work was because I was learning Go.

And so I would write a bit of functionality, and then a couple days later later I'd read through that functionality and say this code is terrible, I've got to rewrite it now and so yeah, I mean I rewrote Inspector probably two or three times in all of it I mean I rewrote all of it probably two or three times just because I ramped up on Go pretty quickly

but it still takes a couple months to get a feeling for what does idiomatic code look like? What is proper error handling? Where do you use interfaces and pointers and value objects and all that kind of stuff? Was that fun for you, or was that frustrating? It was fun-strating. Okay. And I make up a word. I think you just did.

It was fun and frustrating. It's always frustrating because you want to just be able to do something. So there's this cognitive dissonance as you try to write Ruby code and go. Right. But again, it's one of those, one of those paths where I realize you've got to walk down this path to, to get to the destination, which is being a journeyman programmer, not a beginning programmer in this thing. And so I enjoy that. I enjoy that process. So yeah, it did take me a couple of months. And for the first month I was definitely really frustrated, you know, things like, how do I take a string and split it up by commas

and get an array of those strings? How do I convert a byte array into a string? How do I convert this type into this other type? That's stuff that you rarely ever need to do in Ruby, but it's critical in Go. It was stuff that I had to learn all new. Yeah, I just wrote my first production Go. It's a small API for a customer and a long-time Ruby and JavaScript developer,

so you get things ingrained in your fingers. Maybe you have to have the docs open if you forget an API, but you're not searching, how do I do this in JavaScript or in Ruby? I've just found my Google search astronomically increase with Go over the last month and a half. There's a great site called gobyexample.com, which if it's like, I know how to program, just tell me how to do this in Go.

He has a great, here's how you do JSON parsing, here's how you do X, Y, or Z. Those kind of sites are super valuable. Why did you pick Go over your bread and butter? That's a good question. So Blue Pill and God are both written in Ruby. So I have always used Monit and shied away from them because to me a monitoring package needs to be as robust and as simple as humanly possible for reliability purposes.

And I don't want my application written in the same stack that is monitoring it. That's a, you know, you've got possibility of them both dying for some reason. You know, if your Ruby VM somehow breaks, well, now your monitoring solution breaks too. I always loved the monet simplicity. The fact that it only used a couple megabytes of memory. The fact that it was just a single binary to start. And so I wanted that type of simplicity in Inspector.

What I didn't want to do is write it in C or C++. So when something like Go or Rust came along, I said, well, these are perfect next generation system languages that I can use to build this type of infrastructure without having to deal with memory management and pointers and all that kind of stuff directly. So that's one of the reasons I did it in Go. The other reason I did it in Go is simply because the language has a really strong standard library where I didn't need to pull in any third-party packages at all to implement it. Inspector has no runtime dependencies aside from the Linux kernel. It's just a single, it doesn't even use libc.

So you think that choice has paid off so far? Well, we'll see. I have, at the very least, I've invested in learning Go and become not a beginner Go programmer anymore, but I'd say a journeyman Go programmer. So in terms of investing in myself, it paid off. But it remains to be seen how well the commercial product sells

and how well the open source project is taken up. It's still early days, having been launched 24 hours ago. It's often that whenever you try something new like this, whenever you go from Ruby to Go, that you often compare. Can you give us a comparison of what you love about both or what you love more about Ruby or what you love more about Go now that you're, um, experiencing the, the awesomeness that it is.

Sure. Um, Ruby is fantastic for building a big thing. If you can leverage like rails, you know, you just, you really can't beat building a website in rails. It's still,

it's still the best thing out there as far as I'm concerned. I would not want to build a large website in Go. I think that would be inappropriate. I think the Ruby flexibility and prototyping speed is still much faster than Go's speed. Where Go shines is where you've got something very simple, very focused that you want to build, and you can sort of hold the code in your head and just build it out really quick. Go's speed is

Go's runtime speed is really nice for sure. Running my test suite takes a tenth of a second. Of course, Ruby can do some of that if you structure the code correctly. But yeah, I think Ruby has points where it shines in terms of these frameworks like Rails, like Sidekick, where you can build these large-scale apps pretty quickly. And Go is more of a sharp focus tool for building

smaller, lower level things is typically how I think of it. In your list mentioning the introduction to Inspector, you've got several things that you have on the plate that you're supporting in terms of writing alerts to Slack, HipChat, Campfire, FlowDoc, some of the common hit lists of popular kind of collaboration tools. You also mentioned that it's brand new. It's not 1.0 yet.

Can you talk about, since this is new, and for those listening, you're probably listening as much as maybe five days after the recording of this. So when we say it was released one day ago, it was actually like six days ago, technically, depending upon when you listen to this but um can you talk a bit about you know this early version not 1.0 version um in terms of feature set what you've put in it and like maybe where you see things going and and possibly even how um how how actually using go supports some of the lifespan you see for this? So the base inspector, the open source version of inspector, will monitor any service that's integrated with your init system.

It will monitor daemon-specific metrics. So it understands MySQL. It understands Nginx, Redis, Memcached. Those are the four I launched with. I see there are tons of opportunity for people to contribute their own daemon-specific metrics. Things like Cassandra, Kafka. There's a whole world of Java infrastructure, for instance, that isn't covered at all. But I'd love to have integration with just more infrastructure that people use to build their apps.

Those happen to be the four things that I use, that the Revit community uses often. But maybe Python folks, maybe, for instance, like Celery or RabbitMQ would be more examples. So daemon-specific metrics are in the open source version. What else is there? or RabbitMQ would be more examples. So daemon-specific metrics are in the open source version. What else is there? So you're going to monitor your CPU and your memory of your process, your daemon-specific metrics. You can monitor the host metrics, things like swap, disk space usage,

CPU usage. And then you can also get an overview of all the status of all the services that it's inspecting at a given moment. And you can also see a graph of a metric in the console. So if you're at the terminal and something, an alert fires, you can actually see the history of the metric right in your console without having to open up and find a graph or something like that, which is pretty nice. Now, in terms of the commercial version Pro, Pro has the ability to monitor initd, the old legacy stuff, that ball of mud that I referred to.

Oh, so you didn't want to do it, but you'll do it for money. Exactly. Exactly. I like that. That's an example where if people have legacy services, if they're an enterprise and they just don't want to touch the thing, but they do want to use Inspector and Monitor, you pay the money and the problem is solved. Yeah, I mean, part of this hard line that I'm having to take with features

is trying to guide people to author better applications. And sometimes that means I'm not going to support the old way of doing something because I really genuinely feel it's not the right way to do things. Now, if people want to pay me money so that they continue to do it the old way, then that's their choice. But, you know, I'm taking a stand here and saying that initd is not the right thing to do anymore. So, yeah, so initd is supported in Pro. And then, yeah, as you said, chat rooms for teams who want alerts to be piped into their shared chat room where maybe they've got people in the chat room 24-7. That's a perfect example where you can sort of cut down noise in

your inbox by directing the alerts into the chat rooms. And then the final feature in Pro right now is the ownership. So you can give ownership to various components. You can say, I want alerts for this thing to go to this particular team or this particular person. Because Inspector itself, the open source version, you can only send alerts to a single email address. That's it. Now, what's coming down the pipe? Yeah, I've got a bunch of ideas. One thing I want to put in the open source version is monitoring cron jobs

to ensure that cron jobs are running. If you have a cron job that runs hourly and you deploy your code and that code change breaks that cron job, how do you know? Oftentimes the job will just start silently failing and you won't know until a customer calls customer emails or maybe you don't receive a report

the next day or something like that but having something that notifies what I want to do is have a way for the cron job to notify inspector that hey I just ran and then inspector will say if I haven't received a notification within the last hour or within the last day, to fire off an alert to say, hey, this cron job didn't fire. Let me just say, as a longtime Monit user and relatively happy Monit user, if you do that feature, I will immediately switch.

I was going to say, you seem like you were lamenting with the pain. Like I can almost audibly hear the pain you felt from not having that feature. Yeah, I've looked for solutions. There's some online services where you can, you know,

do your cron job and then, you know, do ampersand, ampersand and then hit some API that just says I did it, right? That it actually succeeded. And then they'll send you emails and stuff if it fails.

Try those. There's other things where you can just pipe it to an email address if it fails. Anyways, they all suck. Well, mine's going to suck just as much. Oh. Oh.

In a better way, though, maybe. Yeah, so that was one idea I had for another feature. You know, the other obvious feature would be sort of a web interface to see an overview of the different metrics you're tracking and to see pretty graphs. Yeah. That would probably be a pro feature. I'm not sure.

But, yeah, anything that's sort of team or collaborative is definitely going to be lean toward pro things like cron jobs though. You know, I can see individuals wanting those as part of their applications. And so putting a cron job checker, it seems like a natural fit. Yeah, I'm for it.

Cool. I'll count that as a plus one on the issue then there you go well cool mike um we uh we tend to ask a few questions at the end of the show but we're going to ask one simple question because that's that's the way we're going to roll around here but um inspectors new it's you know let's say, you know, barely a day old in terms of release. Um, can you kind of give the listeners a way that you're looking for engagement? You know, is there a feature set, like you'd mentioned earlier, you know, supporting different systems, Debian and, and were some of the ones that you'd mentioned that you use. So you're supporting those or are there a hit list that you have a roadmap? How can people jump in and help you launch the open source side and maybe even how to buy the pro side?

Well, what I would love, what I need right now is just users. You know, it is a brand new project, so I would love people to download it, try it out. I'm definitely not strong in terms of the operating system packaging. So dev support, RPM support. I spent probably a month trying to polish it and get it working, but I'm sure that there's plenty of room for improvement there. So maybe some code review on certain areas?

Yeah, exactly. I mean, if you've got more of a De, you got more of a Debian guy or more of a, um, uh, a Fedora guy, uh, wants to, or, or girl for, for that matter, uh, agenda is, is not an issue. Um, if, if anyone wants to come in and, uh, and help me out there, I'm happy to have that. Uh, I've cobbled together what I have right now, but I'm sure there's room for improvement. And the other thing is, is just use it and give me, give me ideas for features. Send PRs. And, and remember the, there's that daemon specific feature where, you know, I want, I want inspector to know about as many of these popular different application components as possible. And so getting a PRS to add more and more of them would be awesome.

So that's, that's definitely ripe for a person for a PRS. Well, awesome. Well, is there anything else that you want to cover Mike in closing before we take the show out? Not really.

I just wanted to thank you guys for giving me the opportunity to come on and ramble on a bit. Cool. Well, we'll have all the links in the show notes, so we'll mention them here on the air, but mparam on Twitter if you want to follow Mike. But we'll have some links in the show notes

to Back to the Code, back to your company site. We'll even link that blog post that Jared mentioned about this fantastic way to have this path to success like Mike has found for sure.

Mike, thanks for coming on the show. We had some awesome sponsors for this show. As you might know, not only are we member supported, but we're also sponsor supported because we work with some really, really cool companies. One of those cool companies is CodeShip. Love CodeShip. Those guys are awesome. Hired.com and also DigitalOcean.

We're hosting on DigitalOcean. We love DigitalOcean, and we think you should too. If you're not using them, then I just make a sad face, and that's just how it goes. But that's it for this week's of ChangeLog, and we'll be back as soon as you want to hear us. Let's say goodbye. Bye.

Bye. change log and we'll be back as soon as you want to hear us let's say goodbye bye