The Changelog: Software Development, Open Source - Homebrew and OSX Package Management (Interview)

Episode Date: September 14, 2010

Adam and Wynn caught up with Max Howell, creator of Homebrew to talk about package managment on OSX, beer, and scrobbling.

Welcome to The Change Log, episode 0.3.5. I'm Adam Stachowiak. And I'm Winn Netherland. This is The Change Log. We cover what's fresh and new in the world of open source. If you found us on iTunes, we're also on the web at thechangelog.com. We're also up on GitHub. Head to github.com forward slash explore. You'll find some trending repos, some feature repos from our blog, as well as the audio podcasts. And if you're on Twitter, follow ChangeLog Show.

And I'm Adam Stack. And I'm Penguin, P-E-N-G-W-Y-N-N. Chatted this week with Max Howell from Homebrew. You a Homebrew user? Yeah, I aim to be. I think I have it and I'm using it and it's made life simpler, but I haven't actually used it in a while. It's let me cut the cord with

MacPorts and Think, some other projects to manage packages on OS X. It's got, I wouldn't say a bad install process, but it's a little beefy for a designer. I was kind of hoping he would talk against that. Training wheels can only come so big. Thanks.

I love homebrew. I especially love the beer theme. We riffed on that quite a bit. Yeah, that was a lot of fun. The notion of kegs and cellars and whatnot. I think we've exposed this in a couple repos we've commented on or had on the podcast in the past, but having fun and humor in the syntax and in the installation processes or the wiki, the documentation, it's kind of fun. Absolutely.

One of the things that he said in the interview that I absolutely agree with is if you're doing an open source project, it's got to be something that you use yourself or you won't be motivated to keep it going. Oh, absolutely, yeah. Mac's built Homebrew to scratch an itch and it's kind of taken off. So I know that I'm a user, you're a user. And for the folks that are hollering for non-web content, here you go. Homebrew.

Yeah. I also want to plug something else as well. We partnered with Jason Seifer of the Ruby show and the dev show fame. He runs a site called geniuspool.com. It's a job board designed to connect employers and job seekers in a very targeted manner. It's got this very cool thing called the Genius Pool Network. It gives extra opportunities for promoting your job to the right kind of audience. We're a part of it. The Dev Show is obviously in it, and the Ruby Show is in it. So if you're hiring a developer, you could choose the Change Log, the Dev Show, and even

the Ruby Show as extra promotion for your job. So head to GeniusPool.com right now. And if you post a job, check the box next to the changelog for an extra $100, and we'll read it on air. GeniusPool.com. Everybody out of the pool. Fun episode this week. Should we get to it? Let's do it. We're joined today by Max Howell, founder of the Homebrew Project,

to talk to us about package management on OS X. So, Max, why don't you introduce yourself and let the folks know who you are. Well, hi, everyone. I'm Max Howell. I started the Homebrew Project about a year ago, and it's been quite successful. Before that, I started programming when I was six, in fact. My dad came in one day when I was playing Super Mario and insisted I stop, which I wasn't very happy about. But he put me down in front of our computer and started teaching me how to program. And when I realized that I could probably make my own Super Mario I became more enthusiastic and since then I've been doing it on and off I never really intended to do programming as a career but having done like

a chemistry degree I decided that I didn't like chemistry very much it wasn't really about changing the world like I expected it to as much as making potions and solutions and small measurements upon them and i uh i fell back on my programming so i uh started working at last.fm a few years ago and it was working at last.fm that made me interested in package management on mac because I found the existing solutions to be not exactly to my liking as a developer. But I never got around to doing it. And I was...

Well, I was going to say that I work at Tweed Deck now. So I left LastFM last year, and now I work at TweetDeck. I work on mobile and client software. Well, I was a last-prem. I worked on mobile and client software as well as a new developer. I'm a new developer at TweetDeck as well. I know Adam's a big Twitter user, probably has lots of questions on TweetDeck in a minute.

But to talk first about Homebrew, so what led you to kick off the project it was just one too many image magic installs uh yeah stuff like that like i had i have a quite large linux background uh before i said my works on a project called amarok which uh was and probably still is one of the bigger music applications on Linux. I joined the project when it was very fresh, but it had potential, I could tell by looking at it. And I worked on that for two

and a half years. And, well, the package management on Linux is mostly great. We always thought there was like a niche area for like a developer package management solution. You don't always want what the system gives you. And while working at Last.fm, I had to do an awful lot of packaging of our software,

like the desktop. We did Windows, Mac, and Linux. And I had to manage the packaging of each system, and, God, I hate packaging. C++, which I never really want to have to do again, is a real bitch to package up into a binary that will work on all the different systems. And when our client had lots of separate dependencies

that were very difficult to manage, and I was using Mac ports, and I was developing on Mac, and I just felt that I couldn't get enough control over what it was producing. And I wanted to have control over it. Quite possibly there's a way to get control over it, but the whole project didn't seem geared around that. It seemed like it was geared around giving people a solution to getting software onto their computers.

Well, I wanted a solution for developers that would allow you to get what you needed for the work you were doing. So I would constantly moan about this at the pub every Friday with people there when we were talking about our work for the week. And eventually someone just told me to do it. I was like, well, I guess I could. So one day I just started making it. And within a few hours, I had something which was basically great, really. It was the way it really needed. I'd had ideas for a few years about putting things into separate directories

and symlinking everything into the main tree so that you could easily manage it without having to use tools. I never understood why you need like these elaborate tools for managing small bits of unix software uh these massive black boxes which don't make any sense because it's not really that difficult there's not much to it so i wanted a system where you could just do it with your own tools uh if you want but there is a convenience tool so that you can get things done quickly if you want so i uh i just started building that and well i put it on github because that's where all the cool people were putting things sure and i chose ruby because that's what all the cool people

were doing and i wanted to look cool. Although I had done some Ruby. It was, admittedly, my first big Ruby project, and I liked it a lot, the stuff I'd done before, but it was mostly scripting stuff. As I say, I was really a C++ developer at this point. I hadn't done much Ruby, which I think you can see in the code base. It does look like a new prototype.

For the uninitiated, explain a bit about the anatomy of a Homebrew package, the formulas. Yeah, so I just wanted something which was as simple as possible, what I was going for. So a Homebrew formula, a package recipe, is just a Ruby script. It's a class, a Ruby class. And you define some metadata at the top, the minimal amount of metadata.

I was like, why do you need all this duplicate metadata for like description and stuff? And I've had people say to me the time since, let's add a description field. And I'm like, well, you know, like the web is where the description is. You go to the homepage. So there's a homepage field.

That's it. There's no description. So if you want know, like the web is where the description is. You go to the homepage. So there's a homepage field. That's it. There's no description. So if you want to know what the package is about, you can type broom home and the package name, and then you can see. I didn't want to duplicate all this information and have to keep it up to date. I wanted the minimal amount of work I had to do

to keep this project going. I didn't want to have to update description fields. And then there's an install function, which literally just runs the scripts that are required to install the thing. And it installs it into its own prefix. So if you've installed Homebrew to use local, there's a directory in there called seller, which is part of the beer theme, which I'm proud of.

And inside that, there's like, like say wget is what you're installing as a wget folder inside that there's another folder which is the version and so it installs there and then it just symlinks it into user local so you have these encapsulated folders it's the gobo linux approach this is where i got the idea from it's not mine gobo linux and maybe they didn't even invent this. I'm sure they would like to claim they did, but I've heard of other projects doing similar things before. But it just seemed like a sensible way to do it for an auxiliary package manager for OSX.

And what we really try is to complement the system. It's not trying to be its own autarky like MacPorts does. That's one of the things I didn't like about MacPorts. So yes, the formula has an install function. it has a few other little bits of things like you can show some caveats in case there's like some issues that the person installing the package needs to know and that's one of the things about homebrew that is also the case it's it's not meant to like hold your hand too much it does the bare minimum which is install the software and then it tells you what the issues are after that.

So you can deal with it because you're smart. You're a developer. You know what's going on. So I want you to have the power to do what you need with the software you're installing. So let's talk about that a little bit. In the installation, you mentioned to delete these two different directories, user-local-include and user-local-lib. Can you talk about that a bit and why and what kind of troubles sprout about when you don't do that?

Well, you know, this is why I don't always see C++ and all that anymore. The entirety of the Unix system is complicated. Basically, user- user local as a directory will get included when you install C software, whatever, really. There is very little we can do to stop it from using those directories.

So if you have libraries or include files or headers in there which conflict with the package you're trying to install, it will cause problems and bugs, which it's very difficult for us to prevent. So really the best bet is just to start fresh there. And we made this brew doctor command in the end

because we try to minimize the amount of support we have to do. We want to help you to help yourselves when you're using Homebrew. So as you're deleting some of these things and the next thing you're going to do is you're going to start doing some installation stuff so the next thing that comes to mind is when to sudo what's uh what's up with the whole solution here with homebrew and well because we're installing from source sudo is dangerous like there's massive install scripts that get run but you can't know what they're doing and i've seen like running like a file system watcher to see what stuff goes on

like some of these packages try to edit core system files and you just don't know what it's going to do um there's ways around that and macports like puts everything in a chome root directory which is the sort of thing that maybe we should do. But instead we just, you know, we're making a simple solution here. So no sudo is a very sensible thing to do. And it feels like OSX, doesn't it? You know, you install TextMate without having to sudo. You don't, you know, churn it to root after you've copied it into your application's directory.

Admittedly, some of the packages you can install with Homebrew, it would make sense to root them, but we trust you as the guy installing the software to know what you're doing, basically. That's kind of the distinction between Homebrew and the other options. Homebrew really is for developers who most of the time probably are just installing a few dependencies. They need to compile the gems or the Python eggs that they need to use.

So we trust you to know what you're doing, basically. And sudo is an inconvenience. Once you've not used sudo to install stuff, you realize it was holding you back right it's much easier to edit the packages you're installing and customize them how you want if you're not using sudo the whole time i've seen people be adamantly against it because especially if they come from linux it's the mentality there that all the packages should be installed with a root command and root command and there's security implications.

Let's talk about dependencies for a moment. So how does Homebrew manage dependencies between kegs and packages? In a very basic fashion, it has a depends piece of metadata for each formula, which is just a name which directly correlates to the file name of the other formula formula are just files on the system that's how we resolve um unique naming the file system is used all over homebrew like when i designed it i wanted just a very simple system simple systems are the ones that work. They're the ones that succeed without anyone having to do too much work. And also they're the ones with the least bugs.

So it just resolves to another file and the packaging system installs that. There isn't anything particularly advanced like other packaging systems have where you can specify versions and stuff. It's not that we don't plan this sort of stuff. It's just in a very typical open source fashion, we've done what we've needed as we go. And we haven't needed anything more advanced yet.

I'm kind of hoping at some point someone will need something more advanced and they'll submit a patch. And that's basically happened for the whole project. Because we based it on GitHub, it had amazing uptake and very, very easy for people to contribute like nothing else like the whole system's based on git i don't know if people realize this about homebrew but you know

you've installed it and that's a git repository you've just put somewhere you can put it anywhere you like as well i don't know if people realize that you put it in your home directory we recommend user local because it makes it so much easier to install stuff, especially gems, like Ruby gems. People put it in their home directory because they're like, well, I don't want to mess around with use local. It's in the use directory. I don't know what that is or whatever they're thinking. I'm sure they have excellent reasons for what they choose.

And it makes it a lot harder to install gems because this is probably C code. It's what people don't realize. It has default directories it looks for stuff and this is why installing gems off of mac ports there's always so much for a pain in the bum it was going to say ass there i hope you guys realized although maybe you want that so i'll repeat it such a pain in the arse because it was in Oplocal and you had to tell the gem and every gem has its own way of knowing

where to look for dependencies and you have to like Google and find a blog post where someone's done it before you can still wear it to user local, it just works anyway what was I saying you can install it wherever you like and it's just a git repository so you can edit anything.

And that's one of the beauties of Homebrew as well. It's a system that builds from the ground up for you to be able to manipulate exactly how you want. Install packages how you want. No messing around with variants and things like that. Just edit the formula yourself. Make it add the extras you need, et cetera. And then just commit that to your own fork on github and then you just keep pulling from mine and git manages all the merging for you and you've got this

almost magical so that's what i wanted from this whole system something where i could manipulate it as i needed for the projects i was working at work and home and uh it just took the pain out of dealing with all these other dependencies and bits of software. And that's really the ultimate goal. When I started the project, I also had other ridiculous goals, like everything should be highly optimized and

ridiculously fast and no universal binaries because we don't need those, etc. And as the project became more popular, it became apparent that these were ridiculously impractical goals. It's the kind of stuff that Gen 2, you know Gen 2 is one of the Linux distributions

where it's renowned for its users wanting to optimize to 11 for everything. And I was kind of going down that route just for the fun of it, and I've since rejected that policy, although not everyone's happy about that. They seem to want it. But the important thing is the system is practical.

It's useful to you. And that's what I try to maintain. So talking about formulas, you're pushing 2,700 watchers right now. Yeah. How many formulas are out there and what are some of your favorites? I can tell you. Because that's the thing. You just step into the homebrew directory.

So let's go there. That's what I'm doing right now in my terminal while you wait. Except it's not responding to my clicks. So I go into user local, then I go into library, because I chose the library name, incidentally, because it's an OSX thing. People think that seller and library, it's like a osx thing people think that the cellar and library it's like a room theme that i came up with but no the theme is beer i gotta be this is like when i was

in the pub and we were discussing this project i was like okay well what am i going to call it what am i going to do and like someone suggested i have a beer theme. I was like, yeah, excellent. That would be fun. As much as now it seems a little unprofessional, it does help with your open source projects to have something fun about it because it makes people tweet about it and it makes people blog about it. And they're like, oh, I found this project

and I thought it was funny. They had a beer theme and there was formula and there was a cellar and things were installed into kegs, et cetera. And if you go go through the source code it's kind of full of beer jokes incidentally i don't know if that's your sort of thing absolutely it's not a good source code but it's full of good comments so let's see there's uh 1167 formula currently which is pretty good going. When I started the project there was five and I added them as I needed them and then after a month

or so the project turned up on a few people's radars and I got a few contributions here and there. As I was saying about being on GitHub, it's just great. It makes it so easy for people to contribute. They just throw stuff at me and for the first few months

of the project i struggled to keep up with contribution after it took off like it only took off when josh peak from 37 signals tweeted about it he was like i'm going to install snow leopard because it was just before snow leopard came out last year like the project is like a year old that's about it i started building it after i left loss of m uh i started before then and uh i just continued at that point and he said i'm gonna source no leopard and i'm gonna use home brew to manage my package manager it was perfect timing in that respect as well because people were installing fresh installs of osx they were kind of looking for an alternative and after that happened um i got like 50 forks that week and then it went up to like 200

in a couple of months and since then it's just it's been ridiculous the amount of forks people kind of fork contribute uh make a pull request and then they keep their fork around maybe uh they do another uh pull request here and there for like formula like i did design the system to be ridiculously simple to contribute to that that was one of the things i didn't like about mac ports i just didn't know how to contribute if i had a contribution so you can type brew create at the command line and then the url for the tarball for the thing that you're trying to make a formula for and it automatically generates you the formula uh as best it can like you'll probably have to tweak it a little bit but at that point you maybe

already have a contribution to the project uh that has really helped so i don't we are the the most fought project on github now since rails disappeared a few weeks ago. I don't know what happened. But I can't exactly claim that's completely because it's the most exciting project. It is really because it is just so easy to contribute to it. People feel almost that it would be wrong with them not to because I made it so simple. So people, do they buy you beer whenever you meet up with Mamita?

And what's your favorite? I've certainly had a few threats of pints. Well, the problem is we're in London, and there isn't such a thriving Ruby community here. It's certainly here, and I won't deny it. I feel that it's more in the States. So, well, if anyone ever wants to invite me over for some of those conferences, I'll certainly come.

But my favorite beer? Well, I'm a big fan of the British Ales, as it happens. I don't know if you guys ever tried one. Favorite brand? British Ale. Oh, is that the actual brand? I just thought that was it. No, it's a type of beer. Oh, yes, yes, the type, right?

Yeah, like you have your microbrews in the States, and they tend to be pale ales and IPAs, and they're very nice. I really like them, actually. But we have the stuff that tastes a bit like pond water. You become used to the taste, and there's some really nice stuff. My favorite is probably there's a new brewery called Brew Dog, which is a Scottish brewery, and they're the ones making these crazy beers you might have heard of that are like 42%.

And they brew them in a whiskey-type fashion. But they do some normal stuff as well. And they've got this one that's a hardcore IPA. It's cool. And it's got like an interesting picture on the front. They've got a very clear identity of brand. But the beer tastes delicious as well. 42% alcohol volume as an 80%?

Yeah. Wow. It's a beer which is distilled. They distill it with ice. It almost doesn't count. It's not very nice, actually, the nuclear penguin. It's the tactical nuclear penguin is what they call it. I could give a high five.

It's interesting. Yeah, the beer thing was kind of, well, I work in Old Street, which has been coined the Silicon Roundabout, which we say with some chuggering. But there is a lot of startups here, and we do meet up a lot, and there's a lot of pubs and we tend to discuss our work with a few pints. So yeah, the beer has been a part of my life, I guess, and that was part of the reason for the theme. So you built the Scrabbler over at Last.fm, is that right?

Yeah, exactly. I worked on all the client-side software are you a music guy uh yeah well you know worked on Amarok for three years I've always been into my music mostly for consumption purposes I went to Amarok because I was interested in making the perfect music application and to a certain extent we did at the time it was revolutionary there was nothing like it and we pioneered concepts like uh sticking the wiki page for the artist in the app so while you were listening to music you could see what the artist was if you hadn't discovered them before uh there was nothing particularly like it before and we were like heavy users of last event metadata and data as well which is how i got the job at last family which is like a testament to working on open source

can get you jobs at cool companies. So now you're over at TweetDeck working on mobile, right? Yeah. Slinging Objective-C or? Well, I just did the Android app, which is in public beta at the moment, but we're going to be releasing to the Android marketplace

soon. So it's a great app. You should try it if you're on Android phone. We decided to try something different with the Android marketplace soon. So it's a great app. You should try it if you're on Android phone. We decided to try something different with the Android app to what Tweeday does currently. You might know the iPhone app. And we wanted to do new stuff and see what people's reaction was to it.

And it's a great app. Really pleased with it. I love making apps. That's where all this comes from. I love making little things. What's your take on the state of open source on OS X? That's the best platform for us, open source, in my opinion, currently. I came from a Linux background.

I switched to Mac tentatively because I appreciated the fact that the UI was solid and I wouldn't have to worry about my Wi-Fi drivers anymore and things like that. The only amount of times I'd recompile my kernel to make the Wi-Fi drivers anymore and things like that. The only amount of times I'd recompile my kernel to make the Wi-Fi work and I was sitting there thinking, I want to be writing my software, not messing around with the kernel. And so in the end I was like, oh, okay, Mac looks pretty good. It's Unix underneath.

I've always said Unix. I don't know where we'd be right now with OS X being a popular platform like it is with developers. The Unix underpinning is essential, in my opinion. And it had a nice GUI, and everything just about works. And I started to find that there was a lot of really quality open-source stuff on OS X.

It's funny how Apple's passion for, like, polish and good work comes through in what people expect their own apps should be like. People aren't happy with it being, I dare not say and offend Linux people, but like it is more mediocre though. I worked on the UI stuff for ages. I did two of my own apps there, which I felt I put a lot of time into,

but like it just, it's a bit messy. And you see like apps likemission and VLC on Mac, and they're much better than the equivalents at other places. And that's the GUI stuff. So speaking of OS X, do you have any other plans or any other ideas that you're brewing? I got ideas for Homebrew. I think it should integrate more with the system.

Initially, I wanted people to use it, you know, so I designed it so that it was self-contained in its own directory so that you could just delete it and you know that everything's gone. There's no messing around. But it's not so useful in that manner. Like, you need to be able to have, for instance, JAR files for Java developers. They need to go in library slash Java.

Otherwise, you have to mess around with your system some more. So I feel that if you install to user local, you're saying that we should put things in the places they should go. And we'll do it via symlinks, so you can still delete it. That's the kind of thing I'm thinking of. I often play around with ideas for tools that i could use like i really want an app which uh sits in my menu train just tells me when people i'm working with committing to projects they've forked from me and like it pops up etc with growl

uh things like that i always almost sit down on saturdays do whatever. I've got loads of little apps I built on GitHub, like a little app that shows you pictures your friends have taken on Flickr and on the dashboard because otherwise I never go there and I never see that stuff. And I wrote a web app for showing you comics that you're interested in. So I just wanted a place I could go and I'd click next, next, next, next, and every day and it shows me all the comics. And so I do do these things.

But on Dice X itself, I don't know. I kind of feel that the future is mobile nowadays, I have to say, but that's just a consequence of the industry I'm kind of working in, my job. You mentioned the Android app. So iPhone or Android? Honestly, I think this will cause me some problems, but I prefer iPhone, I have to say. But Android is a very exciting platform.

It's definitely up and coming, and the flexibility and the things you can do with it are very interesting. Just right now I'd rather have an iPhone because it's got the better apps and they're more polished

and there isn't a really good Android handset yet. I've heard good things about the Droid X. I've heard good things about the G2 which will hopefully come out soon. And the Galaxy Tab looks great. I've heard good things about the DroidX. I've heard good things about the G2, which will hopefully come out soon. And the Galaxy Tab looks great. I've seen screenshots of my app running on it. I'm like, yes,

I'm begging Google to give me one, but I don't think they will. But, yeah, currently the iPhone's pretty awesome. So we're at that point where we kind of wrap up the show, but before we go, we always ask what our guests have on their open-source radios. So what're at that point where we kind of wrap up the show, but before we go, we always ask what our guests have on their open source radios. So what's out there that's in open source that's got you excited,

you just want to play with it? I'm pretty interested by the Mac branch of VLC, the Classes, is it the Classes branch? I have that on my GitHub radar and I watch it myself and I forget exactly what it's called, I'm afraid. But, you know, they're trying to do a proper Mac app for VLC. And every other Saturday I sit down and promise everybody that I'm going to make a great Mac music app.

Like, it's just the sort of thing I'm never going to have time for. So I am interested in what they're doing with VLC. I almost have contributed to that several times myself. It's difficult with open source because there's not many good apps on mobile being done now. I think the most exciting area of open source right now is definitely

server-side stuff. And I don't personally deal with that very much nowadays, apart from through Homebrew of course. You asked me earlier what my favorite Homebrew package was. I think it's definitely SL. It's the app that when you typo LS, you get a choo-choo train come across the screen.

It's great. I committed to typo for brew. I don't know what the common one would be. We should do something similar. Any more questions, Wynn? No, that's it. You want to wrap? The only thing I wanted to riff on really was the i kind of do like your idea for the notification thing for repost and stuff like that so it could be a really fun tool to have because i know for me i hate to see uh to sit

there and like watch my my home page more or less just watch that that stuff. But I guess we do have to get Tail back up when... It's in process. Yeah, well, I think it'd be really useful. Just to see, especially, we use GitHub at TweetDeck at work, through my recommendation, as I would, I guess. And we have private

repos for everything. I want us to release some open source, and we probably will at some point, just for the sake of looking like one of those brand names like Facebook, where they have a GitHub presence. But I had some stuff I did with the Android project, which would be useful, just tools I wrote, which facilitated

some easier Android development. Well, hey, Max, we really appreciate you taking the time to spend some of your afternoon, I guess, what's that, late night for you, over there? Evening. Yeah, late evening. So we appreciate you taking the time to spend some of your afternoon i guess uh what's that late night for you over there uh evening yeah late evening so we appreciate you taking the time to spend uh with us and appreciate you coming on the show a pleasure thanks I see it in my eyes So how could I forget when I found myself for the first time

Safe in your arms As the dark passions show