The Changelog: Software Development, Open Source - Tornado, Hip-hop, Three20 (Interview)

Episode Date: May 18, 2010

Adam and Wynn caught up with David Recordon and other Facebook developers to talk about their wide range of open source projects including Tornado, Hip-Hop, and Three20 as well as OpenGraph and OAuth 2.0.

Welcome to The Change Log, episode 0.2.4. I'm Adam Stachowiak. And I'm Winn Nedland. This is The Change Log. We cover what's fresh and new in open source. If you caught us on iTunes, we're also on the web at thechangelog.com. And hey, look, we're also on GitHub. Yep. Check out some feature repos there from our blog, as well as some training repos from GitHub, as well as our audio podcast. And if you're on Twitter, you can follow

Change Log Show, not the Change Log. And I'm Adam Stack. And I'm Penguin, P-E-N-G-W-Y-N-N. Had fun talking to David Recording and gang over at Open Source, at Facebook, rather, about the Open Source projects. Hey, you might as well call them Open Source with all the projects they've got going on. That's true. Talked about Tornado and Hip Hop and 320, some cool applications that cover a wide range of technologies. Also some new API advancements, OAuth 2 and OpenGraph.

Yeah, a lot of fun stuff going on there. They were really excited too about everything they were doing. Yeah, the passion kind of just oozes out of what they're doing. You can see it. Yeah, yeah. Hey Adam, where can people

catch up with us in person? The Texas JavaScript Conference over at Austin, June 5th. June 5th? June 5th. And you're going to be at LesConf this weekend? Absolutely, yeah. Be over there 5th? June 5th. And you're going to be at Les Conf this weekend? Absolutely, yeah.

Be over there all weekend, enjoying Atlanta. Sunny Atlanta, Georgia. Great episode this week. Should we get to it? Let's do it. All right. We're joined today by several folks at Facebook to talk about open sorcery at Facebook. David, kind of the ringleader there in the conference room, why don't you introduce the folks and each of the projects?

Sure. So I'm David Recorden. Joined Facebook last fall and work on open source and standards here at Facebook. As Wynn said, we have a group of five engineers here that work on a variety of open source projects. We'll go ahead and introduce themselves, talk a little bit about the projects which they work on, and we can dig into them, answer some questions. Hi, this is Hai Ping. I joined Facebook about three years ago. I have been working on this project that's called HipHop Compiler, which is basically transforming PHP into C++, just to try to speed it up. Hi, this is Paul Buhite. I'm one of the former Friend Feeders who joined Facebook last August,

and I'm going to talk about Tornado, which was the real-time web framework that was extracted from Friend Feed and Open Source last fall. I'm Owen Yamauchi. I work on our iPhone application. I've been at Facebook for almost a year now, and I'll be talking about 320, which is our open source framework for iPhone developers. And I'm Scott Mivicker, and I work

on the hip-hop for PHP with Hyping as well. I always love those deep, long introductions there. They're always awesome. This one's like five people deep. I was trying to write all the names down. I thought I'd lost count. I was waiting one more.

I wrote all the names down and kind of got the gist of it. So we got David B. Corden, we got Hyping, we got Paul Buhoy, we got Owen Yamaguchi and Scott McVicker. Is that right? Yeah. Okay. So, I don't know, should we start with Tornado? Sure. Sure. Paul, do you want to start with Tornado a that right? Yeah. Okay. So, I don't know, should we start with Tornado? Sure.

Sure. Paul, do you want to describe Tornado a little bit? Sure. So, Tornado is actually a collection of useful parts, basically, that we extracted from FriendFeed. We took what we thought was probably most reusable, most applicable to things other than FriendFeed, and also things that were

somewhat unique, and package them all up for the world to use. And so it's written in Python. It's actually relatively small. I keep getting this number wrong, but I think it's about 5,000 lines of code. So if nothing else, it's actually just an interesting and enjoyable code base to read through and get familiar with some of the real-time non-blocking concepts that are becoming more important as more web services are trying to do things like IM or real-time updates in the way that friend feed or facebook does and the reason that that can be difficult with a lot of traditional frameworks is that they are based on a blocking model where every request requires a thread or a process on the server and so if you have thousands or tens

of thousands or millions of users connected to your website. That means you have a connection open from every one of those web browsers at all times. So you need a web server that's capable of handling many thousands of connections simultaneously. And so that's really the most core value provided by Tornado. But there's also a number of other features in there like easy authentication modules for Facebook and OpenID and OAuth and Twitter and Google and just a handful of other utilities that we thought were really nice to have, such as command line flags that are very easy to use. So who's using Tornado besides Facebook, do you know? Quora is using it, I believe. That's Adam D'Angelo's startup that's doing the q a service

uh brisley which is jason shellen's um startup they're doing a kind of like a twitter client and more i don't really know the complete vision honestly but they're they've been in the process of switching over um and I've heard of a number of other people using it, but I'm not sure which ones are publicly announced yet. But there's a fairly active community. What is the Quora one you mentioned, the first one? Quora, Adam D'Angelo is one of the other, I don't know if he's officially a Facebook co-founder

or what the status is there, but he was one of Zuck's roommates early on and used to be the CTO of Facebook. And so he has a new startup called Quora, which is a Q&A service. It may still be in private beta. I'm not sure. It's pretty cool though. And what it does, part of the reason they use Tornado is it does some of the same real-time updates. So someone might ask a question and then other people will add answers or edit the question. And because of the real-time functionality, when you're looking at the web page, you'll actually see those updates come in as they occur.

So you don't have to reload the page to see if someone has added a comment. You'll actually see it pop in in real time. How would this correspond to, I guess, some other technologies, maybe in the Ruby space, like Event Machine or Node.js over in the JavaScript space? It's actually a comparable approach. I don't know everything about Event Machine, but I know I think Node.js is based on a similar model where you have, I think they're all based fundamentally on the EPOL system call in Unix, which allows you to basically monitor

a large number of file descriptors in an efficient manner. And so they're all just different wrappers around EPOL at its heart, but done in different languages. So I think Event Machine, and I'm probably getting this wrong, but I think it's kind of a hybrid of C++ and Ruby. We actually tried to keep Tornado as pure of Python as possible. There is a small C++ module for Python 2.5,

but 2.6 actually is completely Python-based. So it looks like with something this fast in the middle tier, your data layer could quite possibly be the bottleneck very quickly. What is normally, I guess, coupled with something like this to provide that sort of speed? Yeah, exactly. So this has actually been a fairly controversial point, I think, in the asynchronous web server community to the extent there is such a thing, which is do you try to make everything asynchronous? How do you handle that?

And the approach that we took was that we found that making everything asynchronous was actually just annoying because it makes your code fairly complex because it's full of callbacks. So we thought that the most important thing was that the external events, web browsers or fetching external URLs need to be asynchronous because those are things beyond your control. But local resources such as a database is actually

okay to block because you need for your database to be fast anyway. And so what we've the approach we took at FriendFeed is essentially just to run a fairly large number of front-end. And that way, we also circumvent the problem with the Python global interpreter lock, which effectively limits Python to only be able to use a single core at a time anyway.

And so what we would do is, you know, on an eight-core machine, we might just run 10 front-ends. And then we would put them all behind Nginx, which is a very fast reverse proxy. Are you guys using this just at FriendFeed, or has it been folded into Facebook properties at all? I don't believe there's any uses at Facebook. Facebook, as I guess the others will talk about, is largely written in PHP, so it's not really something that can be easily adapted for use here. So in the lunchroom, you guys kind of divide up into the Tornado and Hip Hop crowds, and So it's not really something that can be easily adapted for use here.

So in the lunchroom, you guys kind of divide up into the tornado and hip-hop crowds and throw gang symbols and things? Yeah, it's an east side, west side kind of thing. That might be a nice segue. Let's talk about hip-hop for a second. Yeah, sure. This is Hai Ping. So the major problem we're trying to solve is the CPU consumption problem on the web tier the intention is very simple we just want to speed up PHP

that way it can take a lot less CPU which means a lot less number of machines the idea was just to transform the PHP into C++ because syntax wise these two languages are very similar to each other idea was just to transform the PHP into C++ because syntax-wise these two languages are very similar to each

other. And we were just asking the question, why C++ can run faster, but PHP runs a lot slower. After the transformation, a lot of the dynamic symbol lookup can be eliminated. Things are more dynamic in PHP

can just become static in C++. And also we do some type inference. Hopefully the code can be a lot more efficient. So that was the basic idea. And it took us two and a half years, two years, about like one and a half years of development. Initially we had three people, Ian Proctor, Minghui Yao, and I. We spent one half year of coding.

We wrote a lot of code. And then it took us another one year of time, you know, six months of correctness testing and six months of rollout. So adding together, it's like one year of rolling out to all the web servers we have. So right now,

nearly all the web traffic is served by the HipHop Compile program. It's been running well. It's been running with a lot of speed up. I think it's 2x to 3x speed up. So it's been working for us. How long did you say that you were in development on the project part of it? Like a year, year and a half? A year and a half, yes.

We've been writing a code for one year and a half. And then we spent a lot of time just to make sure it really was correctly, meaning it's the same as what PHP does. And so what kind of resources did you have available when you were in development? Was it just two people or was it a small team? Three people.

So Ian Proctor, Minghui Yang, and I, we have three people. And then after one year time, we have another three people join our team. So right now we have more people. We have a total of eight people working on the compiler. We're still working very hard on more optimizations. So do you have to pre-compile your PHP code to take advantage of something like this?

You mean other people? Have other people done similar work before? So if you're going to use HipHop, does your PHP code have to run through some sort of pre-compiler to get this performance boost, or is it still a dynamic language? The compilation only happens during deployment. So during regular

development, people continue to use the interpreter which doesn't require them to compile, so they can still write web pages really, really fast. Only when the code is ready to push, we'll just compile that into a binary and just push that binary. Are there any syntactical

limitations, I guess, between the interpreted PHP code and what you guys can support from the compiled version? Very few. Eval is one of them, which isn't recommended anyway. So to support eval is pretty hard for us because it also takes away some of the optimization we can do. Other than that, there's also two or three minor places we don't

support, but otherwise almost all the features are supported. What would be the use case for someone to consider using this PHP framework? I think it's more useful if you have a large number of machines or you are running a large-scale PHP code base. You really want to save a number of machines in your company, then this might help.

The biggest gain really is just a reduction of the amount of machines needed and faster code, obviously. Yes. A lot of people were confused. Maybe this compiler can help making web pages serving faster, but that turned out to be a bigger equation. When you talk about

web page speed, you also have to count the network time. Not only from the browser to your web servers, but also from your web server to backend servers, to database servers, to memcache servers. You have to count all those network I.O. time. What we are saving is only the CPU time taken by the web servers. So depending on the nature of your web page,

that portion can be big or small. What we are trying to save is really the computation power or the number of machines. How did this project come about? Who had the idea to take off and do this? I had the idea, for sure. But Ian and Minghui, they just loved my idea. And then when we got into implementation, it's everyone's idea,

how exactly to convert different kinds of PHP code into static C++ code. Then it's really just a teamwork. We had to just work out all the details. We had to solve all the problems we had. How far did you get in the process before you started enlisting support from your team and from management?

I think I spent about eight months just by myself just to work out a prototype good enough to show people. It's very promising. It does run faster. That's the time I had the other two

people joining me. Having worked in that corporate type of environment before, I'm just always curious. Sometimes you have to show rather than tell. I'm just curious of how far you had to get in the process before you could actually prove the idea. Well, so you have to realize Facebook is a very decentralized software engineering force. Everyone is very talented. People normally can identify problems by themselves and proposing solutions by themselves.

I don't think our management never will say words to say you're not allowed to work on something especially after communication with other people. Your idea can be appreciated by other people or approved by other people. Um, no, no one said no to me.

So I was able to continue and we were able to even form a three people of a group to continue to work on this, even though we understood that's risky, right? Anytime point we could just say, you know, that doesn't work well and it could fail. So if we zoom out and we look at Facebook as it is now compared to previous to this project,

what's really happened and what's been the gain? Comparing to not having a compiler, then we will require more machines to run the same website. And there are some other benefits. After converting the PHP code into C++, we were able to build C++ libraries that the backend people can also take advantage of. Before the compiler, they were not able to call into PHP codebase

because PHP is not quite reusable by other languages. But after the conversion, we were able to build a small library that people can just take and just call into P3 functions. That's also very beneficial. So basically, we were just building a bridge between our P3 programmers and C++ programmers. That's kind of cool.

That's pretty cool. It's great that you have an environment that fosters innovation like that as well. Thank you. Speaking of innovation, speaking of innovation, I think one of the coolest apps on the iPhone is the Facebook application. Correct me if I'm wrong, but I believe the 320 open source application, the project is kind of the underpinnings of that. Who do we have to speak to 320? This is Owen. I'm currently the developer of Facebook's iPhone app.

And yeah, 320 is originally, it was extracted from Facebook for iPhone by the original developer, Joe Hewitt, whose name you probably know. So it wasn't really developed with a sort of overarching theme. It was just a set of things that he considered useful or potentially useful to other developers. And yeah, 320 powers a lot of what the iPhone app does, including its infrastructure for doing network requests. It does caching on the file system and in memory and network request queuing,

as well as URL routing internal to the app. caching on the file system and in-memory, and network request queuing, as well as URL routing internal to the app. So the app opens different views within itself through this internal URL mapping and routing mechanism that 320 provides. Other stuff that 320 provides is, like, large-scale pre-made native controls, like an interface that mimics the message-composing UI of the iPhone system's Mail.app,

which we use in the Facebook app for composing an inbox message. Other stuff includes... What essentially comes down to a reimplementation of a CSS-like layout engine, which allows you to describe the way a set of views should be laid out or the way an individual view should be drawn. Those are sort of the main benefits that 320 gives us.

How does a project keep up with the ever-involving Apple platform? iPad, iPhone, stuff like that. So 320, one of our main focuses, so I'm not the only person who works on 320, by the way. A former intern of ours named Jeff Verkooyen has also done a lot of great work on it. Adapting 320 to be usable by iPad developers is a pretty high priority for us because,

you know, it contains a lot of useful stuff, but in its current state, it's not really suitable for the iPad because it makes a lot of assumptions about screen size and a couple of other things. Like it's, it would take some effort to make it compatible with, for example, the split view metaphor that the view metaphor that is standard on the iPad. By split, you mean portrait or landscape? No, that part is fine. It's just by split, I mean having one scrollable list on the left and another scrollable list on the right, for example.

Sorry, I've kind of forgotten what your original question was. That's okay. What I wanted to know was, as you guys evolve 320 in this code base, how do you continue to just keep up with this ever-evolving Apple platform that we're dealing with between, we had the iPhone for a while,

we have many applications, lots of different opportunities out there, and now you have this bigger platform called the iPad. How are you evolving 320 to manage both platforms? It's not really clear at this point. Our work on adapting 320 to the iPad is in its very early stages

because there's a lot of code architecture decisions that we need to make, and some of them it's not really clear what the best choice is. So yeah, I don't really have much to say on that front. As far as the evolving Apple platform in the sense of new features that they introduce in the SDK, generally we sort of look at those as they come know, keep up with when they deprecate certain things, which is pretty minor. For something like OS 4.0, which where the SDK is going to have a bunch of new features, it remains to be seen like which ones of those will hook into 320. Some of it depends on if we decide to use those new features in the Facebook for iPhone app. Like that's a major driver of putting new stuff into 320

if our iPhone app requires it and 320 is a good place for it then we do that we certainly applaud you for one of the better Objective-C iPhone open source projects out there are you up to date on kind of the state of open source and Objective-C, especially in the iPhone, and what other good projects are out there that folks should check out?

Well, just to start with, a lot of people have actually made forks of 320 where they put their own modifications on top of it. And some of those are actually pretty good. They implement a lot of new features. And those forks are divergent from our implementation and we're probably not going to reintegrate those into mainline, but it's still, uh, well worth checking out as far as other, um, other frameworks for the iPhone. I don't know of any that are as sort of broad-ranging and comprehensive as 320.

I actually don't know very many at all. The ones that I do know have very specific purposes, like serializing and deserializing JSON, translating between JSON and Objective-C objects, that is, or providing sort of object-oriented interface to SQLite.

Those are the kinds of things we can make use of. But really, as far as the state of open source iPhone libraries in general, I think, you know, there's a lot less there than there could be. 320 is pretty big and in fairly wide use, but I really don't know of that many others, which sort of suggests to me that there isn't really a thriving open source community in iPhone development.

So was that all the projects? I'm looking at my notes here. We've got Scott McVicar left over. I'm not sure. So Scott, you've just got to at least say Facebook for the folks at home. I met the guys at a recent, during, what conference was it during?

At Twitter? Chirp, I guess? Yeah, it was Chirp Conference. Who else has redacted? I should mention Twitter on the Facebook podcast. So it was at a certain unknown social media conference, right? Yeah, and they noticed that my accent's a little on the Facebook podcast. So it was at a certain unknown social media conference, right?

Yeah, and they noticed that my accent's a little on the strange side and I can't actually say the name of the company I work for properly. So it's pronounced Facebook. So Scott was telling me at the meetup that maybe this is a nice segue into just general Facebook, working in slinging code at Facebook questions. Scott was telling me that you guys are all friends with Mark on Facebook. Is that right? With Zuck?

Yes. I think it was like you see him walking around, and he's very involved with the company. He doesn't like he's there. Yeah, I mean, I think Zuck even wrote a patch for one of the features or fixing something that we were going to roll out at F8 a few weeks ago as well. So you're still in the code, huh? Yeah, now and then.

What's good to have the guy at the top of the company still slinging code? Yeah, I mean, I think it really speaks to the engineering culture that we have here in terms of both what Haiping was talking about earlier in terms of being able to go try out an idea, see if it works, work with other engineers or small teams. And really the entire company and products moving forward from an engineering perspective. Our entire design team writes code as well. So generally when we get a mock-up

It's something that you can click on It's not just a Photoshop file that's delivered So I think that really has a huge impact On how we build products and how we build infrastructure A lot of people really care about what they're doing It shows through Thank you So David, your title I guess is Senior Open question, but a statement, too. So, David, your title, I guess, is Senior Open Manager?

I don't know. We made something like that. That's what your Facebook page says, and I guess at the PGA, that means something totally different. Talk a minute about your role and kind of how you heard GATS or sling open source at Facebook? Yeah, I mean, I'm really focused, and my team's focused on making it really easy for anyone at Facebook to use open source, to use standards, to create open source technologies, to release them, and just helping the company make sure that we do a really good job of that.

And not just from an engineering perspective, but making sure that's pulled in from a marketing perspective, or recruiting, or even legal stuff now and then. Open source licenses are a lot of fun to go and understand. But really that fundamental goal, making open source and standards really easy to use at a company, easy to create, easy to release. Going and building developer communities around them. Lots of innovation happening on the platform, too. So you mentioned F8 earlier.

You guys had the OpenGraph and OWASP 2 that came out of the conference, and I've seen a number of wrappers for both emerge practically overnight for just about every language out there. Why don't you tell the folks at home what each of those aims to solve? Well, sure. I mean, so the Graph API is a really simple API that allows developers to go and interact with user data that users have given them access to, being able to go and sort of, it's extremely restful.

A lot of ideas were inspired by the FriendFeed API. And then we use OAuth 2.0 for all of our user authorization to that. So really got involved in helping drive the OAuth 2.0 standard inside of the IETF. It's really simple. If you've played with OAuth 1.0, you had to work with signatures, you had to figure out multiple types of secrets and tokens.

With OAuth 2.0, it has specific flows, whether you're in a web browser, a desktop app, a phone, an Xbox, you get an access token, then you just make API requests over SSL using that access token. So that's been going extremely well. Developers are really loving both the graph API, as well as just how much easier OAuth 2.0 is to work with. And then we also released the open graph protocol, which allows you to add some basic metadata to any web page so that users can go and connect to it inside of a social graph and so that they can like that page and that that page is really represented well with a graph so that we

understand what type of page is it is it a movie that the user's interacting with or is it a website is it an article as well as some the user is interacting with or is it a website? Is it an article? As well as some other information such as title and stuff like that that you want to know. I noticed on the Open Graph protocol page that kind of gives the overview that one of the examples is IMDB.

Have they implemented anything with Open Graph or was that just kind of a use case? Yes, IMDB still has some of the meta tags that we were playing with and prototyping before F8. They're working on going and using the OpenGraph protocol tags themselves. If you check out Rotten Tomatoes, you'll see them there. If you look at CNN, you'll see them there. They're definitely starting to spread around the web. How does OpenGraph compare to other technologies that are kind of similar,

like OEmbed or perhaps microformats? So I think microformats are probably easiest to start with. Microformats really came about by looking at how are people marking up semantic information in the bodies of web pages and trying to create some patterns around that. We're using the RDFA syntax, which is basically a way of saying, like, you put the quotes a little bit different than if you were using microformats.

We tried to reuse the microformats H-card schema when we're talking about contact information, but really just wanted something that was dead simple for developers so that they can literally copy and paste four meta tags, place them into the head of their page and have it work, not have to go and dig into the body of the HTML, not have to worry about namespaces and different schemas to combine things together. And the other thing you asked about was OEmbed.

I guess OEmbed is more, it's an API. So you go and you take something like a YouTube video page, you discover the OEmbed endpoint, and I think you actually make an API request about it, and that returns a JSON object describing some metadata. So I think the Open Graphs protocol really aims to be simpler than that from a developer perspective, not having to stand up another API, not having to go and make an additional HTTP request. Adam and I both make our livings and our day jobs

on the front end. And just my initial take was, with microformats, I got excited when they first came out. But to tell you the truth, just even as someone that makes his living doing front-end code, every time that I wanted to construct one, I'd have to go out and look at an example. It's just complex.

I have the book right there next to you. I got excited when I saw OpenGraph because my initial reaction when I saw the meta tags was, wow, how could we have thought of this sooner? Thank you. I mean, there's really a whole team of people, and I think a lot of this comes from the fact that we were working with a lot of different partners and getting feedback from large publishers on what they were willing and what they weren't willing to go and do.

So one of the things that we discovered is there's a link tag where it's link rel canonical. And the goal of that is that you say this is the canonical URL for a page. So the example is if you have a bunch of query query parameters then a search engine can know ignore the query parameters this is the actual canonicalized url and search engines understand it and so we were thinking of using this tag instead of the og url property but when we went and started talking to some large publishers they were afraid that adding this tag would have potentially negative effects on their search engine optimization. And so really wanted something separate rather than having something that was tied into what they were already doing from an SEO perspective.

It's about that time where we ask everybody what's on their open source radar, more or less. What's out there in the open source world that just got you excited about what you're doing? And we just kind of wanted to go around the room. I guess we can David, if you want to lead this, you can and just kind of go in turn with who wants to go next. But pretty much the question is

what's on your radar in terms of open source and what are you just down to play with? So what's on my radar? I guess for me it's this... We're seeing more and more examples of how you can really use open source to go and scale large websites.

And it's no longer that open source is just the text editor that you use on your computer or your mail client or only your database. But I think open source is really moving beyond that traditional LAMP stack and that you're going and seeing other technologies which are becoming really relevant in building scalable, dynamic websites today that are open source, that are working really well. So I guess sort of that evolution is pretty interesting to me.

And I guess one question you probably gleaned over earlier, which was probably something you kind of hinted on there, but how has GitHub as part of open source impacted how you feel about open source now and where it's going? I mean, we love GitHub. It's an incredible web interface to go and manage a project, to go and browse through source code. And I think one of the most interesting things is how GitHub really went and embraced that idea

of forks. So instead of saying that forks are a bad thing and trying to ignore forks, GitHub really gives you the ability to easily see who else is going and working with your code, even if they haven't directly submitted a patch. And so that really allows us to go and see what are other people doing on a project like Tornado or on a project like 320 in a much more proactive fashion. Very cool. And I

guess, who would be next to answer the radar question? And if you want to answer the GitHub part of it too, you're welcome to. So I'll answer it, but I'm going to cheat. So I'm going to say I think maybe what I'm most excited about in the open source world is a couple of the projects that are

still in the pipeline here at Facebook. I think there's a couple of cool things that'll be out in the coming year. We love scoops. And I mean, I agree with everything about GitHub. I think it's a great example.

I mean, not only is it a really great product, but it's a great example of how you have a space that's been around for a long time, you know, things like SourceForge and everyone kind of, or Google Code, and everyone kind of assumes that, you know, that's as good as it gets and then all of a sudden

someone comes along and does something that's just fundamentally different and once it's there, everyone sees it and you're like, wow, why didn't we do that sooner? So it's just a cool company. I really like the network overview on GitHub where you can see

all the forks and it's got a visualization of the branches and what commits went into the various branches so you can pull things back in. I think that's pretty awesome. Yeah, this is Owen again. And sort of related to that, the thing that's impressed me most

about the open source community that's grown up around our stuff is the quality of the contributions that we get back. We've gotten plenty of pull requests from people who have forked 320 and GitHub. And a lot of those we've actually integrated back in. We have at least 16 separate contributors to mainline right now. Most of them are in the form of just little bug fixes and patches.

But that makes our life so much easier. Like if someone reports a bug to 320, that's one thing. But if someone reports a bug and includes a patch that fixes it, that's even better. And that happens actually fairly often. And that's been one of the best things about open sourcing 320. As GitHub and just having open source project profiles available impacted the hiring process

over there at all? I don't know specifically. That's a good question. And that's all we get. I mean, I can make up an answer if you want me to. No, I was actually, I mean, I was interested in seeing a blog post by Matt Biddulph, who created Doppler and then sold it to Nokia a few months ago,

where he was actually going and analyzing GitHub network statistics for each city, looking at who were some of the most interesting contributors. And he was using that, I believe, from a hiring perspective. So there's definitely something there if you want to play with it. Well, we certainly appreciate your time today. Anything to add before we wrap? I mean, I guess the only other thing is Google Summer of Code. We're actively participating in that this year.

All three of Hip Hop, Tornado, and 320 have students working on it as part of Summer of Code. So that's definitely another thing that's really exciting for us. When does that take place? So Google Summer of Code is, I think it's actually starting within the next few weeks if it hasn't started already. It's entirely distributed. So Google goes and helps projects find college students and gives them an internship over the summer remotely to work on open source.

I just pulled up the page down here now. That's pretty cool. Wow. All right. Thanks, guys. We appreciate it. Yeah, and facebook.com slash open source if you want to learn more about our projects and other stuff that's going on there.

And also g.com. Yeah, also GitHub.com forward slash Facebook as well. It's basically the same thing. But hey, thanks guys for coming on the show. We really appreciate all you had to share with Open Source. We appreciate your wisdom and standing on the shoulder of giants and out there doing what you're doing. It's super awesome. We certainly appreciate it.

Yeah, thank you for having us. Thank you for listening to this edition of The Change Log. Point your browser to tail.thechangelog.com to find out what's going on right now in open source. Also, be sure to head to github.com forward slash explore to catch up on trending and feature repos as well as the latest episodes of the changelog. Outro Music I'm out.