The Changelog: Software Development, Open Source - Travis CI, Scaling Apps, Riak (Interview)

Episode Date: March 6, 2012

Wynn caught up with Josh Kalderimis and Mathias Meyer from Travis CI to talk about hosted CI in the sky, scaling apps, and a little Riak.

The Change Log is brought to you by Pusher, and they're looking for system engineers who specialize in evented systems. If you're happy to work with multiple languages and you have experience in building evented systems, our friends at Pusher want to talk to you. This is a real hands-on job at Pusher, so relocation or London-based residents are the prime targets are seeking out. Send a cover letter about how you will make Pusher more awesome, your GitHub profile link, and your CV to jobs at pusher.com. Don't push me away! We're also up on GitHub. Head to github.com slash explore. You'll find some training repos, some feature repos from our blog, as well as the audio podcast. And if you're on Twitter, don't follow the ChangeLog show. Follow the ChangeLog. And I'm Adam Stack. And I'm Penguin, P-E-N-G-W-I-N-N.

Fun episode this week. Talked to Josh and Matthias over at Travis CI about hosted continuous integration in the sky. That was a very fun conversation. It seems like they're trying to do a lot of fun stuff, but they're also supporting a lot of different languages as well. Yeah, Ruby's been supported out of the box since its inception, but now we've got PHP, Python, Perl support, as well as Java, Scala, the usual suspects.

So if you're looking for a hosted CI solution for your open source project, private projects are coming, and you don't want to stand up your own Jenkins box, then you might look at Travis CI. It seems like it's pretty easy to get started with it, too. Yeah, you just go in and configure it right there in your GitHub project

page, and you can embed an image in your readme. You see this a lot of times where it has the build status right there in the readme of the project. It just has the image tag phone home to check the status of the build, and you see red, green, or yellow right there in the README. It's pretty cool. Colors accordingly. And then not only that, but they're also using Pusher,

which is quite nice, and they're our sponsor. Yeah, we talk about that in the episode. It's a neat way to power the real-time UI that they've got in Travis. You can just sit there and watch the output from your project or someone else's project as it's happening. Well, sounds like a fun episode. You want to get to it? Let's do it. We're talking today with Josh Matias from the Travis CI project. So you guys want to introduce yourself and a little bit about Travis.

So Josh, you go first. Well, my name is Josh Calderimas. I'm one of the core team members of the Travis CI project. I've been working on the project since February 2011 with Sven Fuchs, and we've since grown to four team members. Matias? My name is Matthias Meyer. I am the infrastructure guy on the Travis team,

and I've hopped on rather recently. And I'm also the author of the React handbook. So, yeah, that is my introduction. Definitely want to talk more about React later in the episode. So for the folks that don't know, Josh, why don't you give a little bit, a big picture of what Travis is and what it does. Well, Travis CI is a distributed continuous integration system for open source projects. It's also technically open source as well, but we try to emphasize that it's for open source projects. So if you think about Rails or Symfony, even no JS Rubinius, they're all using Travis because not only are your logs out in the open, but it

just makes testing dead simple. So we've been testing Rails for the last eight months now, nine months. We've done two and a half thousand builds across different matrix variations so we can help test rails projects for example against different ruby versions previous to rails 4.0 which is in master at the moment or still in development we were testing against three different ruby versions 1.8.7 192 and 193 and now that rails master has moved to 4.0, it's only against 197. Oh, sorry, sorry, 193. We've got some big plans on how we can help increase testing as well to make it even better. For example, testing against different architectures for systems, if it be Windows or 32 or 64-bit Linux. But essentially, we're just trying to make testing easy for

open-source developers, because who wants to actually set up a CI system for their little library in-house and hire a server or remember to run it against 10 different rubies? You guys have some big news. Last couple of weeks, you know, support Python. So adding Python support was fantastic. It was a great reception in the Python community. It was actually one of our biggest growth days. It wasn't just Python support that we had last week.

It was Perl. And in fact, a week earlier, we added Java, Clojure, and Scala, and Groovy. We've actually technically had all of these languages supported for a long time now, just we only supported one version of the language, only one VM version. So if you think about when we added Ruby, we had Ruby 1.8.7 and 1.9.2 and 1.9.3, and that's what we call first-class support. And for a long time, Python and Perl didn't have first-class support. You could only use one variation of the VM, of their VM.

And now we essentially support, I think it's 10 different languages, with most of those having different versions of the language available. So even when you think about Ruby, you can test against Ruby head and Ruby 193, JRuby in 1.9 mode, Rubinius in 1.8 mode and 1.9 mode. There are so many different possibilities of what a normal developer can test against now.

And we generally think that every language developer, all open source developers should have the benefit of Travis for their open source community. So with all of those languages, that entire list of languages that you support, what about the different testing frameworks for each of those languages?

So part of the idea that we've got in our worker at the moment is what we call the builder. And Sven extracted that to its own repo, so it makes it very easy for other people to add different language support. And the builder encompasses a lot of standard defaults that that community might use. If we think about Ruby, we know that

bundler and using rake are some of the very basic defaults that we use. So we can detect if there's a gem file. If there's a gem file, we'll run bundle install. If there's a gem file, then we'll use bundle exec rake. And we apply that to different languages in different ways. So in PHP, there is no standard way of doing dependency installation, but there is a standard way of running the tests. And in Python, there's various standards for each of these, but the general consensus within the community is, instead, we should just prompt them to define that in their Travis email which is the configuration for Travis for your project when we had Dr. Nick on a previous

episode we brought up Travis had just come out or I think you had just contacted me you guys were just getting started with it and he's a big Jenkins fan. What's the selling point of Travis over Jenkins? I'm definitely going to leave this one to Matthias. He's been working with Jenkins a lot recently. Yeah, I just used Jenkins on a project recently. And well, I got to say, the ideas are very similar. It's continuous integration. The ideas behind that are very straightforward. And you find a lot of similarities in Jenkins compared to Travis, for example.

But what repelled me a little bit from Jenkins was the user interface. And the point of Travis is to hide all that from the developer. So in Travis you only have a YAML file where you define your complete build metrics

and stuff like that. And in Jenkins, you do all of that through user interface. And that's not the main selling point, but it's just, I didn't have a lot of fun using that user interface because there's a lot of options you can set on Jenkins and it becomes a bit of a pain to use for me. That's why GitHub, for example, built their

own janky bridge for Jenkins so that they can hook up their U-bot to Jenkins and not worry about the user interface at all. Speaking of the user interface, the Travis CI interface is real-time powered by Pusher, I take it. What sort of challenges have you found just scaling that for so many projects? It's been an interesting ride over the last year. And if you look how the code has changed over that year time, so it's been one of the talks I've given at conferences about how we've really deconstructed Travis into separate deployable apps. And that's helped us scale tremendously.

We run... We're pretty much entirely on Heroku for the platform part, and we use dedicated workers or dedicated machines with the most inefficient way of using a VM technology, which is VirtualBox, because it was handy for development. It's definitely a pain for maintaining, but it works. So on the platform side where we're using Heroku, we're using 192 and what we call the Travis CI.

It's just Travis CI. And that's using Unicorn and 192. We've also got Travis Hub, which is a JRuby app, and that's using the AMQP Java library for its communications. So we're using JRuby for Travis Hub on Heroku as well, and that uses Hot Bunnies, which is actually a Ruby wrapper around the Java AMQP library. That's actually allowed us to scale quite well because we just use native threading and we spin up about 10, 12 different threads

for managing the different queues. And we've also got another project which isn't fully live yet called Travis Listener, which allows us to take the various components of Travis offline, but still listen to the GitHub pings. And also behind the scenes, we're using RabbitMQ for messaging between all of these projects or applications. I know it's not a fair comparison since you usually have to set up

your own Jenkins box as opposed to a hosted system like Travis, but what's the comparison of setup if you wanted to set up a project on CI between the two? I wish I could use swear words right now, but I would highly recommend people don't try and use Travis internally right now because it is a huge undertaking. It's much harder than just setting up a Jenkins in-house.

It's not to say that I'm a huge fan of Jenkins. What I am really appreciative for them for, what I really do respect them for, is that they've come a long way in building a CI system that is easy for people to install, or generally easy for people to install, and it works. But if you've got an open-source project, let's say a RubyGem project out there, and you want to make sure that it works across several different versions of Ruby, what's the process to set it up on GitHub with Travis?

So to set up your project on Travis is tremendously easy, and I think this is definitely one of the reasons why it's taken off. There is no huge setup process to go, and we try and get it up and running. And also, we automatically detect a lot of the defaults. So it's as simple as logging with GitHub on Travis. You can see all of your projects, you can turn on the service hook, and then you can either push your code to GitHub and will automatically pick it up, or you can go to GitHub and you can send a test hook. The trouble is that if you're a Python Perl or Java developer, then you'll also need to definitely, you'll definitely want to add a.traverse.yml file. Even if you're a Ruby

developer, you'll want to add that file because by default, you don't want to just test against 187. You'd like to add you know different rubies to be um you know quite compatible to make sure you've got compatibility which version of ruby gives you the most fits to maintain with travis sorry how do you mean with uh so how many versions of ruby you support? We're talking MRI. Oh, I think we're up to 11 different versions of Ruby. Two of them are temporary. Yeah.

Which versions of those are the most problematic to work with Travis, just getting it to run? I'd actually, well, we, so Michael Klischine, so we've actually got kind of five team members on Travis. We've got Constantine House and Matthias, who recently joined, Sven Fuchs and myself, and also Michael Klischine. And Michael Klischine has done an amazing job in maintaining our workers and our cookbooks.

The great thing about using cookbooks is that anyone can start contributing and saying, oh, can you add the service here as a cookbook for it? So Pat Allen from Thinking Sphinx fame, he did the cookbooks for adding different versions of Sphinx so he could then test against different versions. Michael has put a huge amount of effort into getting the cookbooks and the recipes right for RVM testing or for adding different rubies into RVM. And we have to give huge credit and huge thanks

to the whole RVM team. They've put a lot of work in making or adding extra configuration options so we could get adding Rubinius in 1.8 and 1.9 mode and jruby in 1.8 and 1.9 mode so it makes it a lot easier within your yaml when you're travis yaml to say i want to test against 1.8.7 1.9.2 1.9.3 and jruby in 1.8 mode jruby in 1.9 mode etc souby in 1.9 mode, etc. So recently you had the Love Travis campaign. How's that going?

I'm really happy with how that's gone. I'm so appreciative for the community and to the community. They've helped us so much. A lot of companies have jumped on board and helped us with just being able to pay our rent effectively. Sven and I have spent the last six months kind of, I guess, job-free just working on Travis, and being able to have the community help us out like this is just amazing. We've got some big features in the pipeline.

Konstantina's currently working on the pull request support I think that's going to be a game changer I think this is going to explode Travis as well we're going to have to throw workers at it like I can't believe if you think about Spree or

Symphony even Rails, Rails is a perfect example the the usual workflow of a lot of Rails core is they'll get a lot of pull requests come in, they'll check through the code on GitHub, and then they'll hit the merge button. And if it fails, then they have to go into their console and they'll have to revert that and then push it back. And testing pull requests will now mean that you do a pull request for Rails, we will test it, and then we'll leave a comment in the pull request

to say if it passed or failed. And this will just be an absolute game changer for open source projects. It means accepting pull requests, it's all about, did the tests pass? Did Travis say it passed? And can I look over the code to see, you know, is it clean?

Is it doing things right? And then you can hit merge. Yeah, I'm surprised that all the low-tech solutions we're finding to integrate with GitHub around like badges and the readmes for build status and things of that sort. And you're talking about build comments. Do you wish GitHub had more like meta hooks that you could hook into at the repo level? Actually, it's only because of Rick Olson, Techno Weenie,

that we're able to start adding the support. We talked to him last year at Frozen Rails about how we really want to work on pull request support. And the thing that was partially holding us back is that we needed some APIs for when people add pull requests. Because for us to pull GitHub over the 6 6 000 projects we've currently got on there to see if a pull request is being added isn't really going to scale on our end and we just needed to get some notifications if a pull request is added and if it's updated so he worked really well he added that specifically for us which was fantastic and now we can actually

work on that feature which constantine is is most of the way through i think we're probably about two or three weeks off until we can start rolling that out incrementally once we've got those notifications then when even when you update your pull request, we can then retest it as well. So think about pull requests as a discussion. You add a little bit of code, we'll tell you it fails. You add some more code, now it passes. You add another feature to that code, it fails again. It becomes something where all of those Travis comments are going to stay within the discussion, and then eventually it will be, hey, it passes, time to merge.

What's your roadmap for private repos and when you can turn that on for everybody? Well, we're looking into the options right now. We're actually actively, Swin just started actively working on it on getting what we're calling Travis Pro Codename Magnum setup. And we've been looking at different options on how we can make that scale better

because obviously when you offer commercial solutions, you have to think a lot more about SLAs and ensuring a quality of service and stuff like that. And so we've been looking at the options of what we can use, for example, instead of VirtualBox for virtualization on machines. And, well, we figured out ways

to actually shorten the path to getting to an alpha. So I'm not sure if I'm ready to announce a date for that yet, but yeah. How much more complex of a problem is it? Do you have to add multi-tenancy to cover things like that, or is it the same platform

you're just having to worry more about and how to scale that? Yeah, it's actually the multi-tenancy thing. That's what's going to keep us busy the most. And to make sure security issues and stuff like that, because obviously we're going to be dealing with people's private source codes and stuff like that. because obviously we're going to be dealing with people's private source codes and stuff like that.

But yeah, on the application side, it's mostly a multi-tenancy. And that's what Sven is actually currently working on. So if you want to speed that up as a listener, you can donate, pitch in. Open source is free, but it's not cheap. So Josh, tell them what they've won if they do donate to Travis. Well, we'll send you a beautiful picture of Sven Fuchs. Well, we've got a whole range of donation options. We just love anything. If you want to donate a buck, that's perfect. If you want to donate the top, which is like 500, that would also be

amazing. In fact, the one thing that we haven't really highlighted enough on the site is that we had a whole range of amazing developers come out like Aaron Patterson, Yehuda Katz, Jose Valim, John... Now, I keep pronouncing John Leighton's name or John Leighton's name is Leightonon and he loves that because he's like, oh, that's so Leighton. But you can also program with John Leighton as well. So if you donate 500, you'll get an hour online pair programming with them. Aaron's ready to take a day off, you know, the job, off his job just to help out with this.

Sponsored pair programming. I love it. And you get some credit on Travis Pro when we're all ready to go live and you get beta access. But anything that anyone wants to donate is just fantastic by us. So speaking of sponsorship, Pusher backs this podcast

and we're so very thankful for them. Matthias, what has Pusher meant to Travis? Well, mostly allowing us to have a pretty nicely responsive real-time user interface. That's probably, well, that's the one thing we really use it for and that's what it's been really fantastic for,

to have a really short loop user feedback that when a build starts, we can immediately pop up the new library and the user interface and we can keep, you know, as a build proceeds, as new log output comes up from the build, we can immediately push it out and the user interface, and we can keep, you know, as a build proceeds, as new log output comes up from the build, we can immediately push it out to the user interface,

and it's just fantastic for that. Complete with syntax highlighting and color coding. It's awesome. I love the UI on Travis. So, Matthias, you're also an author of the React handbook we mentioned earlier, so what's the backstory on that? Well, the backstory is that I wanted to write a book

on NoSQL databases. And the other part of the backstory is that I worked for Basho for a while. And yeah, that book kind of came about to be the collective, well, brain dump of everything I learned about React up until that point. And it still is so I'm still working on it to get

updates out and to produce more content for example just recently they released React 1.1 and 1.1.1 is actually still in the works and they added some new features which I would like to cover in the book as well so I'm

pretty much still working on it it's still a work in progress but it's been really fantastic to sell. And I self-published the book, kind of built my own production chain for the book and it's just been going very well. Much better than I

originally expected actually. Is React your favorite NoSQL solution? Oh, the answer to this could get me in trouble. I don't know. It depends, right? Yeah. I don't really have a favorite database in general. To me, I don't make the difference that much anymore

between NoSQL databases and relational databases. To me, they're all just databases, and they have their strengths and weaknesses, and that's pretty much how I look at things right now. So, but I do like React a lot. What sort of use cases is React just made for? Well, the one use case that most people that come to React have is to really have high availability, to have a database that can, you know, even when stuff falls on its face,

when servers fall down, you still have availability to accept read or write requests no matter what. And that's where React really shines. And that's where I really like it for. And yeah, the problem with that is that it obviously brings a lot of trade-offs and a lot of people have some difficulties, you know, getting into these trade-offs or changing the mindset from relational databases where you have stronger consistency, where you have different data models and things like that. And that's where people struggle a little bit with React. And that was kind of the point of the book originally, to make that part a lot easier. Because it's just right up to right

until the end, I added stuff on how you would model data for eventual consistency, for example. And that was kind of fun because it's kind of mind bending to do that. And just to show people how you would, you know, how you would do that with React. And it's just what you have to do when you have a lot of rights and you have a lot of updates on the same data. You just have to prepare your data for that. And but yeah, in scenarios like that where high availability and fault tolerance is just really important. And it's React is just awesome. I love it for that. But yeah, in scenarios like that where high availability and fault tolerance is just really important, Rearc is just awesome. I love it for that. I want to take a moment to talk to you about Hover.com, our latest sponsor that we're actually quite stoked about because

Wynn and I, we use GoDaddy primarily for all of our domain purchases in the past and with all the recent fiascos, and I'm sure you know what I'm talking about, there was some major need in the area of domain registrations and taking care of that stuff. So lo and behold, we actually found Hover, decided to start using them. They became a sponsor of the show, which is kind of neat as well. But if you go to hover.com slash the changelog, you can save 10% on all orders.

They actually have this valet service that will move your domains for you. So if you're at GoDaddy or anywhere else, doesn't matter where, you give them a phone call, you talk to a human being, they go through all the details with you. Now, obviously, there's fees applied to moving your domains which they can help but that service to you is no cost whatsoever they do all

the valet all the moving all the mx records all the dns reset up for you and that's no cost you go in and give a final approval and they go ahead and take care of it if you have lots of domains like we have like 30 or more domains, they can actually break that up into chunks so it's not just one big hit, and it's quite nice. So Hover.com is a fantastic new sponsor. We gladly thank them for supporting the show, and we hope that you do the same.

Again, you can use the change log to save 10% on all orders at Hover.com or just go to Hover.com slash the change logging to be grandfathered right in. Seems like almost every app has a big data problem now. What about Travis? What sorts of data do you guys have in the back end or is it mostly just within Git indexes? I think the biggest data we actually have are the result logs. For example, the Rails log when a build fails

because something changed in the database adapter and everything breaks is that a Rails log can grow quite significantly. It's more than quite significantly. We had some huge issues for a while where if

something went wrong in Rails or even other big test suites. So Rails, their test suite, when we first approached it, tried to get on Travis, ran for two hours. So their feedback loop was tremendous. And the first thing that Sven worked on was to break that up into five units. So you can now have 20-minute units. So if we could have them running concurrently, it would only take 20 minutes.

But it's still active record, for example, has a huge test suite, which will take a long time. And if you have warnings pop up on every single test, it can grow a 100-kilobyte log to, I think, our largest log that we saw clog up, Travis. Like, it brought the entire system to a halt was 184 megs and we're storing that in a single database column um and it it was quite funny like we're like oh god we're having another warning and someone's test suite like someone would update their rails and now it was all over their test suite, and everything would stop. So we added log limiting to that. So now a test log can only be a maximum of four megs,

and then we add a warning to the bottom to say, I'm sorry, your log is too long. It's over four megs. We've had to halt the test. But yeah, other than that, Travis doesn't really have a big data problem. I don't know about you, Josh, but I'm fine with that.

I'm quite happy with that. What parts are the most challenging to scale? Where are you focusing your brain power on right now? Well, the one part that is actually harder to scale is being able to scale out the hub part and making it redundant and fault-tolerant in that way.

But the solution to that is actually not that hard. It's mostly related to an ordering of events. So, for example, when we push stuff, when the worker just continuously keeps pushing a lock output through pusher out to the user interface. And right now it's assumed that that always comes up in order. And the simplest fix to that problem would be to make sure it doesn't come up in order, if you know what I mean. And that you have some sort of buffering in the client

that can keep track of sequences where the log currently left off and where this chunk that just came in belongs. And if there's a hole in the data the client still has, it holds out on displaying that chunk of data until the rest arrives and stuff like that. So from my point of view, that's the hardest part to scale. The rest is pretty easygoing. So I wanted to get personal here a little bit. So Josh, you are quite the globetrotter.

You speak at various conferences across the globe. What areas have the most vibrant Ruby community that you've noticed? The States is completely vibrant. It's been vibrant for a long time now. What I was surprised about was I went to Ukraine and Russia recently source and how it can benefit them. There's probably a little bit of hesitation to jump in and make as much of your code open source. And I think that's where a lot of the Ruby community

has kind of grown. It's like by having an open source movement, we've become very active of talking online and pull requests and issues and helping each other. I really enjoyed the Cape Town guys as well. Cape Town in some ways reminds me of New Zealand in that it's quite far removed from a lot of areas. Like it's a long flight from Europe. It's a long flight from America.

It's a long flight to Australia. But it reminds me of New Zealand a lot in the wineries and the friendliness of the people. And it's got a fantastic developer community where Ruby Fooza, which is run by the Mad Mimi guys every year. It's a fantastic conference. If there was a conference I could recommend some people to go to, it'd be head to Cape Town and stay there for a month and just enjoy. That's a good question of each of you now. Who's your programming hero?

Matthias, you go first. Josh, can you go first? I was going to be cheesy and say it was you, Matthias. Then I'm going to say Josh. Personally, I don't have heroes. I have a lot of people who I respect for doing what they do. But I don't have any hero in particular, I'm afraid.

Actually, I think I'm going to go for Jose Valim. He puts out a lot of code. He puts out a lot of code, but he also puts out a lot of community effort. There's a lot of code, but he also puts out a lot of community effort. Like there's a lot of main, you know, you can code and you can commit and you can, you can push to GitHub, but it's also about being available and asking questions, answering questions and mentoring people. He was one of the, he was the guy that helped me get a lot of commits into Rails. Like I'd be sitting on IM with him and I'd ask him questions, how to fix this,

approach that. And he doesn't have to do that. People have their day jobs. But he puts in a lot of love and helps a lot of people out. So I really respect that. So Matthias, let me put it a different way. If you could pair a program with anybody in the

community, who would it be? Hmm. I still have to think about that. Well, Matias is going to email, so we'll put it in the show notes. So when you're not hacking on Travis, you're not hacking on React, what open source software out there just has you excited that you just want to play with? Well, I can start with that.

That's what I'm actually prepared for. Well, currently, I'm very interested in distributed message queues. So currently, I have my eyes on Kafka and Castrell. Castrell is kind of what Twitter built as a replacement for their RubyQ Starling, I think, if I remember correctly. And Kafka was built at LinkedIn, and

they're both pretty fantastic. And I would definitely want to spend more time playing with them. And also with Zookeeper, which powers Kafka in a way. Zookeeper is like a distributed process coordination framework. And it looks really interesting.

That kind of has me really excited. But the other part from the Ruby world I'm excited about is Tony Arcieri's Celluloid and all the libraries he's been pushing out. He's been pushing out an amazing amount of code. And yeah, I definitely need to play with that more because Celluloid in particular would be a nice fit for some of the things I have in mind

for some parts of Stravus. I completely agree there. I'm really impressed with celluloid and Sidekick as well. But that's by Mike. Mike's a friend of the show.

Basically, Sidekick is rescue in a more thread-safe manner. Is that right? Well, it's based on Celluloid, but, yeah, it's an in-process rescue, if you will. I think a lot of Ruby developers at the moment are still a little bit hesitant to use threading because threading is seen as, well, you know, there are threading issues all over the show.

How do I solve that? How do I work with it? And I think we need to kind of get over that hang-up and learn how to use threading properly because it has, you know, if it's implemented in a VM like JRuby or Rubinius, you've got a great, a whole set of fantastic concurrency models available

which can really utilize the VM in many different ways. So we shouldn't be scared of that, and that's why I really like Sidekick, is that instead of going for this process model, we can use it within a single code base in a single instance. We don't have to do a whole lot of crazy stuff. We've got a great threading model available. Are tools like

this going to just make it easier to architect our applications as just small atomic services that we can consume? How many services are in Travis? Put it that way. It's not one giant Rails app. We've got four deployable apps, and we've got a total of about 11 GitHub repos.

I just added new relic messaging to the Travis Hub today to test it on staging, and it's interesting to see what's going on under the hood and what are the slow bits. But we're using things like CouchDB at the moment for some of our archiving. We've got Pusher, IRC notifications, email. It's utilizing a lot of services. But we've got four distinct Travis services.

I can talk about external services we use, but I'm not sure. Sure, go for it. Well, yeah, we have a couple of internal services, and just as Josh said, we use New Relic for monitoring the performance and stuff like that. Well, I'm a big fan of using external services myself

because I've been... Well, we're building one, and I've always been building infrastructure services. I recently added Librado's metrics, which is a pretty nice way to get real-time graphs and stuff that happens inside of Travis. And we're using a pretty funny setup for that. The metrics are regularly dumped to our log files, and from there they're aggregated into a log collector, and from that log collector they're sent to a tiny app on Heroku

that aggregates the data again, and then it's pushed from that app into Libratos metrics. So our app itself is totally oblivious to the fact that there is some metrics collecting going on, and it just dumps them into the log, and some other part of the service will take care of it. So I just increased complexity of Travis by a lot. But it's pretty transparent.

It doesn't matter to the app if the metrics actually end up somewhere. And it's just a really nice way. And I have to say, Eric Lindvall from Paper Trail and 7Scale, he built the libraries for that, and I'm very thankful for that. And they're actually both open source.

And what else do we use? We use Pusher for the real-time communication. We use RabbitMQ as a service. We use... Josh, what else do we use? We're using Postgres for the database, which I really love. I think those are the majority of the services.

I'd like to look into using Elasticsearch for some of our search and faceting. I'd also like to potentially look into ZeroMQ in the future for our job queuing. RabbitMQ is great as a messaging protocol uh it has it's a little bit tricky to put in really fine-grained controls like if we wanted a worker to say give me you know these are my capabilities give me x job if it's available otherwise give me y job we have to do we have to actually set up two different queues, and then we have to set up...

Either way, what we have to do with Rabbit is more constricting than what you could do with maybe Xero, where you've got a lot... It's much more bare bones, so we can put in a lot of our own custom business logic for that. I'm hesitant of adding XeroMQ, though, because it adds extra complexity onto the Travis model. We're trying to

keep it as simple as possible as well. I think that's the main selling point from the outside looking in is I'm not sure that you're competing with other CI platforms as much as you're just growing the tent for folks that may not be doing CI just to make it dead easy for them to add CI

to their open source projects that are hosted on GitHub. We're excited about what you guys are doing and the whole team and look forward to see what's coming out of Travis. And if you're out there listening as a user or as a company and want to pitch in, be sure and go to love.travis-ci.org. Can I do a really shameless plug though?

Sure. I actually really want to thank a lot of our sponsors that have helped us out. So I really want to thank a lot of our sponsors, like Wooga, Bendyworks as well. Bendyworks built a really cool iOS open source app for Travis, so you can check out the log streaming live on your phone. Also, Cloud Control and Zing,

these guys have really helped us out. Heroku and SoundCloud, NetApp, MongoHQ. I mean, the list goes on. CanBannery, Ticket Evolution. And this is, I'm probably not saying this right, but Zweitag? No, you said it right. Oh, perfect.

These guys have really helped us out, and we've raised a lot of our money through company sponsorship. So these are the companies that are helping keep Travis alive and helping us pay our rent. So if you're either looking for a job or just want to send some thanks to them, that would be fantastic. Thanks for joining us today, guys. Our pleasure. Thank you. See it in my eyes

So how could I forget when I found myself for the first time Safe in your arms As the dark passion shines