The Changelog: Software Development, Open Source - Celluloid and Concurrency (Interview)

Episode Date: May 31, 2012

Wynn talked with Tony Arcieri, creator of Celluloid, about concurrency in Ruby and his thoughts on Erlang, Clojure, and design patterns.

This episode of The Change Log is brought to you by Pusher.com. Pusher is a hosted API for quickly adding scalable real-time functionality to web and mobile apps. If you're building anything that needs to get data from the server back to the client asynchronously, you need to check out Pusher. They've got a number of tutorials to help you get started, everything from a quick start guide to building a real-time chat client, push notifications, activity streams, and more. We'll see you next time. www.thechangelog.com Welcome to the Changelog episode 0.8.1. I'm Adam Stachowiak. And I'm Winn Netherland. This is the Changelog. We cover what's fresh and new in open source. If you found us on iTunes, we're also up on the web at thechangelog.com. We're also up on GitHub. And I'm And I'm And I'm And I'm

And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm

And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm

And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm

And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm

And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm

And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm

And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm And I'm YNN. Fun episode this week. Talk to Tony Arcieri, aka Bascule I believe on Twitter, about his project

Celluloid and D-Cell and Real. Basically multi-threaded programming in Ruby. This seems to be a go-to project for folks building these types of applications in Ruby. Mmm. Mmm. Stuff I'll never use, huh? Yeah, pretty much. It's interesting

to some folks out there. We mentioned it in the Voice over IP, the adhesion never use huh yeah pretty much it's interesting to some folks out there it's uh it we mentioned it in uh the voice of our ip the adhesion episode i had to think about the project name the adhesion episode 079 was the last time we uh we mentioned celluloid but it's come up a few times on the chainsaw yeah and even though it's not popular with me that that episode was really popular so if you haven't listened to it go back and. There's people out there that listen to the show, I promise. Yeah, sure. Fun episode this week. Should we get to it? Let's do it. Chatting today with Tony Arceri from Living Social,

the creator behind Celluloid and some other projects. So Tony, for those that don't know, you want to introduce yourself. Hi there, I'm Tony Arsiri. I've been doing Ruby about seven years now, I guess. Kind of jumped in with the Rails bandwagon there in 2005. Got on board with like Rails 0.11. This was back when everybody was deploying with fast CGI, and Lighty was the cool name of hotness of the day. So I guess I've been around a while.

So why Celluloid? Why don't you give us a little background behind this project? So I used to do a lot of network programming in C, and when you do stuff in C, you have to build your own abstractions uh so I had kind of an idea of what I wanted to do or always wanted to do in C and I discovered Erlang and Erlang had sort of taken all the stuff I had wanted to do and uh put a really nice face on it and Erlang was kind of how I discovered things like the actor model and basically seeing approaches to concurrency that also work for distribution.

When did you start the project? I started it around this time last year. It's a little bit over a year old now. I was like lying in bed one night kind of you, you know, around like 3am or something, trying to go to sleep. Just kind of thinking about, so I was working on a programming language, Raya, which was like trying to bring Ruby to the Erlang VM. I was thinking, you know, do I really need to make a new programming language or can I just take all the Erlang ideas back over to Ruby, which is something I tried before. I tried Revactor, which was another project trying to bring sort of Erlang-y ideas to Ruby, and I abandoned that, switched to Raya, and I was like, again, like, you know, maybe I can bring all this stuff to Ruby. We don't need a new language, and everybody can just write this stuff in Ruby.

Let's dive in there for a moment. What are some of these Erlange ideas? So the big problem, I think, is Rubyists don't like concurrency and we really need it in the modern world, right? We have computers with multi-core processors. You know, we have Rails applications that sit around and do nothing while we're talking to the database. So all these things can be solved with threads. And Rubyists just don't want to use them, and I would really like to change that. What are some of the libraries that you can or can't use if you're using celluloid adapter support? Do you need to really take advantage and exploit this?

So Celluloid will work with any thread-safe Ruby gems you want to use. So if you want to use Rails, you want to use Sinatra, you can actually drop Celluloid in in the background. Right now there's unfortunately no Celluloid Rails plug-in type of thing to kind of give you a happy path for that. But if you're using Rails in thread-safe mode, you can just go ahead and drop in Celluloid. Let's talk about some of these, I guess, sub-projects of Celluloid. There's the I.O. project, there's D.Cell and Reel.

What are these three? So the first one you mentioned was Celluloid I.O., right? Mm-hmm. Okay, so Celluloid I.O you do evented IO inside of an actor. So it's basically a combination of the actor model and the reactor pattern. So basically each of these objects is kind of, each of the concurrent objects slash actors is kind of like an event loop in and of itself, right? So the actor model works by having actors,

you send them messages, and then they can send you messages back. So when you're waiting for messages with Cellulite.io, you can also wait for other IO handles in the system. So you can have one actor that's potentially waiting for uh you know 10 000 possibly other uh sockets or something right so it's similar to like event machine or no js but you can have as many of those actors as you want in the system you can kind of uh isolate what connections they with. So you don't have one event loop

dealing with all your clients. And then in the same event loop, you're like trying to make outgoing connections to other services. If you've ever had to debug somewhere where one of these event loops is getting stuck, it's really nasty. So celluloid, uh, celluloid IO specifically lets you isolate, uh, which actors are handling what, so that's easier to debug. So the canonical example of these types of, um, projects are usually chat servers, but let's get a little bit more specific without divulging anything that you can't share. What types of apps at LivingSocial are you building with this sort of project? So we're not actually using Cellulite.io at LivingSocial. We're presently investigating DSAL,

and we're building an internal platform as a service. So it's some of the monitoring and automation around that. We're investigating using DSAL, but it's still in the preliminary stages. Gotcha. So Adam Keyes on the commit list here on Celluloid.io just assumed that it was a living social joint. Yeah, yeah. No, I mean, he's just a fan. So what about DSL?

So DSL, one of the neat things about the actor model is it's really easy to extend from building concurrent systems inside of a single VM to distributed systems that run on multiple computers. And that's because the actor model provides you this really high level abstraction with, you know, these sort of vague terms. So, I mean, the basic idea is an actor is a mailbox, and that mailbox is an address. So it doesn't really matter where that actor lives. It can be in the same VM, it can be on the other side of the world. As long as you have its address, you can send it a message. So Decel lets you build systems with celluloid that span multiple computers. How drastically does that change your architecture? So the idea is it shouldn't. Basically, you can prototype everything inside a single VM.

And then when you actually want to pull it apart and distribute it into separate services, it's just as easy as starting services in different VMs instead of all in the same VM. So a couple of times here in the docs, you mentioned 0MQ. We talked about that when Zed Shaw was on the show. What's the selling point for Zero MQ other than, I suppose, speed?

Well, it isn't actually speed. So, I mean, Zero MQ is very fast. It's built, Diesel and Celluloid ZMQ were built on this library called FFI-RZMQ made by Chuck Reams. He's done a really good job

of optimizing that as much as possible and getting the latency as low as possible but really it's not about speed it's about uh having a higher level message transport than tcp itself so uh 0mq is some really neat features as far as if your network gets partitioned and you try to send send a message to another node right with tcp that can potentially time out or give you an error and zero mq will hold on to those messages in memory and queue them up and then as soon as that connection becomes available again it can send them all off. So it sort of gives you some basic features of message queues, but without a broker. How long has ZeroMQ been around?

I've known about it for like three years, I think, maybe even more. It's not like terribly new at this point. It's actually fairly mature, and a lot of people are using it now. There's projects like Storm, which is sort of this distributed function processing system built in Clojure, and people are doing really neat stuff with it now, so I'm a fan. Are projects like Celluloid and Decel, this invented model and this distributed model, is this the frontier of Ruby and frontier of some other languages? Or is this just somewhere where you like to play?

Well, I mean, so there's Erlang, which has been doing this thing for like almost 20 years now. Or more than that, I guess. So, I mean, Erlang was really ahead of its time as far as this stuff goes I would definitely say this is the frontier for concurrency in Ruby there's really no other game in town as far as I know so instead of pulling these features into Ruby how come we're not seeing frameworks I guess built on top of Erlang. What's the barrier there? So that's been tried quite a bit, actually. There were a bunch of projects to do that, probably most notably Vertebra from Engine Yard tried to do this.

And it just ended up being overcomplicated. Ezra did a simpler project with RabbitMQ called Nanite, which is sort of, you can sort of look at that as being a little bit similar to DSAL, but DSAL is a lot more full-featured, I would say. Nanite gave you one agent, and DSAL gives you as many as you want that are as easy to implement as just writing or a ruby class uh and then desol doesn't have a broker right so with nanite you had to deal

with setting up rabbit mq and its high availability is a little bit tricky you have to do like a san or drbd or something so uh desol doesn't have any of those problems because it's fully decentralized. What about Reel? So Reel is a web server I wrote on top of Celluloid.io. Some of the goals there were to have a nice modern web server without a RAC API because RAC is kind of problematic in celluloid specifically the way it implements middleware loves to sort of use a ton of your stack right celluloid uses fibers and on Ruby 1.9 YARV at least you only

have a forward kilobyte stack for fibers. So using RAC in conjunction with fibers is kind of impractical, as some people who tried to run a full Rails stack on top of it discovered. So the other thing in addition to just not using RAC that I want to accomplish is having a nice integrated web server built on Celluloid.io with WebSocket support. That feature is kind of vaporware right now, but I've been looking at some stuff like libwebsockets that's available for Ruby,

and it seems like there's some pretty awesome libraries I can tap into since I try to tackle that problem. So Reel appears to be pretty bare metal. Yeah, yeah. So it lacks a lot of the routing and DSL and some of the things we've come to expect from Ruby web frameworks. Yeah, yeah. So I suppose it's geared more towards single purpose, just ultra fast types of servers?

Well, there's that. And also, I've been trying to get it working with WebMachine. So Sean Cribs kind of pushed a proof of concept of a WebMachine driver for it, and I just need to add some missing features to Reel and

write some tests for that after there, and I think you can use WebMachine for all this stuff where you need a higher level of abstraction. Sean Cribs is from Basho. Yeah, yeah. So also slings a bit of Erlang, if I recall.

Yeah, yeah. So what sort of projects are you building if you're not building these low-level projects? I mean, I'm always fascinated by folks that are building frameworks and libraries that ultimately other developers build, and that's what they're doing exclusively. But what sort of, I guess, user-facing work are you doing? So I don't really have any super serious projects

I'm building with Celluloid. A couple of them I built in IRCbot called Cellulon. Sort of similar to Hubot, but more Unix philosophy where each script is just a script. And if you print from it, it prints to the IRC channel, that kind of thing. The other one is called the Cryptosphere. This is a project I recognize

is kind of above my technical ability right now, but it's something I like to hack on. It's sort of like a peer-to-peer distributed encrypted data store that's completely decentralized, so anybody can join. I guess the closest projects today would be something like Freenet. There's also GnuNet.

Well, your projects have come up on a couple of episodes. The most recent ones were the Adhesion episode and then also Travis CI. So do you have, I guess, other developers that are working on their projects ping you and asking you questions about how to integrate Seloid or are you actively marketing the project? Yeah, definitely. So there's an IRC channel just pounding celluloid on Freenode. And actually there's been quite a bit of traffic in there lately.

So yeah, like Ben Langfield, I believe is his name, for adhesion, right? Right. I mean, he isn't on the RSC channel per se, but he's definitely been one of the main ones talking to me. And obviously Erlang was created for telephony, so I think that's a really good fit there. But, yeah, I mean, there's been a lot of people who already have a project and they've,

you know, been dealing with some of the issues of threads and they're like, I just want something simpler. So it's, it's been a lot of investigation of, uh, rewriting existing projects with celluloid. Are you seeing an uptick in adoption of these projects? Yeah. Or not, if not adoption, then interest. So, you know, just people who've, they basically are to the point where they're like,

I need a framework because my multi-thread program is too complex. So definitely a lot of interest from that type of person. How much influence, if any, have you gotten from either Node or Twisted or some of the other frameworks in the other languages? So definitely I've gotten a lot of inspiration out of Python. There are a lot of similar projects to this that I had sort of looked at when I was originally developing Revactor. Some of those are like, there's one called Camellia,

another one called Eventlet. So those projects were pretty influential. See, beyond that, I mean, the main influence of celluloid in general is Erlang. hand from the ever popular Hacker News website. And right now, two big events are happening. The 100th issue of Hacker Newsletter and 10,000 subscribers. So to celebrate, 10 lucky subscribers who open the 100th issue

which comes out June 8th will win some very cool prizes. For full details subscribe today at HackerNewsletter.com Shifting gears for a moment, we had at hackernewsletter.com. Shifting gears for a moment, we had a plan to talk about LightRail. And from the time that we set up this interview to today, there's been another project that's been introduced, which is the Rails API. So we were talking before we started recording, and it sounds like you're going to kind of shut her down on light rail yeah so i mean

the whole goal of light rail was uh when i was working at strobe i was there with uh three rails core members uh yuhuda katz carl lurch and jose valim and uh carl and jose had built a Rails 3 stack specifically for Strobe, which had been stripped down to action controller metal specifically. And from there, we just pulled in the stuff we needed. So I thought I'd just put that out there and see if it stuck. And I talked to Jose, like, is this a good idea am i competing with rails and it turns out i was kind of competing with rails even though jose uh gave me the go-ahead to release it there but uh you know it's been a little bit of a debacle uh trying to

get a standard way to build rails apps that are only JSON APIs. But now I would say don't use LightRail. Check out Rails API instead. From Santiago Pastrino, I should mention. It's on the blog. Any big differences between what they've done with Rails API and what LightRail is was? So Rails API contains a lot of stuff that's an evolution of stuff that Jose had originally developed

that we shipped in LightRail. He had this thing called Wrappers in LightRail that actually has been pretty much completely rewritten and replaced by this thing called Active Model Serializers. But the basic idea is there's a canonical way to represent JSON. So, you know, some, like right now, everybody makes their own JSON APIs, and they all look completely different. So active model serializers was trying to give you a way to standardize so everything that sort of talks this specific JSON format

can all talk to each other, and you don't have to write a bunch of one-off JSON generators and JSON clients. Is that set a level above the models? Are they mixed into the models? Is it a presenter pattern? How does that work? It's its own object that interacts with the models,

but has some context beyond what the model does i think right now the context is only the current user but it uh it provides an abstraction for serializing json and specifically stuff like you have a client who wants to pull in a resource but that has a bunch of associations and you want to grab those all in a single request. It gives you an abstract way to do that. That's a common use case. I think that's a lot of times where a lot of these projects in this space kind of fall down for me, is they assume that you've got one representation of this particular resource throughout your entire API. And a lot of times there's little nuggets that you want to share in this

context that you want a fuller representation in another context. I've been using JBuilder, which is the HHS project recently, and I've liked it for that reason. You kind of handcraft and roll your own JSON in that regard, and it supports partials and views. Terrible name, but neat project. I mean, I think that's kind of the underlying philosophical debate is like, is a serializer closer to a view,

or is it actually some higher level abstraction that's completely different? There's a lot of room for, I guess, implementation details when you come to build an API in Rails. I know that the Rails way is to really support bare array APIs, and it seems like inevitably you want some sort of wrapper envelope or something or some sort of response object where the actual return value is hanging off of that so that you can see total number of records and pagination info and some other things that just unless you stuff those

into headers that are just difficult if you return bare api arrays yeah i mean so there's actually some security concerns around bare apis or bare arrays I mean, because arrays in JavaScript map onto objects, so you can redefine things that arrays do, and potentially a malicious script can get access to that data that way. So Bear Arrays

in general are bad. The real advantage of something like ActiveModel Serializers, I think, is it handles relational data. So when I was talking about including other resources, there's a bunch of ways to do that, right? You could sort of nest the resources you want inside of the one you retrieved. And that's bad because you could end up nesting it in several places. And then the question becomes, which of these, which of these is the canonical one, right? If you have three

copies of the same resource inside your JSON. So Actimol serializers flattens that all out and then uses IDs as a way to associate, you know associate the parent data to its associations. When I came to Ruby and Rails, I guess 2006, one of the selling points was convention over configuration. It was just an easy on-ramp. It seems like we're getting a lot more complex with the problems we're trying to solve. Do you see any issues with just introducing so many different decisions for the new Rails developer that's coming to the stack? I think it's the other way around. I think

it makes it so the end user has to do less because more of these decisions have already been made for them. You know, some people may not like that. Some people may want to build their own JSON serializers. Having done that by hand so many, many times over the years, I'd love for there to be just one solution for that problem. Do you consider yourself a polyglot? I do in that I investigate a lot of languages. I like to learn little tricks from them. But really the only two languages I'm super comfortable with are Ruby and Erlang. So I mean, there's not a lot of other languages I'd be really confident in building like a large project in right now. So a lot of times I'll ask folks what features of languages would they want to

steal and bring into their favorite language? And you, you've gone down that path with Erlang of actually, you know, and more than one way tried to, to fix that problem. Any other features, some, any other languages you've used other than the concurrency problems that you'd like to solve in Ruby? So the two big ones I've seen kind of getting a little bit more attention lately are object capability systems. So the idea of a capability system is you can have complete control over what

objects in the system another object can access. So this is sort of an outgrowth of the actor model itself, actually. So you can introduce an object to another object, basically. And as soon as you do, that object can access the other object. And where this is a huge concern right now is in the browser, because we're trying to do mashups, right? We're trying to pull in third-party code. But right now, that code can basically do whatever it wants.

It can go nuts on the page. It can read anything in the DOM. It can transmit stuff back to whatever server it wants via like a script tag or image tags or anything like that, right? So the situation there is kind of out of control and there's a lot of people at ECMA trying to sort that out right now. There's actually a language called E, which nobody's ever heard of, but it was built on object capability systems,

and I think those are pretty interesting. The other thing I really like comes out of this framework called Killem on Java. That's this idea of linear ownership transfers. So right now when you write a multi-thread program in a mutable state language, any thread can just trash that state, right? Like any thread that has to handle the object can do whatever it wants.

And ownership transfer systems are a way to prevent that by handing off ownership of a particular object to another thread. So you can go, I'm done with this object, here you go. And if you try to use that handle again from the original thread, it raises an exception. So I think that's a good way to build safe multi-thread programs, even though you have mutable state. You mentioned a few patterns. There seems to be this scale between developer and then programmer and then computer scientist. How fluent does a developer coming to the Rails stack or Ruby stack nowadays have to be in these patterns, or how much of them are just baked into the tools they use? I'd still like to say onboarding in Rails is fairly easy once you get over the hurdle of installing Rails, especially on a Mac.

But I think really there's not a whole lot of background you need to get started in Rails right now. I think they've done a really good job keeping the API clean and easy to use.

So I wouldn't say you have to be a computer scientist to use Rails. There seems to be an uptick, though, in interest, I guess, in these patterns in the Ruby community. I'm hearing a lot more about DCI and some other patterns that keep rearing their head every couple of weeks. Do you have an opinion on DCI? I have a fairly negative opinion of dci itself uh if you actually read the description it cannot be implemented in the ways that

uh rubius uh what they're calling dci uh you know they they explicitly say it can't be implemented in any of those ways uh so the first is through a mix-in at runtime, basically, right? Like you have a module you extend on an object. So that one's bad because it blows the method cache. So there's another way to implement it, which is through delegation. If you actually read the DCI paper, they say you cannot use either of those to implement DCI. So I'm kind of confused as to what DCI actually is if it isn't either of those things.

How many of these patterns do you think have affinity to the language they were originally thought up in, I guess, like a better phrase? I think Ruby can handle most object-oriented programming patterns so it kind of depends the reason I ask that, a lot of times when I'm in JavaScript circles I see Rubyists that are getting into Node try to port a lot of the module behavior that we take for granted in Ruby into JavaScript and just start pulling their hair out

because JavaScript is a fundamentally different language. Yeah, yeah. I mean, so you could ask Hudo about that. He's actually done a full mix-in system inside of Ember.js, which has its own object model. Thoughts on hypermedia APIs? I definitely like not having to construct URLs. If that's really the only thing hypermedia gets you, I guess that's a win.

We need to do a show on this. I'm still kind of on the fence myself. I like the promise. I mean, everybody's got this dream of pointing some sort of code to a root URL and just have this wrapper organically unfold. What troubles me about a lot of the things that I'm reading is just the lack of type on some of the returns. And the folks that are building out HyperMedia APIs will talk about, well, it's just MIME types, right?

But a lot of their examples are HTML and XML, which are not typed. And then as, you know, just a Joe Blow Ruby wrapper developer that I am, there's things we take for granted that Jason has just made like a dial tone. We don't even think about it, right? If I had to sit there and construct objects for every MIME type I'm going to consume, I'd go crazy. Yeah, I mean, that's where you get into stuff

like soap did code generation and that kind of thing. So, you know, I mean, perhaps it could be done at runtime. Perhaps you could point something at an API and have it metaprogram all that stuff for you. I'm not really sure. You know, I'm kind of on the distributed object side of that debate, right? Like, you know, there's all this stuff that requires a lot of standardization if we're building HTTP APIs, which just goes away if you're using distributed objects. I should say, I'm quite ignorant on the subject so far.

I'm learning more, but we need to do a show with Micah Munson and maybe Steve Klabnick, our contributor here on the ChangeLog. He's got a book coming out on hypermedia APIs just to talk about the subject. Seems to be really hot, but also kind of, it's one of those things where, you know, if you are in JavaScript circles, if you want to end a debate, you just start screaming Crockford, right? And if you're talking about big data, you'll say Dynamo and Verner Wuggles, right? But if it's HTTP and REST now, it seems like people just scream fielding.

Nobody really understands that paper. But yet we all claim this is what fielding would have wanted. I mean, I think we have the problem solved fairly well. It's all just kind of bike shedding around these little minor issues. I guess my big thing is we're still trying to embrace REST. Yeah, yeah. And I'm happy when I stumble across a REST-ish API. As long as they're not wrapping,

clearly wrapping every database table as a REST endpoint call, that's fine. As long as I don't have to deal with a schema or document type definition, I'm cool with it too. But it seems like we're raising the bar a bit on people that are still trying to get their heads around REST-ish APIs. Yeah.

So when you're not hacking on Celluloid or Decel or some of these other projects, what's got you excited as far as open source? So there's a project I've been contributing to in Python called Tahoe, the least authority file system. I just redesigned their landing page there because it's kind of ugly. But this is a really neat tool. So I mentioned the Cryptosphere earlier. Tahoe is kind of the inspiration for that. But it's a great tool for if you want to run a cooperative backup system with your friends.

If you have a bunch of people who have cable modems or run servers or whatever, you can kind of just install this on all those computers and so long as the bandwidth and storage is more or less free to you, then you get free collaborative backups with this and you can also use it to share

data. I've been really enjoying that. I'm definitely a big fan of Travis CI. I just got to hang out with all those guys at RailsConf, except for Matthias. I finally got an invite to the private version of that. It's quite well done. Yeah, nice.

I have not seen it myself yet. What about your text editor? I use a few different ones. I still use TextMe as my primary editor. I've been sort of investigating MacVim, so I've known how to use Vi since I was like 13 or something, but I don't really like using it as my day-to-day editor, I guess. So I'm a Mac user, right, and OS X has these sort of ubiquitous Emacs-style key bindings

throughout the whole OS. So if I go to switch to Vim I have to like sort of context shift to Vim mode instead of just using the Emacs key bindings I played around with Emacs quite a bit when I was learning Clojure and it seems

pretty cool but I just couldn't get over the learning curve and you know I wasn't really sure if I did I would really actually enjoy it. For now, I generally kind of stick with TextMate, and I use Vim on the command line for sort of short, one-off type edits. What was your thoughts on Clojure?

I really like Clojure. I think it solves a lot of the problems that made Lisp impractical in the past. Just being on the JVM, they can tap into the whole ecosystem of JVM libraries. And they finally, well, they being Roach Hickey, finally added syntax for some other data structures besides Lisp, right? So it has maps and vectors that are really handy. As far as trying to build something big in Clojure,

I'm just not sure I would really prefer doing it in Clojure to Ruby. Like, in terms of everything like maintainability and my ability to sort of comprehend the code base, I think it's just easier to do in Ruby. Although I think probably the result if I wrote an enclosure would be a lot faster and a better performance. Do you have a programming hero? Yeah, it's a tough question. I guess I have a few of them. I'm a big fan of the Erlang guys. I've never met Joe Armstrong, but I've met the co-creator of Erlang, Robert Verding,

and they're both pretty cool guys. I think they were really way ahead of their time with Erlang, and a lot of this stuff is becoming relevant now that we have these massively multi-core computers and really fast Internet, and everybody's building distributed programs now. I ask all three of those questions now because the last three episodes, I always assume that nobody's ever heard this show.

So we kind of end the episodes on those questions. The last three episodes we've hung up and someone's pasted in our chat later, hey, dude, you didn't ask me X question. I still have my answer ready. All right, nice chatting with you today, Tony. Thanks for giving us the lowdown on celluloid and the other projects. Appreciate it.

Yeah, cool. I see it in my eyes So how could I forget when I found myself for the first time Safe in your arms As the dark passion