The Changelog: Software Development, Open Source - 10 years of RabbitMQ (Interview)

Episode Date: July 21, 2017

We are thrilled to produce this show to honor RabbitMQ’s 10th anniversary. Karl Nilsson and Michael Klishin joined the show to talk through 10 years of RabbitMQ — one of the most widely deployed open source message brokers with more than 35,000 production deployments worldwide.

Bandwidth for Changelog is provided by Fastly. Learn more at fastly.com. And we're hosted on Linode servers. Head to linode.com slash changelog. This episode is brought to you by ElixirConf 2017, held September 5th through 8th in Bellevue, Washington, just across the lake from Seattle in the home of Microsoft, as well as Amazon. It's two days of training on September 5th and 6th

and two days of conference and community on September 7th and 8th. Get face-to-face time with core developers of Elixir, Phoenix, Ecto, Nerves, and more. Learn from over 40 speakers and keynotes about how top companies and developers are getting performance gains from Elixir and surpassing their competition. There is no better place to discuss, collaborate, and socialize with other Elixir professionals and enthusiasts. And the ElixirConf organizers have been generous enough to give us a $40 discount. This is exclusive to us. You can't get this anywhere else.

Head to elixirconf.com to learn more, and use our special URL, elixirconf.com slash changelog, to get that $40 discount. And now, on to the show. Hello and welcome to the ChangeLog, a podcast featuring the hackers, leaders, and innovators of open source. I'm Adam Stachowiak, Editor-in-Chief of ChangeLog. Today on the show, Carl Nilsson and Michael Klitschner join the show to talk through 10 years of RabbitMQ, one of the most widely deployed open source message brokers with more than 35,000 production deployments worldwide. We're thrilled to produce this show in honor of RabbitMQ's 10th anniversary. All right, we have a great show for you today. We are joined by two people, Carl Nilsson and Michael Klitsch.

Both engineers at Pivotable and have been working on RabbitMQ for a while now. Guys, we're here to celebrate 10 years of RabbitMQ, an open source project that is a message broker used all around the world. So first off, thanks so much for joining us on the show. Thank you for having us. So just to associate names to voices a little bit, why don't you guys take turns introducing yourself and kind of give us your role with regards to the RabbitMQ project. Carl, let's start with you.

Okay. My name is Carl Nielsen. I'm an engineer in the RabbitMQ team. I've got a history in.NET, a lot of functional programming, and that's pretty much it. Right. I am Michael Kishin. I am also an engineer in the RabbitMQ team. I have been a RabbitMQ contributor since maybe 2009 or so. And back then I was doing a fair amount of web development and also, I guess, data infrastructure kind of stuff. And that's how I arrived at what today is referred to as microservices. And it went from there.

Very cool. Well, we are interested in RabbitMQ for a couple of reasons. First of all, we love to celebrate milestones, and we talk about sustainability a lot on the changelog. Any project that gets to 10 years of success and deployed all around the world and used by many corporations, we consider that success. So we'd like to celebrate with you guys,

and congratulations on 10 years of this project. We're also interested in it from, I guess, from that sustainability angle. This is a corporate-sponsored open-source project. It was created 2007 inside of Pivotal and has existed inside and outside of Pivotal as an open-source project, which is interesting. A lot of open-source projects either start outside of a company and then move their way in or always stay outside. We're seeing more and more corporations doing open source

from the very beginning with projects. But I'd love to hear the Genesis story a little bit. I know, Michael, you said you've been involved since about 2009. Can either of you recall the beginning of RabbitMQ and why it needed to exist in the world? Sure. So, first of all, I wasn't around in the very early days. So if one of the founders is listening to it, Alexis Mathias, I'm sorry if I get something wrong. But that out of the way, Rabbit was actually started in 2006, but Pivotal didn't

exist back then, but let's start. First things first, right? So in 2006, the landscape of open source projects was pretty different. For example, Ruby on Rails was like 18 months old or something

like that. Less than two years, probably. Many open source data services, data stores, messaging technologies, most of them did not exist. And when it comes to messaging specifically, messaging is definitely not a new concept. Financial services companies have been doing that for various reasons since the 80s, maybe even earlier. But there were commercial products in the late 80s.

And by 2006, several folks who have been doing financial software for a while, they were slightly fed up with mostly vendor-looking, I guess. Because there was a number of large players, who we who will rename unnamed that were charging quite a bit. And moving between those tools was many months, if not years, that kind of undertaking. So RabbitMQ was actually started as implementation of something called MQP. And today there are two different protocols called MQP, but we will get to that. So someone, I believe, a GP Morgan Chase, came up with this idea,

hey, let's develop a new standard. Yeah, all the XKCD standards comic jokes applied. Every single one. But there weren't that many open source messaging technologies. I'm sure there were some, but I honestly cannot name any of them that actually existed back then. And there was, so that was, I guess, in New York and across the pond in London, a couple

of people were working together at a consultancy looking to start a company maybe to do something to do a thing of their own and they discovered this mqp thing it made sense to them they wanted to try this technology called airline and yeah that's how rabbit was born and 10 years later it supports over 20 programming languages multiple, has accumulated a fair amount of technical depth, which we can get ourselves into, the kind of things you shouldn't try in your distributed system. And yeah, it is used surprisingly widely. Yeah, according to the website,

there are more than 35,000 production deployments of RabbitMQ worldwide at small startups and large enterprises. Lots of different people using this. Rabbit describes itself as an open source message broker. Now we also use the word, I mean, MQ, we use the word Q in the title. And as we talk about these things, is there a difference or is it just, are we talking semantics

between a message queue and a message broker? I think there is a difference, but it's pretty subtle. For example, a lot of people use messaging, and they think in terms of queues or logs of operations, that kind of thing. But the broker part is actually optional. There is a project of about the same age called ZeroMQ, started by Peter Hintjens, which doesn't really have this broker component. It's a library that you embed into a tool, well, into more than one tool, most likely, and they communicate using various messaging patterns. There is this queuing aspect, but there is no broker aspect, or rather every single application plays that role a little bit.

It's a pretty different architecture from RabbitMQ, but I guess most messaging technologies have this middleware. You have a node or a bunch of nodes that clients connect to, and those nodes do the routing, store messages, expire them, deliver them, acknowledge them, all that jazz. Yeah, just to add to that, you know, about zero MQ, a message queue doesn't have to, as Michael said, doesn't have to be a centralized thing.

You know, your queue could be local, and I guess that's what 0MQ does. It's kind of almost directly addressing queues on a remote system rather than through an intermediary. And one of the patterns that 0MQ, for example, have is to implement a broker using the 0MQ libraries. That's one of the patterns that they achieve things like service discovery, discovering services that you don't know already directly. So it's almost like a pattern, but it's a very useful one

and one that you can do a great deal with. So the significant difference with a broker is that you're dispatching messages to remote nodes, whereas with only a queue, that does not necessarily have to happen. You could just be queuing locally and you could build in the dispatch on top or next to. Almost.

It's more about addressing some remote queue directly or through an intermediary. If we want to look at it in very simplistic terms. So if you know about the broker, you don't need to know about anything that actually is interested in that particular message. You just give it to the broker

and the broker will take care of the rest. Whereas if you go like in the 0MQ, you're just using bare-bones 0MQ, then you would have to know about where you want messages to go to all the kind of interested parties unless you implement a broker in the middle. Gotcha. So let's talk use cases. I think there's probably many of them but you guys have been working on RabbitMQ for all these years

I'm sure you've seen a lot of them in production and probably help people out with them every day. What are some of the real perfect use cases for a message broker? So RabbitMQ is a fairly generic tool. There are, just like with data stores, there are messaging technologies that specialize in certain areas. For better or worse, RabbitMQ is fairly generic. So if you need to make multiple applications hosted on typically different machines,

if you need them to get to talk to each other, that's one use case. And specifically that translates into scenarios such as you have a web app in Ruby or JavaScript and you have something that does data crunching in one of the GVM languages using all the great data processing tools there, or it integrates with existing.NET tools, or, I don't know, you work at a small company that is daring enough to try Elixir, Haskell, all of those less commonly used tools. So messaging is a great way to connect

those. To be fair, messaging isn't the only way. There is a book called Enterprise Integration Patterns, and please don't fear the title. It's actually a very sensible book. And it's not specific to those working at huge companies. I think it was published in the 90s or so. And yeah, you can integrate using shared databases. If you have tried that, you probably know that you have to be careful. You can integrate using Unix sockets, local file system, all kinds of things. But once you go distributed, your options are more limited. And so that's where messaging comes into play.

Recently, we have seen quite a few use cases, like two major tides that lift all messaging boats. They are called microservices. And for the record, I hate the term. It's about as specific as cloud computing. And IoT, which is even more specific, right? So yeah, you have your internet connected dishwasher. You need it to talk to your internet connected light bulb for whatever reason.

And all of that has to be service oriented. That's another couple of very, very broad areas where messaging is used. And then, of course, there are problems where Rabbit doesn't try to necessarily specialize in them, but there are stream processing systems that are usually more than one tool,

but one of the components is definitely something that is a messaging technology. And that has all kinds of applications. From tracking your runs, when you are varying your, I don't know, pick your favorite wearable device to connected cars

to, I don't know, software updates and all those connected cars. You name it. I think in modern world messaging, and I by no means equate messaging with RabbitMQ, is just about everywhere. And it will only go into get worse.

Let's put it that way. Yeah, absolutely. Just thinking about the term microservices now that you've mentioned, I never thought about it, I guess, critically. But how small does a service have to be before it's considered micro? That begs the question.

Is there a standard yardstick for that? Is it a 250 lines of code? Right, yeah, exactly. Is it API surface? Is it lines of code? When is your service considered no longer micro? Oh, that's getting too fat now. It's just a regular service.

I think for microservices, the useful thing that I feel has come out of the discussion around microservices is don't make your services too big. Focus on that single piece of capability, business capability that you want them to have. And yes, when you have that many components, obviously you need a way to connect them. And a messaging broker is a great way of doing

that. Because as Michael said, if your microservices are written in different languages on different stacks, messaging system exists for specific stacks, right? They've existed for specific stacks for a long time. For example, in.NET, you could use things like.NET remoting to send messages from one thing to another. But if you need to go make your Ruby web app talk to your Java backend, it's a great thing because of the protocol.

Because the protocol is standardized. It's a wire-level protocol, and you can use that to connect. So integration like that, I think, is a really, really good one. But one thing I wanted to add to that, you know, what is the use case? I think one really important use case, in addition to kind of discovery and integration, is, you know, why do you have a queue? You know, you could do discovery and integration without having the queue bit, right? It's just root messages directly.

The queue is there to hold messages until they can be processed, right? It's a way of flattening sudden peaks in your activity or sudden outages of certain services so that altogether you build a more resilient system. I think that is also one of the major use cases for why you would want a queue in between the thing that generates something and the thing that processes.

Yeah, having used queues in applications, it's always awesome when you have some sort of production problem and you know that you're not dropping those transactions, right? Those actions, they're just going to queue up and they're just going to keep on building. And when everything goes back to being normal again, it's going to work that queue back down. And like you said, it's resilient.

Whereas if you didn't have that in place, if you were just brokering messages and nobody's on the other end of the line to answer the message, then you're just dropping those things on the ground. So speaking of messages, Erlang, you mentioned that they wanted us to try out Erlang back in 2006, they wanted to use it. And when it comes to distributed systems and sending messages, seems like that was probably a very good choice as it's a language built specifically for sending messages between switches and other telephone operating system type of things. Ten years later, thousands and thousands of lines. You got 168 contributors, at least on the RabbitMQ server.

Surely once you hit the clients, there's probably thousands of contributors. 156 releases. It's been a long time, almost 17,000 commits. And you're still on the server repo, 94.3% Erlang. So this is a longstanding Erlang project. And I don't think we've had on the changelog a show about a specific project in Erlang, which reminds me of a new friend I met at OSCON last week. Mark Allen, wanted to give him a shout out, an Erlanger who listens to the changelog.

And I told him, we don't do too much Erlang coverage, but here we are talking about RabbitMQ and talking about Erlang. So there you go, Mark. Guys, tell us about that choice and the use of Erlang over the years and how that's played out for the project. First of all, I think I recall a ChangeLog episode

where you did have an Erlang project, which was started around 2007, I believe, for React. That was probably the last one you listened to. Could be. Now I know how to get you to listen. Erlang shows. Not necessarily. Yeah.

Yeah, we do cover it from time to time, but it's been a while. Yeah, so in 2006, I remember correctly from all the unwritten stories I think Matthias Radestock just wanted to explore Erlang for an infrastructure kind of tool Erlang may or may not make sense for your web app at least if it renders

a lot of HTML, that kind of thing but for infrastructure tools I think it's it should be on your short list right and yeah so they just started building a prototype

in Erlang and the rest is history Erlang itself has changed quite a bit in these 10 years and RabbitMQ still supports Erlang releases that are I don't know maybe 4 or 5 years old and I can tell I don't know, maybe four or five years old. I can tell you that in the last maybe two or three years, Erlang has become... well, the number of words and sharp edges there is significantly smaller. And then Elixir emerged,

which... I'm not supposed to say this, not everyone in our team agrees with it but I'm going to do it anyway I think Elixir is what Erlang should have been all along and not this you know looks like Ruby aspect what experienced developer gives a shit about syntax

it's something that you pick up in a few days and never think about it much again but because there are very practical improvements in Elixir something that you pick up in a few days and never think about it much again. But because there are very practical improvements in Elixir, ranging from it has good associative data structures,

so maps, dictionaries, hashes, whatever they're called, in your language of choice, that's a huge thing for me because besides Erlang, I've spent a number of years working in Clojure and before that Scala and before that Ruby and Java

and I can tell you that working with prop lists of these lists of pairs in Erlang it's pretty painful. Something that is like a few lines and it's obvious what they do in Clojure which has a very nice collections library in Erlang can be several times more lines of code,

but again, the lines are not the point. It's just, you have to write this every time and in every code base, it looks slightly differently and you have to spend your brain cycles figuring out what is going on. So that is much better. And it sounds like something maybe silly, but you use those things every single day, right?

I think we should not ignore the cumulative effect of such small improvements. Another thing that I really like is that Unicode and Elixir is not an afterthought. We have seen interesting issues that you can only run into if you use command line with like a Chinese locale or something like that. And as far as I know no one on our team unfortunately can read or write Mandarin or Cantonese. So yeah, we would really like to not have to fix those issues. And Elixir, they're much less likely to occur. And yeah, I think more sensible compiler error messages,

also a great contribution, and so on and so forth. Anyhow. Let me stop you there for a second, Michael. I want to ask Carl. You said everybody on the team would agree with that. Carl, he just said that Elixir is what Erlang should have always been. Agree or disagree? In one word, I'm probably one of the team members that disagree.

I do think Michael's point about PropList is I'm 100% behind that. To call it a data structure is probably being very, very generous. I do think Erlang, if you take the modern versions of Erlang, where we now have good map data structures, and I think I enjoy writing Erlang, I think it's good. I think Elixir is nice. It's nice. Michael knows this. I've got strong opinions. I like typed, statically typed languages very much. So I like languages where, you know, the types help me along.

And neither Erlang nor Elixir solves that problem particularly well. Right. Yeah. But I do, you know. So a question from the outside. So as an Elixir user and Erlang, not even an observer, but once in a while I'll have to take a look. And I guess as an Elixir user,

there's a lot of Erlang underpinnings that are being used. I'm speaking of APIs and stuff. They both run on the same runtime. Has the increased interest and has Elixir community growing and people getting excited, has that reinvigorated the Erlang community?

Like you said, Michael, a little bit ago, that there's been a lot of the rough edges have been polished down and there's been a lot of the rough edges have been polished down and there's been a lot of stuff changing over the last few years. Has there been any back and forth where the Erlang community and team that works on the core language

has said, oh, okay, we need to step up our game, or is that just perceived? I personally think that, well, let me give you an example. I've been to a small Erlang slash Elixir conference in early April in Rome, and shout out to the organizers. It was a very nice event.

And so most people there, at least those to whom I have spoken to, told me that they have considered or tried or used Elixir. That said, I think less than half actually use it. But Elixir was mentioned in, I don't know, 80% of talks, something like that. A few talks were exclusively about the experience of adopting Elixir, including mine. And I think Elixir definitely sparked a massive conversation in the community about... So it takes a lot of effort and dedication to produce a programming language, right?

So not everyone is as crazy and dedicated, I mean it in a good way, as Jose Valin. So do we really need to develop Elixir or is improving Erlang okay? And I don't necessarily have an answer to that, but I think the Erlang team and the Erlang community have recognized a lot of things that are obvious from the outside. If you come from Ruby or Java or C Sharp background, but they're not at all obvious to them. And there were many improvements in the last few years. And I expect this to continue for, I don't know, the foreseeable future. I agree with that. I think

Elixir has had a positive effect on Erlang. I mean, Jose, Elixir compiles down to Erlang. So it doesn't even skip a step and go straight to whatever Erlang compiles down to. And Jose is active, contributing to Erlang OTP, to the Erlang compiles down to. And Jose has been active contributing to Erlang OTP,

to the Erlang platform. So I think not just from what Elixir does and how its approach to tooling and the developer experience, but I do think it has a positive effect. That said, I think Erlang OTP is probably another open-source project that's been open open source for a very long time and also very successful. Coming up after the break, Jared moves the conversation to lessons learned. It's important for any project to learn and grow from their mistakes, especially a project like RabbitMQ that deals with complex architectures in an ever-changing world. We discuss how to choose dependencies, what happens when you don't preserve the quality of client libraries, and the politics involved in messaging protocols like RabbitMQ.

All this and more, after the break. This episode is sponsored by CircleCI. CircleCI is a continuous integration and delivery platform that helps software teams rapidly release code with confidence by automating the build, test, and deployment process. They recently launched version 2.0 of their platform with a focus on providing faster build times thanks to advanced caching strategies and flexible resource allocation. Super fast build cycles ensure quality code by using SSH access and local builds to quickly troubleshoot and remediate. Flexibility to run CI and CD without limits.

There's no pausing work while environments update and language inclusivity frees up your team to use any tool chain or framework because CircleCI supports every language that runs on Linux. And finally, control. Workflows let your team run, build, test, deploy stages as individual jobs, which lets you fully customize your development process.

There's a ton more to learn about CircleCI, so head to circleci.com slash changelawpodcast. Once again, circleci.com slash changelawpodcast to learn more. All right, we are back talking about RabbitMQ, talking about Erlang and Elixir a little bit. And guys, one thing we just wanted to clarify on the other side of the break was Elixir's compilation process, just so everybody's clear on how it works. We did look it up during the break just to make sure. You guys want to lay that out a little bit?

You don't have to go into the whole details, but explain clearly how it goes from Elixir to executing code on the hardware. Right, okay. So Elixir would be, the Elixir source code would be passed into some kind of abstract format, an abstract syntax tree in Elixir.

That is then translated into the Erlang abstract format, i.e. what you would get if you passed an Erlang file. And then from there, it hooks into the standard Erlang compilation flow, which compiles down to core Erlang, which is an ML-like, simple, machine-processable functional programming language. And then from there, it will then produce Beam bytecode, your.beam files. Very well done. There's a nice little Medium post

by Javier Noria, if that's how you say his name, called How Does Elixir Compile Slash Execute Code? We'll add that to the show notes, so those who are interested in the step-by-step and a reiteration there of what you said can go and read that. So very good. Just wanted to clarify that it doesn't go from Elixir source to Erlang source, which some people may have thought that. So thank you very much. Let's talk about mistakes made. Who wants to kick this off and where should we start? I can. So in a messaging system, you will find three major areas

where you can make mistakes. You can make mistakes in your protocol, right? Even if you adopt an existing one, you can make mistakes in the implementation, of course. And you can make mistakes when developing client libraries. And of course, in 10 years, RabbitMQ definitely did all of those. So let's start with the latter one. It's a relatively straightforward thing.

So maybe it's obvious to web developers, but folks who work on infrastructure tools sometimes kind of ignore this fact. If you maintain client libraries of any kind, ignore their quality at your own peril. It will come back to bite you hard at exactly the wrong moment to the point we have seen projects that build on top of RabbitMQ or use RabbitMQ as their default messaging choice, having issues that came down to incomplete, buggy,

or just overly opinionated client libraries, that, of course, it doesn't occur to the user that, hey, it's a library problem. What will get blamed is your server. And it happens 98 times out of 100. And then there are two more experienced people who know how to debug distributed systems and they can actually provide you some details

so you can improve things. So that's just the general quality of client libraries. Some of them were not getting the attention they needed. Let's pause there for a second. So client libraries, how many of them are there, roughly? And then how many are supported directly by you all, or how many are completely third-party, and how do you make those decisions and draw those lines?

So I honestly don't know how many there are are i think we support more than 20 programming languages of course some are supported better than others but most likely if you can name a language there is a client library of some quality and multiple languages have multiple client libraries, again of varying quality. Pivotal supports, I think it's currently three. It's java.net and Erlang. Erlang because it's used by plugins that we support, so it would be a little bit weird to not support that. But we are interested in adding official support for more. That said, many client libraries are either maintained by our team members, I maintain a few,

or we try to contribute to the extent the time allows. And of course there are completely community developed and maintained client libraries, but at some point their authors interact with us one way or another.

So just a side note, because I think this is a helpful thing for anybody when pulling in a dependency or deciding on a library. Say you decided to use RabbitMQ and you're using Ruby, and I'm just making all this up. Say maybe there's five clients out there on RubyGems. How does somebody go about picking the one that's the best quality? Maybe this is too big of a question, but what would you guys do?

How do you know what's the best? What's supported? What's good? So in terms of Ruby, you cannot go wrong because I maintain four of those five. There you go. All right, pick a different language that you don't maintain.

Oh, man. Let's say C Sharp. There you go. All right, pick a different language that you don't maintain. Oh, man. Let's say C Sharp. That probably maintains that one, so it's even better. On a serious note, if you go to RabbitMQ tutorials, you will find six tutorials. Almost all of them are ported to, I don't know, probably 10 or so languages. And, of course, we try to use libraries

that we would recommend to particular beginners. But just in general, what else is there? I think because messaging is, at least to get started, it's a relatively small API surface area. You can probably compare a couple of libraries quite quickly. In my opinion, you should start with the one that's documented best. And that's both because I care about documentation of open source projects, but also because I think it usually has a correlation

with the amount of time the maintainer spent on it. As I said, always true. I think Rabbit has a very good Haskell client, but it doesn't really have much documentation besides the types, right, which only tell you so much. Right. So, yeah, just take a look at what is mentioned on rabbitmq.com. There are client libraries that are so bad in some ways

that we don't have to do this often, but we recommend against. So yeah, if you see that a client is not mentioned on rabbitmq.com, chances are it's on that list or it's very, very new. Could be that. If you're still not sure, come to Rabbit and Q-Users, our public mailing list. It's a Google group. And ask.

Yeah. Very good. Good answer. I'd agree with that. I mean, looking at the client and developer tools list, it is a quite long list on our website, the devtools.html. And yeah, I mean, you would use the principle, as Michael said, as you would with any open source project, which pretty much all of these are.

You know, is it documented? Is it, you know, recently, you know, is it maintained? Does it have an active, you know, community? Do people respond? Yeah. So it's the same kind of thing there. And obviously, go for one of the supported ones. You know, you obviously can't go wrong with that. So it's the same kind of thing there. And obviously go for one of the supported ones.

You obviously can't go wrong with that. Yeah, that's kind of what I was looking for was using these client libraries as a lens to a bigger question, which is how do you choose a dependency? And I think you guys drilled it. The problem is, maybe it's not a problem, but the situation is it's a holistic decision. It's difficult to quantify you have you look at the docs you look at the code perhaps you look at the community

tests perhaps and you see is it is it maintained or are there you know 600 open issues and uh 40 pull requests against it and none of those have been addressed and so there's like all these things have to go into it before you make a decision. So I was just curious your guys' take on that. It sounds like it lines up pretty well with my own. So let's get back onto the topic of mistakes made. So not preserving the quality of client libraries over the years has been a mistake because ultimately everybody comes in and

thinks it's the server. Cause it's gotta be somebody else's code, right? It can't be mine. And so you probably have a lot of stray issues or misfires with regards to pointing the blame with not keeping the quality of those client libraries up to par. What else you got in terms of things you guys have learned from mistakes

made? Oh man, how much dirty laundry are we building through there? What else you got in terms of things you guys have learned from mistakes made? Oh, man. How much dirty laundry are we building through air? So let's start with the protocol because it will be a short one. Because to be honest, this topic is so politicized as far as the technology community goes that, yeah, I'm going to be careful. So different protocols, many messaging protocols, they'm going to be careful. So different protocols, many messaging protocols, they look similar on the surface.

And like I said, it's relatively small API area. But don't let that fool you. There are protocols that are much better designed and don't make operations of your system harder. Remember, this is a distributed system. 98% of people don't know how to debug them, myself included.

So there are protocols that had operations or features that ended up being not used. The MQP-091 has a few of those. So we deprecated them, we extended it with a few others that are actually useful. But yeah, it's very difficult to get a protocol right with the exception of TCP. And even TCP had like dozens of extensions, right? Try to Google for TCP RFC and you will get lost. Except for TCP, I actually cannot name a protocol which just gets it right and it's great and it works at web scale whatever that means to you and yeah so it's a very difficult problem so next time you are trying to

design your own messaging protocol which a lot of people perceive as a trivial issue yeah don't there are enough of them Probably, almost certainly one of them suits your needs. So these protocols, the queuing protocol, AMQP, a few others supported by RabbitMQ, Stomp, MQTT, do these operate at the application layer, like on top of TCP, or are they at the TCP, like instead of TCP, or on top of? All of those are based on TCP.

There are application layer protocols. That said, RabbitMQ has an add-on that accepts UDP traffic and sort of republishes it. It is kind of a protocol and not a standard one by any means. It's not TCP-based, not a standard one by any means. It's not TCP-based, but predominantly messaging protocols

are TCP-based, I think for good reasons, but of course it doesn't have to be that way. So one thing you mentioned is that the protocol conversation is politicized. Is that because different corporations come up with their own protocols and then try to get everybody to use them? Why is it a political thing and not just a technical thing? That's a good question, why politics exist.

I'm not necessarily ready to answer that, but I have a couple of thoughts. One is messaging protocols are typically designed by committees of some kind. There are exceptions to that. There are protocols that were designed at a particular company or just by one particular person who wrote the spec and then published it on the internet. But most of them are designed by committees and even though typically people in those committees are technical and good meaning and have experience with existing messaging

APIs or technologies, like I said, it's a hard problem. People make mistakes. People make compromises that make engineers' lives harder, even though maybe for sales people it becomes easier. And yeah, everyone tries to get their idea in some of those vendors are actually they compete in the market in multiple areas especially these days where huge corporations you know everyone competes with everything everyone else on everything and so on and so forth and then there is marketing there are protocols that there are

people who deeply believe that to build an iot, you need to build it using a particular protocol. I'm sorry, but that's bullshit. To be honest, you can use anything. You can use HTTP 1.1, which is in many ways a great protocol for messaging, probably not so much, but you can use HTTP 2, which is much, much closer to messaging protocols. But it just doesn't matter. There are ridiculous marketing claims like this binary protocol is more efficient than that one by 20%. And when you take a look at what they actually compare, the kind of workload, yeah, they compare the smallest frames or whatever it is called in that protocol.

And that frame is usually sent only once during connection lifetimes. For those not familiar with messaging, in most messaging protocols, connections are supposed to be long-lived. So you have, you know, days, potentially days of traffic, but you only measure like something that is sent in a fraction of a second, and then you come up with a marketing material. So things like that. It's very depressing if you're an engineer and you have to navigate this thin ice, and you have to implement it,

and sometimes you have to implement more than one thing, and then people from different committees come to you and ask you, hey, why do your tutorials use this protocol and not mine? Yeah, that kind of stuff. So RabbitMQ out of the box today in 2017, if you downloaded the server, fired it up, and attached to it the client, which protocol would you be talking?

Would it be AMQP 1.0? What's the default configuration? So the default configuration for all the clients that we publicize is the AMQP 0.9.1. As Michael mentioned earlier, AMQP 1.0, it's a very different protocol. It's a completely different protocol, really, than its predecessors, if you like it. So AMP 1.0 was standardized probably 2012, which is obviously way after Rabbit was started, while the AMQP protocol was still at... Michael, you might correct me here. Was it 0.8 that RabbitMQ started to implement of AMQP protocol was still being, still at, Michael, you might correct me here.

Was it 0.8 that RabbitMQ started to implement of AMQP? Yep, the oldest one we still support, actually, we just don't advertise it, is 0.8. It's very, very close to 0.9 and 0.9.1. Yeah, all those are kind of compatible in a sense. But then after some pause, I believe a slightly different committee got together and standardized AMQP 1.0, and it ended up being a very different protocol, not containing certain parts of the previous versions. However, it has been Oasis standardized and RabbitMQ implemented, but we implemented it as a plugin.

So it's an optional adding. Gotcha. Just looking up that, I was trying to look up that AMQP 1.0 release date and I found it on Wikipedia, 30th of October, 2011. It also says the working group, we talk about politics and why they exist in these things.

The AMQP working group grew to 23 companies. This was back then, including Bank of America, Barclays, Cisco, Deutsche Börse, Goldman Sachs, JPMorgan Chase, Microsoft, so on, Red Hat. You could just keep going and going and seeing. Once you have that many vested interests involved, and that's just AMQP 1.0. Like you said, there's many others.

It's hard for mere technical prowess to become, I guess the meritocracy that we all engineers want to see exist in the world is shrouded by the corporate interests. And that's just the facts of life right now. Yeah, I mean, when you've got that many actors participating to a protocol, I mean, you have to satisfy them all, right? In some shape or form, right?

So the protocols end up being complex. I mean, AMQP 0.9.1 is a reasonably complex protocol. AMQP 1.0 kind of adds a couple of notches onto that in terms of complexity. That said, it doesn't mean they're not well designed. It just means that they are very large and complex, and it takes time to implement them. And there are other protocols which are simpler,

which are more single purpose. These are general purpose protocols. You could do so much with them, which is why they are so complex. Well, we can probably talk about mistakes made and lessons learned all day because you guys are open to. And I'm not saying it's because you made so many of them,

but I appreciate we learn so well from our own failures and from the failures of others, especially failures of others, especially failures of others who have had success. So let's do one more mistake made that you guys want to hit on, if you have one, and then we'll take our break and come back and talk about the project kind of in a holistic community sense on the other side of the break. But give us one more mistake and we'll go from there.

Oh, sure. Homegrown distributed systems algorithms. Do not do that. Many issues that people report either directly or they are the root cause in RabbitMQ today. And by the way, for the record, what some pundits say are issues in RabbitMQ, what actual users who deploy it at scale think are key issues are. There is some overlap,

but they are not the same thing. So distributed system algorithms, right? Yeah, Rabbit has seen its fair share of reinvented wheels, and some of them were okay. Also, remember that in 2006 in particular, the awareness of certain

distributed systems papers was much lower. I'm not saying that the papers weren't there, although some like the Dynamo paper which started React, Voldemort, Cassandra, it wasn't published. Some others were around, but the awareness of the developer community about them was very different. So that's one of the reasons why Rabbit has a few things that were reinvented. And we will be replacing in the next year, or however long it takes. Because it's a bit like replacing a chassis on a plane that is in flight. It takes time and you have to be careful.

So I think, yeah, those things result in very, very, very real operations, technical operations-related ramifications, and certain issues and considerations, like about how do you design your apps, and so on. It's a very deep topic, but in general, if you feel like you need to invent your own consensus algorithm or replication algorithm or cluster membership algorithm, don't. There are enough of them published by PhDs or people at companies that are, you know, 100,000 times larger than yours, likely,

there is something for you to pick, investigate, toy with, and maybe eventually adopt. That's a huge and very, very costly mistake to make. So if you had the opportunity to start fresh today with a brand new, and you had the same goals in mind for the message brokering system, everything that RabbitMQ does, but none of the baggage, none of the history, and none of those homegrown distributed systems,

algorithms in there. So take the plane out of flight, to use your analogy. What would you do today? So I can cover this briefly, and Carl definitely has opinions about that. So RabbitMQ has a mirroring implementation, which we are already replacing for 4.0, which in many ways was a mistake. You can take a look at github.com slash Rabbitmq slash internals. It's a repo with a bunch of markdown documents that describe how it works internally. You can take a look. So that is one thing we're

exploring. Raft, although it can be anything else, to be honest, Raft just seems much closer to how Rabbit operates internally in many ways. Another thing that I would do is, so Pivotal sells RabbitMQ on Cloud Foundry. So it's a sort of RabbitMQ as a service that you can deploy on your own infrastructure or a public cloud. And as we were automating that, we had to make certain decisions

because of how our deployment tools work. And some decisions there involved again a custom consensus algorithm that has let's put it this way, it depends a lot on a particular node which you can choose, but it's still one node. Yeah, there are all kinds of things that can go wrong with that node that affect how easy it is to automate and operate your data service. So those are, again, very important considerations. I would definitely revisit those.

We were lucky to address that with very little code and a reasonably straightforward decision, but that's another area where I would definitely like to start from a clean slate. And lastly, RabbitMQ supports multiple protocols, but internally it's, in some areas, very heavily skewed towards one protocol. Again, it started as an implementation of one protocol, and then we added three more. In fact, there are plugins and that even more. It's obvious what we should have done in retrospect. It wasn't obvious back then.

But having a protocol agnostic abstraction would help us in many ways. But it's not so much an operations or scalability thing. It's mostly protocol semantics often do not match and it is confusing to the user. In fact, sometimes it is confusing to us who work on this every day. Yeah, Carl, maybe you have something to add or go deeper on one of those. Yeah, maybe just kind of fill it in a little bit. So if we had to start from scratch, I mean, so I want to attribute some of the mistakes down to the temptations that writing code on the Erlang VM kind of gives you, you know, because, you know, Erlang, it does have things that other platforms don't have. By default, it has things like distribution, so you can connect Erlang nodes across different machines. Now, the original use case, as those who know a bit about Erlang are well aware of, is switches,

where you have maybe only two machines or a small number of machines that sit in there. They're all connected through the same switch, so very reliable hardware. And you don't experience all the problems you might experience when you deploy Erlang, distributed Erlang. You connect all these nodes, and you deploy them into a cloud environment, right, where you might actually be crossing the internet in order to connect.

You might, you know, you're definitely crossing. You don't really know what kind of hardware your infrastructure is running on. So I think that's one of the things I would address crossing, you don't really know what kind of hardware your infrastructure is running on. So I think that's one of the things I would address right up front, rather than kind of writing something in Erlang and then making it distributed, which I think is how RabbitMQ kind of moved from being, you know, how their clustering approach kind of happened, but instead kind of address clustering and how it distributes this data from the get-go to build that into the core of the application.

I think that's very important. I mean, it does reasonably well in a cloud environment, but in order to do really, really well, you kind of need to design that in. And yes, Michael talked about achieving consensus and building safety in things like queue mirroring. That's another thing we would address. I would definitely address upfront.

May I add something to this, Carl? Sure. So you bring up a good point is that distribution, going distributed should not be an afterthought. You typically cannot go from a system that is designed for a single node to distribute it easily. In some cases you can, but in most it will be very, you will make a lot of mistakes. For example, it will heavily influence your protocol. What do I mean by that? So for example, most messaging protocols assume that that client connects to a single node. That places a lot of limitations in how your data can be partitioned across

nodes, to how you can handle failure in the client, to consistency levels that you can achieve maybe. For example, Cassandra has a, well, it's not a messaging system, but it's a distributed system that I really, really like. It has Cassandra clients, modern clients at least, connect to multiple nodes at the same time. And they can do things that you absolutely cannot do if you have A, only connection to a single node, and B be a protocol that just completely overlooks and glances over this area. This is where purpose-built protocols can really help you. And you will feel it

once you have to operate your stuff in production and teach developers about how to write their code that is resilient to all kinds of failures and that kind of stuff. And indeed, I agree that in Erlang, there are libraries that... In the standard library, there are primitives that are distributed. And unfortunately, some of them are actually pretty shitty quality. Or at least they cannot be used as a general purpose library. You should really think hard about what it does and how it works before you adopt it. And it's very tempting to just take that and here I

have a three node system, oh and I can deploy it and it kind of works. Yeah things get a lot more interesting once they start failing and that's where you realize that some of those awesome built-in libraries are not all that awesome after all. After the break, we talk about one of the most important aspects to an open source project, it's community. We talk about how community fits into a corporate-sponsored open source project that's 10 years in the making, how it's grown, how it's changed over the years, and what you can expect from the future of RabbitMQ. Stay tuned. This episode is brought to you by Hired.

Hired matches outstanding people with the world's most innovative companies. At Hired, your dream job is waiting to apply to you. Instead of endlessly applying to companies hoping for the best, Hired puts you in control of when and how you connect with interesting opportunities. The best part is Hired is completely free to you. It won't cost you anything. In fact, they pay you to get hired. Head to Hired.com slash changelog. Don't Google it. This URL is the only way to double the hiring bonus to $600. Once again, go to Hired.com slash changelog to learn more. And by Sentry. Sentry shows you

everything you need to know to find and fix errors in your applications. Don't rely on your customers to report your errors. That's not the way you do it. Use Sentry. You can start tracking your errors today for free. They support React, Angular, Ember, Vue, Backbone, node frameworks like Express and Koa and many, many other languages. That's just JavaScript I mentioned. View actual code and stack traces, including support for source maps. You can even prompt your users for feedback when front-end errors happen so you can compare their experience to the actual data.

Hit the changelog.com slash Sentry. Start tracking your errors today for free. No credit cards required. Once again, changelog.com slash Sentry. And now back to the show. All right. We're back talking RabbitMQ with Carl Nielsen and Michael Klischen, staff engineers at Pivotal, working on RabbitMQ for many years now. And we want to see how RabbitMQ plays

with the greater open source community. You guys work for Pivotal. Many of the team members do. And it's always interesting to hear how projects like these interact with the outside world, so to speak. So let's pick back up with the history a little bit. We talked about how it started back in 2006, published, I believe, in 2007, because that would be your 10-year anniversary. And Pivotal didn't quite exist back then. So tie us back into the story of Pivotal and RabbitMQ

and then how the community fits in. Sure. So back in 2006 or 2007, it was Rabbit Technologies. And I think the team was like three people. And in 2000, maybe 2008 or 2009, RabbitMQ started getting some traction. That's when I discovered it, at least. And messaging and the need for more open source messaging technologies started emerging.

And by 2000, I think, 10, VMware, or rather Spring Source, a subsidiary of VMware, acquired Rabbit Technologies. The team has gotten a little bit bigger, maybe five or six people, if I remember correctly. And in 2013, EMC and VMware decided to put a bunch of their acquisitions into a single company, which has this platform and cloud-centric focus. And RabbitMQ turned out to be one of those projects. So what kind of project is it? Is it run entirely by Pivotal staff?

Is there outside companies working on it? Are there individuals? What's the team look like? So our team is seven, I think, soon to be eight engineers, and a couple of folks who work on, you know, maybe less engineering topics. And we have contributors from companies such as, I don't know, Mirantis, a lot of small companies. I believe someone from a huge state government-owned German corporation contributed recently. So yeah, there are all kinds of users who happen to contribute as well. And I would say overall, we have maybe 10 to 11 people

who are active or regular contributors. We use GitHub for almost everything. So it's relatively straightforward. The only thing we don't use GitHub for is questions and root cause analysis and please investigate this for me kind of issues. But the rest happens on GitHub, and so it's pretty reasonable. Okay. So maybe describe the relationship between the project and Pivotal

in terms of what it does for Pivotal, how Pivotal makes money around it, how it supports seven people working on it that are on salary. Lay that out for us. Right, so Pivotal is a bunch of projects, but the crown jewel is Cloud Foundry, which is a platform as a service. We're not going to go into too much detail around that,

but it's a lot of services on top of IaaS, your AWS, Google Cloud, vSphere, that kind of tools that let you think in an application-centric way, right? So if you have ever used Heroku, you probably know what I mean. Your unit of currency is an app. You push it. It is run for you. You don't

necessarily care how that is happening. You have data services accessible to you, MySQL, RabbitMQ, Redis. You can add anything, to be honest. Cloud Foundry is very extensible. And then there are various monitoring and security auditing and

deployment and continuous delivery tools, a bunch of spring integration libraries, and so on. So Pivotal is primarily a Cloud Foundry company. I wouldn't expect other people to disagree. And RabbitMQ is one of the data services that Pivotal supports, but it's a relatively interesting one. So the most popular data service, as far as I know, is MySQL. That's not very surprising. Then there are RabbitMQ and Redis. And RabbitMQ and Redis are probably a bit more specialized, right, compared to MySQL.

MySQL is just about everyone uses it for something. RabbitMQ's role is once you have a platform, well, first of all, you have a bunch of apps there and it's cloud native and microservices ready and all those buzzwords. But those microservices need to talk one way or another. Otherwise, why do they even exist? So you need messaging.

At least in many cases, messaging is the right choice. I'm not saying it's RabbitMQ, it's messaging in general. So you would use RabbitMQ because it's available or one of the other services. I think many use Redis because it's efficient for them, right, for messaging. And so RabbitMQ becomes this glue, the enablement layer for microservices and that spring builds on top that other tools use. But there is another aspect. So imagine that you're a huge corporation somewhere on Wall Street.

You adopt a path of some kind. It can be Cloud Foundry, it can be something else. You still have decades of legacy IT infrastructure, right? And you're probably not going to throw all of that away you still have decades of legacy IT infrastructure, right? And you're probably not going to throw all of that away just to be microservices and cloud ready. So you need to interconnect this new world, which is shiny and great. For the record, I really enjoy using Cloud Foundry

when I have to. It's a very, very nice tool. But you have all this, you know, stuff written in the 80s, 70s in Cobble and LDAP in Windows and lots of Java 1.3 kind of stuff running. You need to, again, integrate all of that into the paths. That's, again, where messaging comes in.

And I think some of the traits and differentiators of RabbitMQ, thanks to Erlang and certain design decisions, because not all of them were mistakes in RabbitMQ, it's very extensible. And I typically say it's too extensible. There are people who do crazy things as RabbitMQ plugins. So Rabbit integrates very well into existing infrastructure.

It has supported LDAP for I don't know how many years, probably seven or eight. with MQ plugins. So Rabbit integrates very well into existing infrastructure. It has supported LDAP for I don't know how many years, probably seven or eight. It supports different protocols, which matters when you have to integrate with existing systems. It supports all kinds of client libraries, including relatively obscure things such as Common Lisp

or OCaml, as much as I like OCaml, and I know Carl does. So yeah, name a language, name a protocol, name something to integrate with, and there is a good chance RabbitMQ can help you bridge this gap. So RabbitMQ in this sense becomes this oil in your Cloud Foundry engine that lets it run smoothly and integrate with the rest. And yeah, in some cases even make it just feasible to adopt that kind of stuff. But Pivotal Steels also sells support for RabbitMQ.

And we used to have, we have a commercial edition. It's the same code or 99.9% is the same code. We used to have certain commercial extensions, but I think a year or slightly more ago, we open sourced all of them, and PrivateMQ is as open source as possible. We only use private repos for internal docs and infrastructure, but unless you work for Pivotal,

you're not going to contribute to that. Yeah. Very good, very good overview. How about one last question, you know, you probably, you're not going to contribute to that. Yeah. Very good. Very good overview. How about one last question? Because we're hitting up against our time here. How important to you guys personally,

and I guess Carl and Michael, you guys can both answer this separately, was RabbitMQ's open source-ness, like the fact of it being an open source project and not completely proprietary. How important is that for you guys personally, working on the project, working for Pivotal, and being a part of it? Is that something that you guys care about,

or does that just happen to be the case? Yeah, I think we all, everyone on our team, are strong believers in open source software. I think it's important for us. Many of us probably have spent years in the enterprise writing stuff that no one ever ever really seen. You only see the effect of it. And actually to be able to contribute to such a long-standing 10-year-old project, really well-known open-source product,

it's certainly something that you can be proud of and something that you enjoy talking to people about. So for me, it's very important. Right. So I think it's great to have an opportunity to work on open-source projects. So thank you, Pivotal. But I see it slightly differently.

I think that working on an open source project is awesome, but it also has less than awesome parts that are rarely discussed. And in fact, I think some relatively recent episodes of the ChangeLog, so please go listen to that, cover topics such as sustainability of open source and stuff like that. Working on a reasonably popular open source project means that whether you like it or not, you're getting questions 24-7. You have high expectations from your users, even though they haven't paid a dime for your services.

Most of them will not provide, not only they won't contribute, they won't even provide details for you to reproduce an issue. And yeah, some of them are very snarky on Twitter and stuff like that. So I have to say it's generally awesome and I enjoy it. But there are also aspects that on some days you think, hmm, I wish no one had access to this project's tracker or I didn't have to answer on this mailing list to this person. Right. And so on and so forth. So, yeah, there are also those days and not enough people i guess speak speak up about them but in general it's fun and my favorite part is you get

to communicate with people from all over the world if our website analytics is any indication something like 40 plus almost 50 percent of RabbitMQ users are actually in Asia or let's say emerging markets. And yeah, you interact with folks who may or may not speak good English. I'm not a native speaker myself, for example. And sometimes it is frustrating. Sometimes it is hilarious. But you discover use cases and opinions and points of view and concerns that do not exist in the country where you are based.

This is pretty awesome, and I honestly have never had this kind of experience in working on closed source projects, even though I definitely had colleagues in various countries. Yeah. Well, thank you for pointing that out. I think it's important to give the other side of the story, especially since we do believe in open source and we think it's the better way, but that doesn't mean it's always the great way. It's not all, as I say, unicorns and rainbows.

Just to point out a couple of changelog episodes, like I said, we have been talking about this a lot lately. For those interested, check out 242, which is The Burden of Open Source with James Long. We also have episode 246, First Time Contributors and Maintainer Balance with Kent C. Dodds. And then finally, another recent episode is Open Source Lessons Learned with Zeno Rocha, episode 248. We'll link those up in the show notes. So those who like that topic

and want to think more about that and hear from other people talking about kind of the other side of being an open source maintainer can listen to those. Guys, this has been a great conversation. Thanks so much for joining us. Any last thoughts before we let you go? I would like to, again, if this episode

was a couple of sentences, I would say please go hug your open source project maintainer, whatever you use, because it's a bit not always rainbows. Some encouragement sometimes helps a lot. And don't reinvent distributed system algorithms. You have no idea how much trouble you would be getting yourselves into.

So yeah, over to Karl. Yeah, I mean, I agree with all those points. RabbitMQ, it's a 10-year project. It's deployed far and wide. I think I'm just going to let RabbitMQ, it's a 10-year project. It's deployed far and wide. I think I'm just going to let RabbitMQ speak for itself, really. Very good. Well, again, congrats on 10 years

of RabbitMQ to you and your team and to the community. And thanks a bunch, guys. Hey, everybody out there, go hug a maintainer. Thanks, guys. Thank you. Thank you. All right.

Thank you for tuning in to The Change Law. We love producing shows like this that celebrate the big wins in open source. Share us the pains, share us the struggles, but ultimately it's something the community can build around. If you enjoyed this show, share it with a friend or two. Rate us on iTunes. Thanks to our sponsors elixir conf circle ci hired and sentry also thanks to fastly our bandwidth partner head to fastly.com to learn more and we host everything we do on linode cloud servers head to linode.com change log check them

out support the show the change log is hosted by myself adam stachowiak and jared santo it's edited by jonathan youngblood and the awesome music you've been hearing is produced by the mysterious breakmaster cylinder you can find more episodes just like this at changelog.com or by subscribing wherever you get your podcasts thanks for listening Thank you. Bye.