The Changelog: Software Development, Open Source - A protocol for dying (Interview)
Episode Date: June 4, 2016Since airing this show, Pieter passed away due to his battle with a metastasis of bile duct cancer in both lungs. But rather than listen to this show with sadness, listen with a happy heart and let's ...celebrate Pieter's life, and what he has accomplished. Thank you Pieter from the bottom of our hearts for your time on this show and for all that you are. You are loved by us my friend. This show will forever be a very special show for us. Pieter Hintjens is the creator of ZeroMQ and The Collective Code Construction Contract (C4), a writer of many books and protocols, as well as a developer with decades of building software and communities -- he's someone who's given so much, and continues to give - even up until the time he is planning for his death.
Transcript
Discussion (0)
I'm Peter Hinchens and you're listening to The Change Log.
Welcome back everyone. This is The Change Log and I'm your host Adam Stachowiak. This
is episode 205 and on today's show we talked to Peter Hinchens and we talked to Peter about some
really deep subjects. Specifically, if you follow Peter, you know that Peter's dying.
He has a metastasis of Bobbitt cancer in both of his lungs.
He's not expected to live very long, a couple months based on what he's told us.
This is a sad show, but at the same time a happy show because we talked to Peter about his extensive open source career,
his career in software, the books he's written, how he's given back,
how he cares about open source communities. Such a deep, deep show. I really hope you love this
show because this is one of those shows for us that just was magic to make. And it was a pleasure
to have this time with Peter. Our first sponsor of the show is our friends at CodeShip. And I
talked to Ethan Jones about the high performance and security of their new Docker platform.
Take a listen.
So we built CodeShip with Docker with security very much in mind.
So at the start of every build, we spin up a fresh EC2 instance.
At the end of that build, we spin that instance down.
An instance is never reused even between your own builds, much less between builds of any other customers.
We also don't do things like cache your dependencies or cache your images locally on our infrastructure. So rather, we support remote caching for those things on your own repos.
And all of that is basically to get to the point where your code is never living outside of the
CICD process. None of your application is ever stored on our servers
and nothing you push through CodeShip
is ever going to be saved, persisted, reused,
artifacted anywhere once that build completes
other than the explicit commands
you run in the middle to do that.
The side effect of that architecture
is that because these things happen
on these EC2 instances in the middle,
that gives a lot of flexibility for performance because you can scale up that EC2 instance or scale it down based on the
trade-offs you're looking for. So if you want a lot more resources or if your application has
a ton of read-write ops or a ton of memory usage, we can sort of up that EC2 instance for your
builds. So it makes it really flexible in this sliding scale performance way.
But the conversation was really more around security
and around keeping everything as protected
as we possibly could.
How nice is it to get performance
as a side effect to security?
Yeah.
That's awesome.
All right, that was Ethan Jones of CodeShip
talking about performance and security
with our new Docker platform.
Head to CodeShip.com slash changelog to learn more.
Tell them we sent you.
Use the coupon code, the changelog to learn more. Tell them we sent you. Use the
coupon code the changelogpodcast2016. Once again, the changelogpodcast2016. It'll give you a 20%
discount on any plane you choose for three months. Head to CodeShip.com slash changelog.
And now onto another show.
We've got Peter Hintjens joining us today.
And this show, Jared, like many shows we've said before,
has been pretty much years in the making.
I remember linking out to Peter's blog several times over the past couple of years
in Change Law Weekly.
And if you're a reader of Change Law Weekly, you've seen that.
Lots of great recent posts from Peter from that blog there. years in ChangeLog Weekly. And if you're a reader of ChangeLog Weekly, you've seen that lots of
great recent posts from Peter from that blog there. But what's the best way to open the show
up, Joe? We got a fellow that's been in decades of software development, leading communities,
ZeroMQ, Living Systems, several books, an expansive blog. I mean, how do we open this
show up? What's the best way? Yeah, it's a tough one to know exactly where to start. Of course,
we always like to talk about people's origin stories so peter we definitely
want to hear that uh from you but maybe we'll first mention that peter's a large majority of
his work has been building distributed systems and protocols and he's written 30 protocols
and even recently on his blog he's written a protocol for dying which I think is something that we'll definitely have to talk about as well.
Peter, thanks for joining us.
Glad to be here.
Thank you, guys.
So I guess the way we typically start a show like this, especially with someone with such deep history, I'm sure you've shared it sporadically or specifically in your blog.
I haven't seen an exact post that just shares your history. There's been some recent posts that have touched on that, but how far
back should we go to find out what got you into software development? So I was very lucky because
I chose computer science in the days when this was just a very young thing at university. It
was the first year I think they had this in York University in England where I studied. And it was the most boring, tedious, horrible stuff. It was just better
than the math, which was even worse. Stuff like database normalizations, and I couldn't stand it.
And my mom bought me a little computer, a VIC-20. And I began writing games in BASIC, as many kids do.
And then I began writing games in Assembler, because the BASIC was limited.
And then Commodore very kindly sent me a Commodore 64 and a printer and a disk drive for free.
Wow.
And I was selling these games, and the Assembler was kind of, it's a lovely CPU.
It's a RISC.
It's got almost eight instructions, I think.
It's almost got nothing in hardware.
You do everything in software.
And I got into this just obsessively.
And I was doing really badly at university.
I mean, I was getting, you know, like 30%, 40%, 60%,
really bad results.
So I took a year off and just programmed for a year
and made games and sold these games
and paid off my university costs and so on.
And then after a year of that,
I went back to university and picked up,
and it was all new.
I had not seen this stuff before.
It was all from the year that I'd missed,
or the year after me, kind of.
And it was so easy.
And my brain just kind of switched into absorb mode and I understood how things worked in the computer you know in computers
and software I figured it all out and I could just pick up a syllabus and read
it through one time and understand it and that's how I went through the exams
and got really good results although my end degree was pretty useless and so there was this kind of love with you know making difficult things work and learning it and i was really
really good at that and i got a job building tools this was in belgium where i ended up by mistake
thanks to military service and i got hired by a company although i was much too young but they
hired me anyway despite that and we were building tools in a small team for this huge company selling really crappy business software.
But using our tools, it was pretty good business software.
And so I was always building kind of infrastructure from the very beginning, tools for making other things.
What do you think the reason is for that?
Why do you think you were trying to do the infrastructure side versus like a front end side or something? Well, you know, selling games is fun and getting checks in the mail with money or even
cash in the mail is a great thing. But when you make stuff that you can give to other
people to make stuff with, then it's somehow more fun for me. You have this, you see the
results of your work multiplied and you can make things that can last longer that
can be reused above yeah and i really like that when you make things that can be reused
over and over again the tooling and the layering in software has become for me the most important
thing the reusability of what i design you know you start with this this is going to last for two
weeks this is a very simple thing people will will use it, they'll throw it away.
And then you think maybe it will last for six months.
And at some point in my work, I said,
okay, this thing I want to make, I want to last 50 years,
which it didn't.
What we were making then was a protocol
which was quite trashy in the end, the MQP.
But that concept of making stuff that lasts 50 years came out,
and I think that's a valid goal.
Do you think anyone else entering software today has that kind of goal?
I mean, it seems like we're in an instant gratification,
kind of keep it and throw it world today where something new comes out every year
and everybody says a year in software or in the Internet world
or Internet age is like seven years,
compared to like dogs or whatever.
Do you think people have that kind of perspective?
And if not, why?
I don't really think it's a generational thing in the sense that there was a time when there were more hackers than there are.
I think that's like a certain slice of people that just really like to understand how things work and are
interested by by solving that kind of problem and have the capacity for it because it can be very
difficult you're talking about concentrating on things for months and months and months
and concentrating on the right things that's also very difficult to figure out what to actually work
on and i think that that that mindset exists I think before computers, it was doing other things.
Today, it's still there, and it's still looking at problems
which are very difficult, maybe not the same level of problem
as we worked on when I was 20 or 30.
But I don't think it goes away.
That's an interesting perspective to say, to find the right things.
I know, Jerry, you could probably share this with me.
I think it's the journey to find the right thing for you to work on.
People say their dream job or whatever,
whatever term you want to use to describe that.
I think it does take a while,
especially as chaotic as technology is these days, to get into it.
There's so many layers.
There's so many things.
Just to get to ground zero, which is writing code,
is sometimes a journey all on its own.
People have that problem every day.
But finding the right thing, the right thing to work on.
It seems like, Peter, you found the right thing fairly early and then just sort of iterated on that right thing for you.
It's pretty random.
And there's all this constant pressure to do the wrong thing.
And people paying you, people telling you what you should be working
on there's these huge waves of fashion in software that you know this is the new thing and everyone
should be working on this and i guess i was very lucky in my first job to work with the guy life
um a danish kind of a crazy astronomer programmer and he had this very simple rule that you know
everything is bullshit and the more
people shouting something the more it is bullshit and his thing was to always look at the mass
market and then look where they were wrong and look at the little areas where you could do things
that everyone considered to be impossible or crazy or stupid where there was actually real profit
and he had a real talent for that and i think think I picked that up and used that kind of for a long time,
that things that were impossible, things that were, you know,
outside the mainstream experience were almost by definition going to be
more interesting than what the mainstream was doing.
Peter, the right thing for you, if I'm following along,
is low-level communication protocols in a
and I'm sure you do other things as well but that's you know that's kind of your
quote-unquote claim to fame especially around zero MQ and the legacy that you've
built up with that project and the communities that you've built over the
years in a recent post when I referenced a protocol for dying, you mentioned that you were
lonely as a young man and perhaps even somewhat autistic. And I always think about people who
focus on protocols as their life's interest or as the thing that excites them or that they're good
at because they're so necessary and they're so low level. And of course, ultimately, it's
a mechanism or a way in which two parties can communicate. And I wonder if this loneliness,
this autism, perhaps, or your desire to communicate with people and you didn't have that,
does that play into your fascination and your life's work around protocols?
So the protocol I'm most proud of actually is one called C4,
which is a contract for collaboration.
It's the rule book that we use in the Xerium Q community.
And it's a protocol, it's defined as an RFC with all the formality
and it explains how the group of people
who are trying to make some software should work together.
And I really like that protocol because it works really well.
I mean, we've tried it now for years and it works almost magically well.
And it's written kind of from an autistic viewpoint
where you don't really care what people feel as such,
but you do care that they do feel.
And so there's this kind of very brutal approach to human nature in there where you know how do you solve bike
shedding well you look at the emotions involved you look at the at the why
people argue if you look at the fallacies that people have in their
minds and then you you reposition the things so they go away and to do that
takes a certain distance from you know it takes a certain distance if you're involved and
you really care too much then you can't do that i think so i love i mean i love working with people
and i've i've i've i think i'm i think i'm hyper social um i've made hundreds or thousands of
friends in the last years as i've gone around the world with conferences and so on and that's one of
my great pleasures and happinesses
in life is people other people i appreciate that what you had to say in that article uh well in
this article especially around your desire to reach out to when you when you go to conferences
i had never really thought about it like that like everyone there wants and expects to talk
is is a quote i'm pulling from uh from that uh that post of yours. And you said you rarely talk about technical issues.
Read the code if you want.
And it's kind of funny you're on a podcast like this
to obviously talk about this and many things.
And we'll probably talk about some technical details.
But I never really thought about when I go to conferences
that what people's desires are for going there.
And maybe even in your case, Jared's question to you
and how you're saying you're a bit more hyper-social, I think it's kind of interesting to pursue these strange, chatting with strangers or being in unique situations to sort of force yourself to talk to people like that.
Yeah.
And this hit me a while back, it was a very, I was in a panel debate in a conference in Greece, and we were discussing, someone was asking about the physics of software and why
bridges don't fall down and, you know, software does. And it just, I had this, just this wise
crack answer where I said, well, the physics of software is people and psychology. And you figure
that out and you can make solid software. And everyone was just, they were just dumbfounded.
They had no answer to that. And this struck me as like, okay, that's it.
That's the key to what I want to do is people.
And it's our limitations and how we work together
that lets us build big systems.
I think this is becoming proven and obvious now,
but then it was kind of a bizarre thing to think about.
You know, we don't work well alone.
We work well in groups.
And how do you organize
that it strikes me this interest of yours um i was surprised when you agreed to join us on the show
um and just to kind of point out the the elephant in the room if people are wondering why
peter is writing a protocol for dying um that you're terminally ill. You have, is it a, you said a metastasis of bile duct cancer?
Yeah, so I'll tell you the story briefly.
So about five years ago, well, it must have been about seven,
eight years ago, I ate some bad sushi.
That's what I think happened.
And there's this little parasite that lives in fresh farmed fish
in Southeast Asia.
And if you don't cook the fish or really freeze it very solidly,
then this parasite gets into your duodenum and attaches itself to the bile duct
and begins to produce carcinogens.
And this is one of the major killers of men,
specifically around 50 in many Southeast Asian countries.
But these are very poor people.
And this disease is basically just an ignored disease. It's almost unknown in the West and it's almost unknown in, you know,
wealthy countries. And so there's been very little study into it. And there's very few people
who survive the diagnosis. And my family has no history of cancer. We have no, I have no liver
problems. I mean, I do like to drink. I've always enjoyed
that. But I have no liver issues. And I have no, you know, other reasons for cancer. So
I guess a few years fast forward, and I begin to turn yellow. So I checked myself into the
hospital and, and they have a diagnosis of eventually of bile duct cancer. And it's operable,
they say, Yeah, we'll, we'll cut you open. And this is terrible, terrible operation.
It's a, they rewire my, my whole my whole digestive system basically and i shouldn't have survived that that normally
that was i already diagnosed by that cancer you're dead that's that's it's very fast moving and it's
very difficult to to see it until it's quite late so i've had five years to put things put things in place and you know i've always felt that this was like
borrowed time um i didn't take it i was very happy to survive of course but i was kind of
kind of euphoric for about a year and then i began to figure out okay what happens if i actually die
and i've begun to put things into place little by little and aim towards you know being being replaceable being redundant
and as far as that's possible and then i've had scans every year every six months blood tests and so on and nothing nothing nothing and then this year in um yeah february march i have this cough
which won't go away and i go and get the scan finally and it turns out it's come back and it's
metastasized in both lungs and it's it's pretty pretty invasive and so my oncologist looks at me and says well yeah
and uh now we're doing chemotherapy which is also pretty pretty brutal but it's really just
for sure i think for data there's no there's no real there's no real evidence that this
that this cancer responds to any chemo at all
so yeah so yeah you have you have limited days and so i was i was quite surprised that you'd
be willing to you know spend an hour and a half on a skype call with two people you've
never met before in your life but then i started reading more about you and and your life's work
around building communities not just around building software and this interest that you have in,
in meeting people and,
and seeing new perspectives and how that's one of the things about your life
that you've loved.
And so then it started to clarify why,
why you said yes.
Oh,
this is awesome.
It was even,
it was even tough to ask him to,
to come on the show.
I don't know if that made it actually into the show or not,
but just, you know, I've been fans of Peter's for years now,
and it was just a touchy subject.
I was like, I don't know if that's okay.
Is it okay to ask somebody to come on the show,
considering we've wanted to have you on the show for years,
and it just never, you know, became a thing.
We just never reached out yet.
And, you know, I guess on that note, too, you always feel, I mean, this is a side note for just in general, but you always feel like you out yet and you know i guess in that note too you always feel
i mean this is a side note for just in general but you always feel like you have time you know
yeah and whenever you're hit with the the moment when you don't have time it's sort of like you
know it's it's obviously not a good thing but you sort of kind of keeps you in a high gear to make
some action and do some things i think you hear my kids in the background that's okay
that's perfectly fine
my three lovely little kids
well I have time actually because I've
kind of stopped programming
for the moment I don't think I'll start
again I'm still writing
I just got a thing today
in the delivery
from China I think called FooWrite
which is this bizarre
steampunk
typewriter, which I'm trying out.
And it's quite fun. So I'm writing
and I have time.
To be on the podcast with you guys is just
absolutely awesome. I would do this every day
if I could, seriously.
Okay, well, we won't feel bad about it then.
Let's
crack open the idea of
this protocol. i noticed in the
near the end you said you're not going to open an rfc for this which is a request for comments
i guess walk us through how you actually since you've had such a history with protocols how
you actually write a protocol for this so i had these biopsies done and bronchoscopy and it was
the most horrendous thing and from that I
got these infections in my lungs so I went into the emergency with my my whole right side of flame
and they pumped me full of antibiotics and I went through this whole thing and I was lying there in
bed and I finally got my pc after about a week my laptop and one of one of my readers said look why
don't you write an update on the article you wrote about, you know, five years, surviving cancer, five years.
So I just turned out this article.
And I said, yeah, a lot of people talking to me.
And I see a lot of kind of, there's a lot of strange interactions when you talk to somebody who's obviously very, very sick and has a terminal diagnosis.
Right.
So let me try to frame this as a as a simple human readable i didn't want to make it technical kind of human
readable um story that people can look at and say yeah okay okay um and i just turned this out i
didn't really edit it or go through very much go through it very much from where i was lying in bed
very sick and it was the thing is when you're feeling really bad in hospital at least for me having visitors and having people talking to you is is a real pleasure like the
worse you feel the better it is to have company around you that's the irony yeah i don't think
everybody has that experience with me that was how it was i had days with like 10 15 people coming in
to see me people coming in from everywhere from america from finland white guy flew down consta flew down
to see me it was amazing and i wanted to capture the the positives of that and you know what felt
good and the negative some people was very strange and i wanted to document that and say okay these
things that people do sometimes are very strange and you can probably if you document it maybe
people can learn to avoid doing it yeah i certainly appreciate
the the way you frame it with bob and alice and you sort of remove yourself from it it's not like
this is what peter wants you know but it you know it's the it's the uh the thread you know that's
that's uh ran through that with you writing obviously obviously you know but Bob being the dying person and Alice
being the person who
what she should not say to Bob
I mean I know
I've seen people say in
Facebook comments like I've been
sadly in a family where we've had several people
die and so we've had some experience
in my family with loss
and people passing from us,
whether it's from cancers, instant death, you know, like heart attacks or things like that.
And we've, you know, I've been a family who've suffered loss and, you know, on Facebook or,
you know, someone says something, whether it's like early news or it's actually after someone's
passed away, they see all these weird things like hang in there or keep hope
or all these things that they're just trying to be polite,
but in the end it's just not constructive, it's not helpful,
and in the end it's just not loving.
And so speaking of this protocol you've written in a human-readable way,
and I think the way you've written it from Bob and Alice's perspective,
even removing you from this where it's not like a personal offense to Peter,
nobody, and even this show, like Jared and I, we talked earlier about how we
had this, you know, uncomfortable feeling of having you on the show,
not so much because of you, but because we didn't want to offend you. You know, I think that's
what's interesting about this protocol you've written in a human
readable way to share with people what to say, what not to say, and even how you've
embraced lots of people in your hospital room when you're, you know, when you're in the
hospital and what that's like for you.
So I think that's pretty, that's an interesting perspective when it comes from this protocol
you've written.
What I've noticed, I think there's this real taboo about, well it's a taboo caused by by distance i mean
we've we've lost this kind of naturality to to dying it used to be that we died of mostly of
old age and surrounded by our families in our own homes um i mean probably to through lack of
medicine and proper care but death was something that everybody participated in. And we've lost that.
I mean, most people, apart from,
most people die away from home.
They die in hospitals.
They die in road accidents,
but the dying isn't a social process anymore.
And that means that the living
are very, very uncomfortable with that.
It's become something taboo and hidden.
And when you have to cross that boundary and talk over it, what do you do?
What do you say?
There's just no culture for it, no background for it anymore.
We've lost that.
And I think that's avoidable.
And I think, so I'm really lucky to be in Belgium as well.
I mean, I didn't like this country in the beginning.
I came here by force.
I was conscripted and I was like, okay, screw you guys.
It's still a year of my life, but it turns out to be a pretty good country and they
have laws for euthanasia and planned death as i like to call it this notion that you can construct
your own your own death properly and i saw this with my my father who died at easter and this is
how we did it as a family. He had euthanasia.
He was very, very, very feeble, very old.
And it was a very loving and calm
and untraumatic way to end someone's life,
which he was going to die anyway.
And it wasn't about, you know, killing him.
It was about changing from a total lack of control
as a family to to control
and i think that's something which will become more accepted worldwide as a kind of a human
right this notion that you can control your own destiny you can control your own death
and i like that aspect of involving people in that process little by little.
That dialogue with people, even with, you know, even this podcast, we're discussing my death.
And that's an interesting thing, which I find healthy. It's a first for the show, honestly.
I mean, it's something we definitely haven't done before.
And it's something I never even expected that we would do.
I mean, when the show first began, our tagline was open source moves fast keep up it's still part
of our our mantra but just that we cover the fresh new of open source and this is the most unique
open source out that we've ever covered which is a protocol for dying um you know it's of many things
we'll talk about with you today so it's not the only thing but it i think the reason why it makes
sense to bring that up early is because it doesn't make sense to talk about it later in the show like it's something that the listeners are going to know about right away
as a as a foundation for the conversation we're about to have uh and go deeper into it and it
actually this is probably a good time to take a break um we're about 20 or so minutes into the
in the show and a time for a break but let's come back and touch on a couple of the things that you say
that are positive ways to talk to someone who's dying.
I think you mentioned things you like, for example.
So let's take this break.
We'll come back and talk about some things that are constructive to talk to you.
So that way, when other folks feel invited to speak with you,
whether it's on Twitter or through some social way or actually face-to-face
in these upcoming meetups you have, they have, you know, aside from your post, they have you actually saying some things that help them understand how to talk to you, how to treat you.
And so let's take this break and we'll be right back.
Every Saturday morning, we ship an email called Change Law Weekly.
It's our editorialized take on what happened this week in open source and software development.
It's not generated by a machine.
There's no algorithms involved.
It's me, it's Jared, and the rest of the ChangeLog team
hand curating this email,
keeping up to date with the latest headlines,
links, videos, projects, and repos.
And to get this awesome email in your inbox
every single week,
head to changelog.com slash weekly and subscribe.
Hi, we're back from the break and we're talking to Peter about a protocol for dying.
It's obviously a unique thing to speak about, but Peter, you've done very great with making this readable by the world and sharing different ways to talk to someone and to not talk to
someone around
who's passing away. Walk us through a couple of the things that are positive on the positive side,
how to speak to somebody that is in your situation. So it's actually quite simple. I mean,
it's company. It's having people around you. This is how I see it. Having the presence of people,
having people beside you. It's funny's funny you know the worse you feel the
nicer it is to have somebody there who's just present and if we're going to talk it's going to
be about the things which are in in my mind I mean being Bob then I'm very selfish of course I'm
obsessed by the tubes in my arms and the effects of the chemotherapy and how much I've been vomiting
that day and how you know how it's better than yesterday and how tomorrow will be and my focus is very limited I'm not thinking
about next week or next month I'm thinking about the next few minutes the next few hours that's
that's when I'm when I'm sick in the hospital if I'm at home about other things and we can talk
about stuff that we've done we can talk about you I mean, if you're in the hospital and you're sick,
then it's as Bob, then your mind is on that.
And if you're at home and you're feeling better,
then your mind is elsewhere.
But what it really comes down to is having company
and having the excuse to be together
and to chat about just random rubbish
without it being heavy or difficult or painful
and without there being this emotional burden.
I mean, that's the worst is the emotional burden,
people who are very sad or who feel very confused.
That's almost impossible to deal with.
Yeah.
I think the key thing about the protocol, right,
was, I mean, the parts about what to say,
what not to say, they're quite trivial almost.
The key thing was
involve people involve other people it's like don't you know don't hide your your your sickness
don't hide your situation don't be afraid of what people will think of you if you're actually that
sick that you're going to die we're all ashamed of being you know vulnerable and being seen as
vulnerable but if you're going to die, you're going to die.
And if your doctors tell you that you're terminal, then tell people about it.
Be honest about it.
Don't assume the worst as a kind of a, you know, the worst possible case,
but be practical about what's a likely outcome.
And involve people in that.
And once people understand that you've accepted it,
then they accept it.
And it gives your community, it gives your family,
your friends time to grieve and time to prepare and time to go through that process with you.
And I have a friend who has breast cancer.
And seven years ago, she was diagnosed.
She had a small scan and she just
refused any treatment. And that was her solution was to deny it completely. And she said, I'll fix
it myself with a diet and with whatever. And seven years later, she's still alive, but the cancer is
eating her up. And she's not told her parents. She's not told her friends. It's a big secret.
She's really ashamed of it. And when she dies and she will die it'll be absolutely catastrophic nobody will have
the chance to have said goodbye or to even have understood it so i think that's the key thing
about this protocol is you know be honest be honest with yourself be honest with your your
your community your social circles your family your friends. One aspect of that and preparing for death is sorting out your life, you know, closure,
wrapping things up, making sure that your desires moving forward are held.
That's something that many people that aren't ready to deal with their own death or they
feel like it's far away from them often don't do.
And so you leave people in the lurch them often don't do and so you leave
people in the lurch because they don't know what you wanted yeah for instance you know put you're
supposed to create a will when you're healthy not when you're sick because you never you never know
when your life's going to be over so in many ways it's it's an opportunity that you have you know
we all of our days are numbered but yours are particularly numbered so far that you don't think you have very many of them left that you can actually put your things in order and you can pass the baton.
You have kids, you have family, and so you have all those aspects which are the most important things in terms of passing down your desires and your thoughts and making sure that your kids are well established
and who their father is and all that.
But from the technical software community side, these things that you've been really
your life's work in terms of community building in the software space, there's a lot of stuff
to think about there as well.
And it sounds like you've done some thinking on how you actually pass the baton, for instance, in the zero MQ community. Can you talk to us about that?
Right. So this has been actually a long process. I've been working for years to make myself
redundant. So I'm the gatekeeper of the community. I'm the benevolent dictator for life.
And for a long time, I had to feed the community. I had to put a lot of effort and
quite a lot of money into it. I got the money back. I mean, I got good business from being
the ZRMQ guru, but it was a big upfront investment and it was a continual investment in keeping
things straight and keeping problems away from the community that were potentially very dangerous.
And so it was a very deliberate thing over the last few years to make myself less important.
I was always a contributor and I love writing code, but I didn't want to be the person making the decisions like, when do we make a release? Or do we merge this patch? Or is this person a
valid contributor? I wanted these decisions to be made by the group in a decentralized, autonomous, reliable, robust fashion. And that succeeded. And that's kind of
the core of our process, the C4 process. So we still have, of course, dependencies for things
like domain names and trademarks and ultimately the deciding voice when there is a real conflict or a real
issue. And I'm very lucky. I'm very lucky to have the time to do this, but also I'm very lucky to
have a group of contributors who are absolutely amazingly good. I mean, the level of the people
who've come to, we have these hackathons once a year in Brussels before this huge open source meetup called Fosdom.
We have these hackathons for a few days.
And the people who come there are just awesome.
They're amazing.
And one of them is my friend Doron,
who is an Israeli.
He's a bit younger than me and has more hair than me
and smiles more than me.
But he's been using our process he's been using zero mq
he's the author of the windows net mq uh library which he's rewritten three or four times and the
man is he's a fantastic developer and i've seen him work with people and i've seen him basically
as a younger version of myself so i'm like okay doron um you you idolize me i'm going to give you
the baton and he's like my god what do i do with that i'm like, okay, Doron, you idolize me, I'm going to give you the baton.
And he's like, my God, what do I do with that? I'm like, okay, basically, I'm going to pass you
the domain names. And I will tell the XerionQ community that I've chosen you as my successor.
And they can argue, they can bitch, but they won't. I mean, they trust me. And it's a good
decision. And it comes down to actually this very interesting situation where we have no foundations and we have no committees and we have no centralized infrastructure.
We have one guy who owns a few domain names.
He owns essentially 0mq.org and pays for that and has the consensus agreement of the community that he's the guy in charge.
And that's it.
And it's a very lightweight baton to pass on.
Imagine that there was a foundation with a board.
We'd be in years of argument.
Wow, yeah.
I mean, that's actually an interesting thing you bring up
because, I mean, just a year back,
we can rewind to IOJS and Node and that separation,
that forking process and fragmenting the community
and then the obvious corporations involved,
and now there's Node.js Foundation,
which they're doing a lot of great stuff,
but in light of this scenario here,
I guess it's maybe a bit more specific
when you have a BDFL model, right?
Maybe that's a little different,
whereas in the Node.js Foundation,
if that's the example we're given,
since I've said that,
it may not be the only example,
but it may be a little easier to pass that baton
because there's no particular BDFL in place.
There was, but there isn't anymore.
And I guess that's sort of the walls they've been breaking down.
I mean, that's an interesting topic all its own.
And the Foundation model is one that I've used
and I've been part of several times.
And it can work very well.
The problem with it is that it has specific weaknesses and it's vulnerable to hijack
basically and it's vulnerable to interference by bad actors. And it's actually quite simple.
You can just get onto a board by pretending to be a good actor for a while. And then you start disrupting things
and you can destroy an entire community
just by sending a few emails
and praising the wrong people
and attacking the right people.
And it's so easy to do.
And I've seen this done
and I've been at the receiving end of that
in large communities,
which were destroyed as a result of that.
And the cost of setting these things up,
the cost of having meetings and regular meetings.
And it's just endless, endless friction.
And so one of my ideologies for ZeroMQ
has always been utter decentralization.
Every project is autonomous.
Projects agree to use the rules they want to.
Any project doesn't want me as dictator
and didn't want me dictator,
just didn't choose me as dictator.
It was very simple.
I was only there by approval of each project.
And there are hundreds of 0MQ projects,
thousands even.
So it's a model which is,
I designed it very deliberately, I think,
based on having seen many worse models in my life.
And it seems to work pretty well.
How about things, maybe this is going a little too far on the subject,
but how about things around your corporation
and some things I noticed on the XeromQ site
with support models and contracts like that?
How does that change?
How does that translate as part of this?
So I never tried to make my company a very dominant.
It's visible, but it's kind of visible on the margins of CRMQ.
We've always kept CRMQ as a first-class citizen
and the business as a second- or third-class citizen around that.
Most people that use CRMQ never ask for support.
They don't need it.
And there's a few customers now and then,
people who are potential customers who want support.
They need it.
They want a contract of some kind.
They hunt around for a contact address,
and then they email me and ask for consultancy
or for a support-level agreement,
whatever it is, a service-level agreement.
And it has to be that. I mean, it has to be
some kind of commercial hotline
for people that want to pay.
Because otherwise you leave money on the table,
which is just silly. So I assume
that whoever takes over, yes, I mean, whoever
has taken over, Doran will replace
iMatics with his own company.
My company will be
folded at some point in the future.
It has no real, I mean, it has no real i mean it has no
real structure i've always kept it very small just myself really maybe a loop back around to your
c4 protocol as you said it's your thing you're most proud of and um perhaps i'm a little ashamed
i hadn't even heard of it until you mentioned it so So maybe it deserves to have some light shed upon it.
Collective Code Construction Contract, which is a hell of an acronym.
C4.
C4.
What's that word?
Alliteration.
When we get everything, start with C's.
This seems to be a model that you said is working really well for ZeroMQ.
I'm curious exactly what the model is
and if there's other projects that are using it, if it's, you know, something that you think more
people should be using, but they aren't give us the, give us the details on C4.
So where C4 came from was we've been trying to codify best practice in our community. And of
course I like writing protocols. I like writing, you know, well, contracts, honestly, I like writing contracts, weirdly enough,
protocols are contracts. And we had this, we had a number of really, really big problems in our
community for years. And these problems were difficult to understand because they were hidden
by all kinds of layers of stuff going on. But it boiled down to the conflict between a few very dominant
contributors and the mass of i would say less aggressive and less ideological contributors
and users and this conflict is is very real it happens in many projects and you'll see it when the core maintainers go off on an ideological tangent.
They annoy their whole community and it can be very, very disruptive and very destructive.
And so we had this problem and I was the person trying to keep things calm and trying to make stable releases while there was, you know, two or three major unreleased versions still in the pipeline.
Each one breaking APIs and breaking wire protocols and completely out of control process. And it ended in a fork, which I thought
was very satisfactory. And as it ended in a fork, which I provoked by writing the first version of
C4, where I said, okay, look, I own ZeroMQ. I mean, it's my trademark. I've invested in this.
And everyone here is welcome,
but you are basically working on my property.
And these are the rules as of now.
And the rules are that we will aim for maximum participation.
That's our first goal.
I don't care about the software in terms of its code.
I care only about the people and the way they organize.
And I trust that they will make the right software.
And that's already is a very brutal statement to say to people who are focused on software first
and people... Seems so counterculture. Yes, it's a very aggressive thing to say.
And that annoyed a lot of people, well, a few people. And then I said, right, this is how I
see it working. First of all, we use pull requests. I'm tired of this kind of cargo cult model we had
where you post your patches to the mailing list
and somebody may or may not merge them.
Well, it works for Linux kernel, it'll work for 0MQ.
It's a trashy model.
It's based on no understanding of actual dynamics.
It's just based on trying to imitate other projects.
Like, well, you're going to use GitHub pull requests.
That's it. End of discussion.
Everybody can use GitHub.
Everybody will use GitHub or whatever platform we're using.
It may not be GitHub in 10 years' time.
And then I'd been using this approach for my book, The Guide,
where I'd basically been merging people's contributions.
I had pull requests, people sent me code, and I merged it.
And I noticed that there were very few mistakes.
People got things wrong.
People made bad translations of the sample code.
I wrote the sample code in C, and people would translate it to every other language and send the pull requests.
And people got it wrong.
That's fine.
Others would come in and fix it.
But there was almost no misbehavior.
And I was merging pull requests. And I just stopped reviewing it. I just began merging everything, just blind merging.
And it worked very well. I'm like, okay, this model actually works. So I began discussing with
other maintainers and writing blog posts about this notion of being much more accepting to
contributors and not reviewing their code aggressively,
not making value judgments above all on their code and moving towards a kind of a contract where
you can bring in any, you can solve any problem you must solve, you can bring in changes to fix
problems, but you may not break existing applications. And that's the core of C4 was
this process for welcoming contributors and defining rules that forced maintainers to be welcoming, even if they might have their personal opinions.
And to give this kind of this right to anyone.
If you're using our product and you have a problem and you have a solution and you send a patch for that solution, it will be merged onto master on our main development branch. If you follow a few
basic rules about, you know, style and about the size of your patch and so on, commit message,
but we will not judge your patch. And that's completely radical in our business. This notion
that the ideology of architecture is a problem rather than an advantage. And this notion that your elite ninja programmers are your actual handicap and not your asset.
And it made a lot of people very worried when we started that.
And then we noticed that our contributions began growing.
We had more and more patches.
The email lists became very happy.
There was no arguments in the email list became very happy. There was, there wasn't been no arguments in the email list since then.
No more strange discussions about who did what and why.
Now and then there are,
you know,
pull requests with threads,
which get a bit long and they're not merging it.
I go in,
I just press the merge button,
end of discussion by killing is finished.
And it's,
it's very simple.
If you have a problem with someone's pull request
with their patch,
you make your own pull request on top of it.
It'll get merged as well.
And this notion that you can fix problems
by moving forwards,
not by focusing on the problem.
Right.
It's so powerful.
I think so many people get fractured
and paralyzed by that.
You know, I mean,
you seem to have like really struck a chord with this
because one thing I love the way you opened up with focusing on the people, right? paralyzed by that you know i mean you seem to have like really struck a chord with this because
one thing i love the way you opened up with focusing on the people right i mean sure you're
creating software but in the end of the day it's the humans behind it that's creating this thing
and only by the stability of that uh of that community around a project or a code base or
whatever will the actual source code become
positive in its own mission?
That to me is so profound to think like that and so countercultural.
I kind of mentioned that earlier as I tried to interject a little bit, but I didn't want
to interrupt.
I just think that's so counterculture.
It doesn't seem like that's how it is out there in the open source way now.
It seems like this is so different and unique.
I'm more extreme than that.
So, I mean, I'm kind of way out there sometimes.
So I have this, although, you know, my job has been for many, many years, software architect,
and I build architectures and software, and I can plan out whole systems in layers and layers and layers,
and they will be built and they will be shipped and they will be they will work okay that's my it's been my job for many years
but i've come to believe that architecture is a fallacy it's a fallacy um it's a bit like this
meme i saw of a some kind of a weird um dog it's been bred and it's cross-eyed and it has
you know broken ears and it can barely walk
and then there's a wolf and the meme is like you know evolution and this is um yeah what is it
intelligent design right the second one the the bread dog and basically i've been i've been
pushing kind of silently for people to understand that evolution is where we want to be going towards
rather than intelligent design. By that, I mean that as individuals, we're basically incompetent
to be solving the right problems. We think we're competent. We believe we're competent.
We tell everybody how competent we are. But in we're blind we're limited we are we are
only looking at a very very narrow area and where it's distorted by all these assumptions and
fallacies and and beliefs which are wrong and it's only by incremental trial and error with a large
enough group and a diverse group that you can actually begin
to solve the right problems. And if you accept that, then you understand that the starting point
is the group. You need the group, the collective, you need the diversity, you need the lack of
friction and communications, you need the ability to challenge any assumption, to think quickly together, like an ant nest.
I love the ant nest as a model.
Ants are extremely successful and they have this tiny genome, I suppose.
I mean, ants aren't very complex and yet they do really, really complex things.
And I think that's a satisfactory, I think it's an accurate model for human behavior
and for for technological
development and i think that the the other model where the ant is somehow a super intelligent
thing that can create out of its own brain is a lie it's a fallacy it can work by good luck but
it's just it's actually just bullshit and so my friend life would have he had a thing from texas
called the bullshit repeller which was a a spray he just spray around when someone talked rubbish.
And I think that's this conflict between the ninjas and the community
is this kind of this, you know, the lie against the truth.
Well, you definitely have the experience, you know, that bears it out.
You've been probably there with many opportunities to spray some repellent.
Let's take our last break and we'll continue talking about this as well as community stuff.
And we have some good closing questions for you as well.
I mean, lots to talk about.
We'll probably run out of time and think, man, I wish we had more time with Peter. But let's take that break and we'll continue the conversation in a moment.
For those of you out there who are super fans, and I mean people who care about this show,
listen every single week, care that we stay on the air. We want to invite you to join the membership community for just 20 bucks a year, and we'll give you an all access pass to everything
we do. Access to our members only Slack slack room exclusive discounts from our partners 50
off the changelog store and of course you support us so we can support open source
hit the changelog.com membership to learn more and we appreciate your support
all right we are back with peter hinchins and peter we've talked about a lot of things on this call we've talked about or around zero MQ which is the project that I knew you for now I'm starting to think maybe C4 will be
your legacy but nonetheless zero MQ a very interesting project and we've just been talking
about the community around it and the way you've managed it and the way you're passing the baton on
we haven't given too much to the project itself
its technical merits its history why people should be interested in and even where it's headed so can
you give us the rundown on zero mq yeah so the story is kind of it's prometheus and stealing
fire from the gods and giving it to the mortals to cook their hamburgers on
um we got involved me and a couple of other of my friends,
years ago in making messaging for the financial industry.
These were my first major protocols.
And I think at a certain point it became clear
that what we were building was actually really interesting
for other people too.
And the first people we were aiming at as customers
were smaller financial businesses,
small trading houses.
And so we focused on raw power and speed
and making it open source
and trying to get license fees
from these people who of course never paid anything
because they were so greedy.
They wouldn't pay a cent,
but it was great fun.
And then we kept making it simpler and more accessible. And at a certain
point for me, it was like, okay, we've actually cracked the problem of building distributed
systems. We know this stuff works. You know, our software ran the Dow Jones Industrial Average for
years and it worked and it was really efficient. It was straightforward. XeriumQ is better than that.
It's 25 or a hundred times faster. It's simpler.
This stuff is really, really valuable. But our target audience doesn't really understand the
problem space yet. And one of the biggest handicaps is the complexity of this whole
thing. Distributed systems are very complex. And so we've been working towards making that simpler and simpler and simpler as a metaphor
as a model as a the learning of it and when that works it's a thing of beauty so one of the stories
on my my article people have been writing on my article all kinds of comments and one of the
stories from somebody is this guy who was asked to make a prototype for the NFL.
They have put RFIDs on the players and they put little scanners around the field.
And they can collect all this data from where people are, where every player is.
And they wanted a system to bring that data in and pump it out to their machines. And so this guy wrote in a weekend, a zero MQ program in C++
that pulled in and distributed. And it kind of worked. And then he went off and forgot about it.
And a while later, his company folded, he was fired. And his friend at the NFL said,
come, come, come work, come work, come work. And it turns out that his code was being used
as the basis for the nfl's next gen
statistics system which is one of their products and it's the same code essentially hasn't been
changed very much there's a team around it but the core of that was you know i don't know how
many lines of you know 500 lines of c plus plus and it worked basically first time and it's still
working and it's the same the same model has been working out for a year and a half and very successfully and deployed as proof of concept
and deployed in production already for a year so that's being rolled out and it's a it's a product
and this is thanks to a little library which solving the right problems in the right way um not you know for a certain
class of application right always yeah and so our um our journey with serumq has always been
to make that more accessible easier to use less wrapped in mystical terminology and in strange
concepts that you can remove and it's been removing concepts, removing concepts
making it simpler over time.
Do you consider ZeroMQ
complete or do you
feel like there's a lot
that can continue to remove concepts or continue
to innovate beyond
your direct involvement?
So ZeroMQ is a community of
projects and so it cannot
be ever complete. There's always going to be new
things we're speaking of no js and one of my last gigs was to build a no js binding for 0mq a new
one which i didn't manage to finish huh so if there's somebody who knows no js and c++ because
it's a c++ nan it's a horrible beast and if someone is up to that there's a C++. Nan, it's a horrible beast. And if someone is up to that,
there's a lovely piece of work there.
Probably impossible to do, probably crazy.
In fact, I don't think anyone can do this.
SerumQ is also the core library.
It's the core messaging patterns.
And in there, we're actually still simplifying those patterns.
So for example, Doron Sommach, who is the new dictator,
actually implemented a bunch of new patterns,
which are very, very simple. They're very nice. The names are nice. There's a client-server pattern.
There's a radio dish pattern, which uses UDP. There's a scatter-gather pattern. And these are
simpler than the existing patterns. They only take single message parts. There's no multi-framing.
There's no strange structure in the messages. You send one thing, you receive one thing. And these are still raw.
They will be, it'll take a year or two before they get into broad use, but they are very elegant
and they're very nice concepts. And in fact, we stole that idea from the fork of ZeromQ from
NanoMessage, actually, which had this kind of approach, simpler.
So I'm not averse to people forking stuff
and then us stealing those ideas back.
I think that's great as well.
Why not, right?
Yeah.
Makes sense.
I've always wondered this myself
as someone who doesn't go this deep into system software.
And I kind of get the idea of what messaging is,
but can you walk us through like a
an overview for those out there who are listening thinking like you know i just i'm following along
i get it but what exactly is messaging in a system like this why does why is zero mq needed what is
it what is the true problem solves so if you're you typically use tcp to connect systems and if
you're using tcp then you have to solve solve a bunch of problems to make it actually work.
You have to frame, because TCP is a stream of bytes.
Messages typically aren't streams of bytes.
So you have to frame.
You have to handle errors.
You have to handle asynchronous, asynchronicity, because you'll have multiple threads in your application.
One will be sending, one will be producing data data and the two have to talk to each other with buffering you may send a thousand messages in
one go but the network may be quite slow so you have this asynchronicity happening at all sites
you have to be able to deal with different languages maybe you have c plus plus talking
to python maybe you have java in there somewhere or do you do you want to just impose java on the
whole you know your whole ecosystem yeah maybe people do that and and so on and you have these um 20 or 30 quite
specific problems that you do have to solve if you want to make production quality communications
between two or two million pieces of software and people do solve these problems over and over and over again.
And mostly they solve them in an ad hoc way. They have a library in their application,
which they maintain, which is theirs and which is never really looked at by other people,
which is their best guess of how to do this. And it works, but it doesn't really scale. It
has performance issues and they can push maybe 5,000 or 20,000 messages per second
through this stack.
If you take that stack and you open source it
and you put it into a community who are actually experts in distribution,
then what comes out is much, much, much better.
It's much better in every dimension,
not just in performance, but also in stability, also in exceptional conditions, also in error handling, portability, and so on.
And you get this thing which you can plug into applications or you can build applications around rather because it's a design tool and which solves these problems in a way which is which is really good which is
effective which you can rely on and which lets you build now your distributed system without having
to think about that that tcp stuff or that udp stuff happening between your pieces and i've told
i've told developers this in conferences for for years now that if you're still building monolithic
systems then you're building for the past.
And this notion that there is a computer in the building on which stuff will run is, you know,
it's 1950s. And the modern world has billions of computers. Even my house has hundreds of computers
by now, just my kids and their mobile phones and tablets. And for software to use this profitably it must be distributed it must be decentralized and
this problem of communication between these these moving pieces is is still really unsolved until
you have something like zero mq and what you can build on top of zero mq becomes then really
interesting once you've once you've taken away this cost and we build stuff on top, like Zyre,
Zyre is a kind of a ad hoc clustering library where you simply join a network and you will
discover peers automatically and you can talk to them and broadcast to them. And it's reliable. So
you can chat to peers and send them stuff and they will receive it if the network is very flaky.
And it's simple to use. It's relatively hard work to make that,
but it's doable because of XeroMQ.
And I've used XeroMQ, for example,
to build IoT demos with clusters of devices
that you switch on and switch off,
and they find each other, they can talk to each other,
they can do interesting things.
And it's just straightforward to build that kind of stuff now.
It used to be impossible.
It used to be literally years and years of work, and now it's a few weeks, a few days even sometimes.
So take us through like what systems out there are using. I mean, based on that,
it sounds like this is like the, which makes total sense to have Ilya on the learning page, like to
Ilya Grigorik, who's been on this show before, internet plumber, so to speak,
that's his self-professed title.
But it sounds like, I mean,
that whether we might know it or not,
that 0MQ is the foundation
for which a lot of the stuff we might do day-to-day
on the internet is built upon.
Ilya was one of our first fans
and wrote about it and helped us to get some footing.
The thing about stuff like 0MQ is that we don't know who uses it.
Really?
Like, literally, we have no clue.
I mean, there's no registration process.
There's no, you know, we don't keep track of downloads.
It's completely invisible.
And when things work, nobody tells us, obviously.
So we only see a very, very small percentage of users.
We see people who are absolutely incompetent to asking basic questions,
which we've answered often in documentation.
So they get told to read the guide, and that's it most of the time.
And we get people who want to contribute improvements.
So they have hit the limits in one
or other dimensions, and they've gone through the code, and they've understood the code. And
they're like, okay, I've got this problem, here's a patch. And we have people using it in a way
which creates a problem, and they're asking for help. And then we have this very small number of
people asking for commercial support. And that's it. So 99.9% of users are just off the radar.
And we have no way of,
and we don't even want to keep track of them.
It's not interesting.
Let's pull something from this post
since we're talking about Ilya.
He talks about different ways that
zero MQ is being used.
And I guess for the listening audience,
hearing what it is and hearing how it's used
is two different things.
So let's talk about it in the wild.
What are some things that people have built with it that you're aware of?
Like in his example is Mongo2.
I remember when Mongo was a big deal with Rails back in the day, but just some different
examples of how they used XeromQ to build something else.
One of the largest deployments of XerMQ. And I mean this physically,
is the large Hadron Collider in Geneva,
in France and Switzerland, run by CERN.
And it's a fairly classic industrial control scale problem.
So they have this 20-something kilometer ring.
I don't know how many miles that is.
A lot of miles anyway,
running around the countryside.
And within this ring,
they have a whole series of machines
which are producing energy beams
and amplifying them and focusing them
and steering them.
And these machines have to run,
when they do an experiment,
machines run in real time.
They're steered and guided in real time
by a software network,
which has to be able to talk to them
and tell them what to do. that's the control system the experiment runs for i don't know a few hours i don't know how
long it runs actually and they collect the data by smashing the beams together and they have this
huge massive uh receiver thing around the smashing thing.
And that produces just a lot of data,
which then gets sent off.
And that's the data part.
We don't deal with the data part.
We deal only with the control part.
And so their control structure is they have these machines.
Each machine has custom drivers, bizarre serial port control interfaces built by who knows what manufacturer somewhere.
And they have these feed into computers, just normal little Linux boxes.
I think about 500 or 1,000 of these computers on a network,
which then talk to central systems,
which tell the operators the status of every machine
and allow the operators to guide the machines and the software, is doing the guiding for them obviously and that network is built using
zero mq currently and they actually cern went through a review of different things they could
use and they were using corba before which was it's old but it was it was a well-known quantity
and they rebuilt their their their systems every every five or ten years
and they went through and looked at zero mq and said well this is it's the best software but it
has above all the best community and we like that and we appreciate that and you know community
translates into you know lower cost also you have more access to experts and so on. And it turns out that there's actually a whole bunch of these ring,
there's also an x-ray factory in Grenoble called the EICRF,
and they're using 0MQ in a similar kind of control system called Tango.
And Tango is now built by a consortium of different x-ray synchrotrons, they're called, around Europe and around the world now.
And they're all using 0MQ as their messaging system underneath that.
So it's become the basis for systems like this.
These are large systems.
These are big, big applications.
And the scale from that weekend project I talked about for the NFL.
Sorry, this is maybe confidential, but it's on my blog anyway so i guess that's not confidential too late i guess
yeah too late so that weekend project up to projects which are you know hundreds of people
working on them um whenever there is you know bunches of computers to get talking together
and some smart person says,
aha, we could use CRMQ for that.
Then there's a win.
There's a win for that person.
He gets, you know, career.
He gets, you know, competence and career and solves important problems.
And his organization wins money.
Let's talk about an unusual topic,
but a topic everybody has to deal with, which is bugs.
And I was just thinking about, as you described,
ways that CRMQ is built or used in the wild, so to speak,
how those like you and the rest of the community behind zero MQ
must feel whenever they have to reproduce or test case a bug,
for example, like to deal with something.
And then I'm on your solve a problem, which is your,
essentially your support page, not the contractual support or
the commercial support, but bug fixes. And the first one says, make your reproducible test case
if you possibly can. And so I'm just thinking in a scenario where you have that ring and several
machines and light beams and smashing, how do you even deal with that? So what is bug fixing like for the community
dealing with this?
So first of all, when people are using 0MQ, if they're smart enough to choose it
and use it successfully, then they're smart enough to build those large systems out in
ways which shake out all the bugs early on. And they're obviously not spending a weekend
wiring up the large Hadron Collider
and then trying to see what happens, right?
Right.
So they have simulations and test benches,
and everything can be reproduced in isolation.
And actually, ZeroMQ lends itself to that really, really well.
So if you look at the examples that I write,
you'll see that they're always, in the guide,
they're always built out from simple to more complex.
And you can take every aspect and test it out and take every next aspect and test it out.
So it turns out to be quite simple to isolate behavior in most cases.
It's rare that you have really bizarre complex bugs caused by a bunch of different circumstances.
That's the first thing. That's always been the case. But what's more interesting is that in the last years,
we've really seen a drop-off, a dramatic drop-off
in the number of issues and bugs in our code.
I think that's deliberate, but that was kind of a consequence
of this process, this magical process.
The C4 process.
Yeah, yeah, yeah.
And it's actually quite obvious.
I mean, bugs come from code
which hasn't been properly tested,
which has not been properly hammered by people,
and which has got speculation in there.
Code which is inaccurate.
And when you hit it with actual reality,
it performs wrongly and things crash or go weird.
And the trick is to break down your development process into very small steps where each step can be tested.
And I don't mean formally tested by scripts.
I mean tested against someone's actual problem and validate it before you go further.
So we used to have in Xerium Q, like I said,
these waves of raw code coming in.
Here are 5,000 lines of new code.
Here is this whole bunch of new changes.
And the bugs came from that,
and they came from the time it took to weed out all of the inaccuracies in that code.
Now we don't have 5,000 lines of new code.
We have maybe 200 patches.
Each of them are five or 10 lines.
And in some cases, they're longer,
but they tend to be quite small.
And they're testable in each case.
The patch is modifying some behavior
which can be tested by a test case.
So you've got a before and after for every change.
It's very hard to make that buggy.
It's very hard to introduce strange behavior
when your changes are small and they are testable.
And it's like the ants going out.
And this is why I get back to intelligence.
The ants going out and foraging for food
and they make a small step and it's in the wrong direction.
They can correct very cheaply.
But if you take a bucket full of ants and throw them into a field they're all completely lost
yeah or else the buggy code is a bucket full of ants and what we build are these very very
efficient little lines of communication where the ants have identified some interesting problem and
they go for that and the more they work the more accurate they are. Quite literally bugs in this case, too. I mean, using the ANSYS model there.
So we have people who basically,
they build on the master version of GitHub,
different projects. That's what they're using.
And they go into production with that
when it's properly tagged, right?
And, you know, match that to our strategy
of accepting pull requests onto master,
and it gets quite freaky.
Yeah, those two worlds collide.
Big time.
Well, they only appear to.
It turns out they don't really collide at all.
It's not as big of a problem as you think it would be.
Well, the thing is that if you accept pull requests onto master,
then when there's a problem, someone will fix it really quickly. That'll get requests onto master, then when there's a problem,
someone will fix it really quickly.
That'll get merged onto master, right?
So in fact, what you're doing is you're correcting master
much faster than you would otherwise.
I assume the people who are building
like large financial systems or monitors and whatnot
probably are not on master too often.
Probably not.
Let's give some encouragement to those.
I mean, what you just said there
with the major drop off of bugs
is certainly encouraging for those
who have heard of ZeromQ,
are into or want to get into this space more so.
And we always like to give a good shout out
to a getting started.
So I know that you have a,
I get the software page on zeromq.org that certainly tells you how to get it, but what is the, what is your favorite
getting started guide that you can recommend to the listeners? What's the best way to get started?
I would say that it depends on your language. And what you do is you have to look for the
language binding that you like best of all, and then use that. Because the actual raw zero MQ is C++ wrapped in C.
The API is kind of weird still,
and it's not the best place to start.
And even the documentation I wrote is kind of,
it starts there and then it very quickly switches
to using a higher level C API.
So the best place to start is if you're, you know,
if you're a Python developer, you take PyZMQ. If you're a Python developer, you take PyZMQ.
If you're a Java developer, you take the Jira MQ.
If you're working on.NET, you take NetMQ and so on.
If you're a C developer, take CZMQ.
And start with those and start with those examples and start using that.
And then only go deeper as you need to.
Only go deeper if you have a reason to. And you'll find examples
for whatever language you're using in the guide.
The guide examples are in every possible language.
But each of these bindings has a community also,
and they'll be able to help you get started.
But in terms of learning Xerium Q,
there's one book, and that's the one I wrote,
which is called The Guide,
and that's The Guide to XeromQ, and that explains it. That's where you start if you're actually
going to go through the actual process of learning this as an experience.
So that's at zguide.zeromq.org. So if you're listening, that's a—
That's right.
We'll put that in the show notes, of course, but I just want to mention that
here. So it's the bullet points in the front of that page.
It says explains how to use your own queue,
covers basic, intermediate, and advanced use with 60-plus diagrams
and 750 examples in 28 languages.
So you've obviously got, I guess, most of your languages taken care of
because it seems like you've got, let me see if my notes are correct here,
40-plus languages that you're covering in terms of bindings,
available online in PDF format.
So is it like your other books where you can read it online
but buy it if you want?
Yeah, so it's available online.
That's where it started.
It's a wiki.
Actually, it's one very, very, very long page.
People prefer that, it turned out.
I had, first of all, written chapters, and people are like, no, no, I want one page, very, very long page. People preferred that, it turned out. I had, first of all, written chapters,
and people were like, no, no, I want one page.
I can print it off, I can go back and bookmark it, whatever.
And it's also available as an O'Reilly book called Zero MQ,
and there's also PDFs you can download.
I mean, O'Reilly were actually very happy with us
having it as available online
at the same time as they made the printed version.
The printed book is really nice.
I mean, it's quite a fat book.
It's a large book, to be honest,
but it looks good on the bookshelf.
Is it as thick as the JavaScript book?
The JavaScript Bible.
I don't have the JavaScript Bible.
That's awesome.
I guess one more topic here on XeromQ is,
you know, we talked about community,
we talked about the technology, not just getting started, but what areas of the community can people fill in?
We always ask the question of how can the community help? It's essentially the basic
question, but where are needs in 0MQ and how can people step up to fill those needs?
That's a really good question.
I think it's very difficult to come in and contribute to a project which has already
got so much mass.
It's hard to know where to start.
So certainly the bindings, again, the bindings are the kind of, these are the these are what is the front end, the front end deliverables to different language communities.
And they always need help.
Like I said, the Node.js binding I was working on,
every binding needs help in, you know, being updated in documentation
and guides for that and so on.
And if you're using a particular language in XeroMQ
and you want to help other people to use it,
then that's a good place to start.
I think also what's, I don't know if it's missing or it's just not necessary, but
meetups, you know, finding other people who are using 0MQ or working on distributed programming
in general and meeting up regularly and discussing what you're doing and how you're doing it.
There's the big problem of confidentiality.
Most businesses don't like talking about their infrastructure projects.
But apart from that, I think that meetups are a really interesting thing to do.
And it's great to meet people who have the same kind of problems as you.
Yeah.
That's interesting that you say that too, because you're so community oriented and you
have such an influence around building community that that's a need.
It seems like it's out of place to have that need.
Yeah, it seems out of place.
So I'm not really sure if it's absent because we've not worked on it or because nobody really needs it.
Yeah. But I'm involved in other meetup groups, and it can be a fun social experience
simply to have people who have the same kind of problem domains as you,
working on the same kinds of stuff.
And XerumQ may be a good excuse for that.
And if I was using it and not really confident enough
to contribute code to it, then that's probably where I would start.
So if we're clear here, I'll switch subject a little bit.
I have a question for you, Peter,
that I'm kind of waiting to ask here.
So one of the things that you mentioned
in your protocol first,
speaking to someone who's dying,
is to focus on, you know, the good things,
to focus on life and camaraderie and history
and talk about old adventures and, you know, discuss the good things to focus on life and, and, uh, camaraderie and history and talk about old
adventures and, um, you know, discuss the good things of life. And so you and I just met, so we
can't really go back and talk about that. You know, the good old days when we're doing anything
together, cause we haven't, but through your work with zero MQ and your writing and your,
your teaching. And like we said earlier, you've, you've been going to conferences all these years
through your company. You've traveled the world and you're teaching, coaching, so on and so forth.
Surely you've built up some good adventures, some good stories. And so I'd love for you to just
share something with us. Just whatever comes to the top of your mind, a story of yours,
whether it's software related or non, I know you have lots of,
you know,
non-software interests,
piano guns,
even though you,
you don't seem to own a gun,
which I think is kind of interesting,
even though you're a certified pistol instructor,
drums,
other things.
So,
you know,
if you have a good story for us,
I just love to hear it.
Oh gosh,
where do I start?
I don't own a gun because I don't own a gun because in Belgium it's, it's very complex to own guns. Um, you have to have
permission from the chief of police and you can only own, I think one or two in there. It's,
it's, it's a real hassle. Um, and I actually was working in Texas for a year for a company,
Samsung, and there's not much to do in Dallas, Texas. You can drive and you
can go shopping and you can go to nice places to meet people and lots of places to hang
out and so on.
A lot of food.
A lot of food, a lot of beer.
In Texas, it's almost illegal to not own a gun.
Yeah, it's a requirement.
It's the other thing.
So a driver's license lets you own a gun.
And I was doing this workshop with a bunch of people from Samsung.
It was our first workshop there.
And we were trying this kind of prototype distributed system.
And it was what eventually became Zyre, I guess.
And they'd been trying to do this for some time,
and they kept failing, trying different ways to do it.
So they had five or six of these really clever guys.
There was one from Kazakhstan. There was a bunch of Americans. different ways to do it so they had five or six of these really clever guys all you know there was
one from kazakhstan there was a bunch of americans there was one from i don't know it was a very
diverse mix of people brought in for this and after three or four days we had like we had built
really amazing stuff and it was all working and little little video games working across
four mobile phones and everybody was very impressed and then just very randomly i was like there's got to be a gun range somewhere and it turned out that the whole the whole class
were gun fanatics shooting fanatics in one way or another and there was maybe one or two who
never shot but who were into it and willing to try it so we found a range went there and shot
and i was like okay i love i love dallas it was it was cheap it was
easy they were friendly the guns were great i was i was able to shoot so over that year i must have
i must have gone shooting a hundred times and i would shoot i would just shoot you know two or
three hundred rounds of ammunition and i got really really good at it it's just a matter of practice and um and then i
decided at a certain point this was a kind of an insane project i mean we all we all went kind of
batshit crazy in this project for different reasons it was incredibly political and it was
we've been trying to do stuff that everybody wanted to to do and so when we had got some
success they were trying to kill us and take it over from us both people from inside the company and outside the company and it was just endless
endless politics and so my job was to navigate that politics and just kind of beat off all the
bullies and keep the keep the team together and alive and fed um so at some point i'm like okay
i'm gonna get my my pistol instructor badge.
So I look up on there.
I become an NRA member.
Really?
Yeah, because that's the NRA, the people who do the pistol safety,
the gun safety in America, which is a really good job.
I mean, I don't like the politics.
The politics is bullshit.
At least as far as I'm concerned. But their work on gun safety is really, really excellent.
And so I went down to waxahatchee texas
and there's this amazing little gun school there uh i think it's called the u.s uh i don't know
something the u.s something weapons something something although we don't say weapon in the
nra we never say weapon we say gun gun i see i was in the military and it was it was against
the rules to call it a gun or things like
that you had to call it weapon it was a weapon yeah it was a weapon where's your weapon soldier
but in the nra in the training it's like you don't call this weapon and everybody in the class with
me they were all they're all ex-military all infantry um you know these these big strapping
men with tattoos on their forearms like their forearms the size of i don't know and all aiming
to get their instructor certificate so they could go off into security
or whatever, or become an instructor.
And it was this
stupid, scrawny Belgian trying to figure
out how all these things worked. And I knew nothing
about pistols in detail.
And then I came to the practical exam.
I mean, going out and shooting.
And of course, if you've been in
the army, you're an excellent shot with your
right hand. Your dominant eye and dominant hand, as you've been in the army, you're an excellent shot with your right hand,
your dominant eye and dominant hand, as you know, right?
But if you have to switch to your left hand and one eye, then you're the wrong eye and
the wrong hand, then you're kind of lost, I guess.
That was the case here.
And so as this test went further and further, they're just completely useless.
And I'd been training both eyes, both hands, So I was the best, the sharpshooter.
And they were all just so angry with me.
I was like, yeah, Belgium for the win.
So I had this.
And then my family came to visit.
My kids came.
I took my daughter to shooting and she was seven.
And my parents were so pissed at me.
They're like, you can't take your kids shooting.
What is this?
Are you crazy? I'm like, well, she can shoot take your kids shooting. What is this? Are you crazy?
I'm like, well, she can shoot.
That's good.
She'll never forget how to shoot.
She's not afraid of guns anymore.
And she knows how to point a gun in the right direction.
And if ever a gun falls on the floor beside her, she'll pick it up and she'll know exactly what to do with it.
So I'm personally neither gun nor anti-gun.
I've never, I went shooting once at a friend's behest.
I got a range, and I had a good time,
but it just didn't really trip my particular trigger.
Oh, trigger.
Nice one.
What is it about it that's so fun, that's so interesting?
Is it stress relief?
Is it the challenge?
Is it the noise?
What is it about shooting a gun that you enjoy?
Well, it's the pure machismo. I mean, for a nerd to be able to go off and shoot things a gun that that you enjoy well it's it's
the pure machismo i mean for a nerd to be able to go off and shoot things and i'm just joking it's
not that at all it's it's it's technically it's difficult and so when you're when you're uh
stressed or if you have a lot of things think about and it's the same reason i enjoy drumming
actually it's the same thing with drumming then to be able to shoot properly you your mind has to has to switch off a certain level completely
switch off and there's like even even your your reactions when you're shooting like you you your
body knows the noise is coming so you're going to jerk that pistol up you have to switch that off as
well so it's meditation and if you get in the zone and you you're not tired and you have
you know your arms aren't tired and you're shooting then it's just absolutely magical how
well you can shoot and then you lost it then you lose it again and so it's kind of aiming for that
zone and it's a it's a form of meditation and it's it's it's fun i like that um there's also kind of
the i guess for a european to be able to go shoot,
it's kind of illicit pleasures for us.
We don't have that really.
Maybe.
I went shooting in Vilnius in Lithuania,
where it's even more random than Texas, I have to say.
But it was great fun as well.
It takes a lot of breathing.
Breath control is like the number one step for shooting properly and you
can't breathe well if you're fussy or thinking about other things and right fidgety or frantic
or whatever that's why you know like motion-based shooting is really tough like uh the show 24
jack bauer that's not real yeah i know what I mean? That never happens. You're never that accurate.
Yeah, shooting in yourself teaches you.
It makes it harder to suspend your disbelief when watching movies and TV shows
because it's so outlandish
that they can do anything near what they're doing.
That's cool.
Peter, I would have never expected you to come on this show
and tell us a story about guns.
You just blew my mind.
Especially if you're here in Texas, that's where I'm at.
It's Houston, Texas.
And so Dallas is like, you know, four and a half hours away from here.
And like I said earlier, it's not true.
It's not, this isn't a fact, but it's basically illegal to drive without a gun.
I mean, it's just so common here in Texas.
We even have a saying that has a cannon in front of it.
And it says says come and take
it because we're so relentless with our guns we have to keep them it's it's a I don't so much have
this belief myself because I'm not a true Texan I'm this is a total tangent but I'm from Pennsylvania
originally so Pittsburgh area is my steel city Pittsburgh area that's my you know home stomping
ground so I I'm a transplant to Texas but i don't i kind of feel that some
similarities between pittsburgh yeah i was in pittsburgh last year and i went shooting in
pittsburgh and it was fantastic very similar yes i mean we have similar i mean it's it's the u.s
but not all states there that share the same things but come and take it is a thing to text
and say and it's it's like it's like that's my kismo right there it is come and take it is a thing the Texans say, and it's like, whoa. That's machismo right there.
Yeah, it is.
Come and take it.
Come and take it.
That's machismo.
I wanted to make a conference called Guns and Code.
I had the domain name, you know, Guns and Code,
and I was going to do this probably in Oregon, you know,
rent a cabin and have, you know, have like a bunch of friends up there
and just rent a bunch of guns and ammunition and, you know, randomly a cabin and have, you know, have like a bunch of friends up there and just rent a bunch of,
a bunch of guns and ammunition and,
you know,
randomly called and randomly shoot.
Could be fun.
I mean,
I would,
I would attend.
Yeah.
Yeah.
I got told it was a bit too politically sensitive.
Dangerous.
Yeah.
I was recently invited to a bachelor party and it was bring your guns.
We're going to be shooting and there'll be food there
yeah like that was just that was the invite i was like well i don't have a gun but i'll i guess i'll
go and just kind of sit back from the main group a little bit and keep my safety but we are quite
quite a bit upstream on that let's ask you this peter so you've kind of you've wound your career
down by choice of course because you're ill, and so you don't have any,
you know, career work ahead of you aside from your writing, which I assume you're going to
continue to do until you can't or until you don't want to anymore. Um, so you've accomplished a lot
and you've traveled a lot and all that written books, given talks, so on and so forth. But what
mountains are out there or what goals or problems are unsolved
that if you had 25 years of a career ahead of you and not if you weren't looking back and you
were looking forward um what was something that you that you didn't get to solve or a mountain
that you haven't climbed yet i guess the thing i was working on which i had hoped to be able to
push forward was you know the internet of shit
I mean internet of things sorry and I have this I have this this kind of this vision that
the it's not even a vision it's just I'm just predicting what will happen the number of
smart devices number of computers out there will keep doubling every 18 months
and it will keep doubling every 18 months and every 18 months again, doubling and doubling.
And that's just the way it's going to go until the world is so full of computers that nobody
wants more. And that's a long way off. And these computers have to talk to each other and they have
to be organized into networks. And there's this whole fight going on right now about who controls
these computers, who controls their data, who has the right to update them and so on. And so we're working on
this this year earlier. I had to stop. I was really getting too sick to work on that, but it was
about building networks of little computers. We were using OpenWRT routers, the Glee 150 AR,
you know, it's $20 and it has this, it's absolutely wonderful.
It comes in a small plastic box and it runs on the battery.
Fantastic little device, completely programmable.
I don't know how many megabytes it has, but enough to run our software on properly.
And so my idea was, look, you have like a house full of these things controlling, you know, they all have IO, they have GPIO, you can control stuff.
Controlling lights, controlling, you know, temperature control, you know, sensing, GPIO, you can control stuff. Controlling lights, controlling temperature, sensing rainfall,
whatever you want, and they cluster together.
You don't have to configure them individually because there's too many of them.
You just plug them in the wall and possibly connect to a Wi-Fi and that's it.
And then you don't program them.
Instead, you send code to them.
As I say, you don't program them by updating the firmware,
you know, whatever is involved right now
with this horrendous process of sending your compiled code
to the thing.
No, no, that's finished.
That shouldn't be happening.
Instead, these little devices should be picking up code
from the network like a browser does
and running it and then
throwing it away and so that's how i wanted my kids to be programming the light bulbs is you know
write a bit of code send it to the light bulb switch on become green switch off at 10 30. i
don't know um and if there are 100 light bulbs they'll all respond the same way and if there's
two they'll respond the same way and that notion of the internet of things becoming the web of little devices speaking some language probably javascript
it'll be javascript which is kind of sad but that's that's how it's going to be i guess
i wanted to make that happen and then i wanted to sell aha make money how do you make money from
open source so i wanted to sell security on top of that and certificates
so that you could get properly signed code
that's been authenticated.
And this is a lovely area.
It goes into things like mesh networking and so on.
Devices can be also Wi-Fi routers.
They can talk to each other.
You can build whole structures like that.
We just talked about mesh networking actually
in
204J.
What number was that we just released the episode?
We've got to go to our homepage and find out.
203, Jewelbots.
They had built, it's an interesting little project.
I guess I shouldn't say little because it's quite big.
But it's essentially programmable bracelets for kids,
primarily girls right now,
but getting kids into coding and programming those things,
and they built their own little mesh network through them
so that if you walk up to another friend
and they've got the bracelet on
and they're your best friend or whatever,
then they change a color,
they have some sort of tactic pulse or something like that,
some unique interaction that happens based on proximity in this mesh network,
which is kind of interesting.
Yeah.
And that's totally up your iOS or IoT, whichever you want to say.
Yeah.
It's that proximity and that something that we lost in the internet,
you know, the concept of proximity.
Yeah.
And I think it's been possibly the biggest hurdle
to people understanding what it really is.
We used to have proximity by, we used to talk
and then we had phone calls,
but it was still conceptually a wire.
Even this, we have a wire that's connecting us somehow.
But you send an email to some address
which represents some person who may pick it up
at some point in time.
It gets kind of weird.
And I think that there's something to be won back
by recreating proximity as your connector,
even if it's virtual.
And mesh networking does that, I mean, up to a point.
If you have very large meshes, of course,
you're back to the internet. That's interesting that have you documented any of this stuff is this
uh i think of like um this isn't a one-to-one but similar to to like elon musk which must be some
sort of flattering remark to you because he's got unique big ideas that he's only got so much time
to do so much you know tesla spacex and you know whatever else i think this is like some interesting
ideas and you've obviously got some wisdom
and some bloody knuckles from being in the trenches
for so long and doing what you've done.
Have you put this down anywhere
that somebody can kind of pass the baton of the idea
or the possible idea?
I write a lot, but it's all over the place.
I mean, I have lots of, if you look at my GitHub profile,
you'll see lots of projects I've worked on.
And there's like desire that implements this concept of proximity.
CZMQ implements this concept of simplicity.
So you have different projects that do different things.
And then I have my blog posts.
And it's one of the problems is if you're writing lots of stuff
in lots of places that you have
fragmentation of anything, which is a grand concept, which brings me to my latest book
actually, which we're going to talk about, which was somebody asked, and I have this
question happening a bit too often.
Okay, how do I build communities online?
And I have these different pieces of text I've written on that here and there left and right
and so eventually I put them together and published it as a book social architecture
which is it's quite a slim I actually got 20 copies from create space I ordered them and it's
a nice little slim book but it's like this is my best practice for this particular problem
it's all in there.
But that's extra work.
I mean, somebody will have to go through my stuff and figure out what I was thinking about.
This is a late article from you,
May 17th, Building Online Communities.
And this is sort of a, I guess, in your own words,
this is an early draft from your book,
Social Architecture.
So how close is this to that copy
that's about I guess about a third
or half of the actual book in the end
I wrote the
article first and then I went back
the next day and re-edited it and added in more
stuff and published it properly with a cover and so on
so this one
we had some we had pulled
out here for talking
through everything like we've talked already through Fair Authority I guess to a degree So this one, we had some we had pulled out here for talking through.
We've talked already through Fair Authority, I guess, to a degree,
with 0MQ and C4, but something that we've talked about probably maybe a little, maybe almost too much.
I don't know if you can actually say that,
but we've talked about it quite a bit in the last 30 episodes.
It's come up at least several times,
which is funding or sane funding, as you put it,
in the toolbox.
And maybe let's talk about that one piece there
from the table of contents.
I know it's quite a bit of stuff going on in there,
but what are your thoughts on the sane funding?
I know it's a short chapter for you too.
Yeah, so I think it's a problem
which has solved itself in some respects.
So it used to be that to build a community,
you had to have infrastructure, which you had to pay for.
That used to be a fact of life.
You had to have servers, you had to have email servers,
you had to have stuff, and it cost money,
and the money had to come from somewhere.
That was the first problem.
Second problem was that you had to do a lot of the work yourself.
We hadn't yet figured out how to really bring in people smoothly and cheaply.
And until we had C4, we didn't know how to do that properly.
So the cost of infrastructure has gone away almost completely.
I mean, right now, like I said, the baton for XeriumQ, it's a domain name or two. the cost of infrastructure has gone away almost completely.
I mean, right now, like I said, the baton for XeriumQ,
it's a domain name or two, and that's it.
And the rest is hosted by GitHub.
We could be using GitLab.
We have access to free email lists in different places and so on.
So we don't have to pay for infrastructure anymore.
And the second thing is that when we come to the actual cost of work, that if the process focuses on people who are using what you have, solving
real problems and coming with their solutions, then it's self-financing. And that was a real
breakthrough. I mean, at that point, we realized that we didn't need to be funding the work upfront at all. We could leave it. And if people were not improving it, that's because they were not using it. by people working at office hours, being paid to improve
it because it's in their own interests.
And those improvements come back because of the license.
And our job is to make that flow work as smoothly as possible.
And that's a certain point, maybe three or four years ago, where I stopped having to
invest money.
And so this thing of sane funding kind of has gone away, at least in software, where you can work with pull requests and incremental progress.
Now, when I wrote that toolkit, originally the toolbox on social architecture, I was
working in political NGOs and non-governmental organizations fighting software patents in Europe.
And the problem we had there was that we had people who were working full-time on very,
very complex technical legal issues, preparing drafts of amendments to laws that could be
pushed through and voted on, analyzing what's going on.
There was a really really really large amount of
working done by unpaid volunteers who were being basically exploited by their own organization
and it was very very bad for them it was toxic and so my job my first job was to fix that problem
and find money to pay these people or stop them working. It had to be either you stop working on this,
someone else takes over, or we find you a salary.
And so that same funding concept really came from that.
If you have people who are necessarily working on something,
they can't be volunteers.
They must be paid.
Otherwise, they burn out, and it's a cult that you're building,
and we don't like cults.
Is that pointing to the last sentence in that section there?
Finally, watch out for individuals who take on too much risk
without adequate reward.
They could be vulnerable to burnout,
something you'll talk about in the next section.
So that's kind of leading into the market curve.
But I think that's a risk anybody,
any community has to watch out for,
is anybody who takes on too much without reward.
Because we are, we're humans.
We want some sort of affirmation back, even if if it's a tat or if it's even if it's not money you know it's some sort
of reward system that has to even in this case with ngos though you know obviously they're getting
not paid you know right by volunteering so that's that's a whole different version of burnout well
burnout i've looked at in from different angles over over many many years
and what i actually think now is that it's it's not so much to do with you know i work too hard
in this job or you know i gave too much for this thing what it really is is the dynamics of a
relationship um with with a bad actor that's true and the bad actor is using the situation to exploit people
and eventually they crack and if i look at my own periods of burnout in my career and i've had it
i've had several of these i look back then i realized that in fact it was that there was
somebody there who was being very very exploitative and manipulative and who was using people and
i was one of many who burnt out around that person.
Hence my interest in bad actors as a topic because they seem to be very, very influential.
There's some people I know too that are trapped in scenarios like that.
And it's so troubling to hear you say this because I think about those kinds of people.
And I've been there too, of course, but none of us are immune to it.
But you just think of like what a trap it is.
And it's almost like,
Jared, we talked about this once before,
where you love the deceptor in a sense,
and like you've been offended by somebody,
but you know, like I'm thinking of things like where it's domestic abuse or something like that,
where you don't go away.
What was it called, Jared?
Are you thinking of Stockholm Syndrome?
I think we talked about that at some point recently.
Stockholm Syndrome, yeah. Where, you you know you fall in love with that or taking them yeah taking away from you have this uh unusual love or compassion for you know your offender essentially and you just don't see the
truth and you kind of get stuck in this trap and so that that actually the last book i wrote before
the social architecture book was about psychopaths.
And because that is what we're talking about here, bad actors and the assholes in the world,
the people who can use emotions as a weapon, you know, they get really angry and then it's gone and it costs them nothing.
And they're never to blame.
They're always the victim.
They're innocent.
They will never, ever see themselves as a problem. They're always the innoc. They're innocent. They will never, ever see themselves as a problem.
They're always the innocents in any situation.
But they manage to hurt everybody around them.
And at a certain point, I'm like, okay, this shit needs to be documented.
I can't say I'm going to stop it because that's impossible.
But, you know, what I do really well is I take complex questions like this
and I break them down and I try to find decent, workable answers.
And I'm looking around the Internet for answers and how to deal with psychopaths.
And there aren't really any good answers.
And this is kind of weird.
I mean, it's like the only answer that people will agree on is run away.
Yeah.
Evade.
It's like, you know, evade, break contact, leave.
It's like, OK, that's fine.
That's easy to say but
don't you get it is that the basis of this relationship is this bond which you cannot break
it's like the first thing that the psychopath does to you if you're in a job or if you're in
a relationship is make sure you have no alternatives and make sure you're stuck
so who's this book for then this psychopath uh the psychopath code it's for everybody i mean
it's not written technically i don't use long words what i mean like what i mean by that is like uh is it for
those who are trapped since we kind of was on that subject is it for people who are in these
you know these these robbing relationships where they're being exploited or whatever and they're
near burnout is that the is that what it's to to sort an early warning system, so to speak, for those people? I suspect if you're actually in that situation
that you won't be aware until a certain point,
and then you start looking for help.
You start looking online, and you see this in forums,
on relationships, and on other situations.
People looking for help at a certain point,
something tweaks in their mind, like, okay, this is not normal.
This has got to be something else.
I am literally going to buy this book when we're done with this,
and I'm going to read this because the last line gets me.
This book delivers practical tools and techniques to survive the most
difficult people.
Jared's not one of them, but we've got difficult people around us often.
It's not Jared, so don't think that jared but uh man that's right and we all have that we all have
that problem we all have that if it's specific to you know our our industry then even better no no
no it's it's uh it's not it's not i i think that we're at least an open source i think we have
actually filtered out many such people but they there are a lot of them around.
I mean, it's like, what's the number one talent of a psychopath?
It's stealth.
It's hiding.
It's not to be caught, not to be seen, not to be seen.
And it's this cryptic thing.
So you look around and you see normality.
But it isn't really normal.
There's this population of people who are, I guess, they're innocents.
I mean, I'm not signing blame here.
And in fact, in my book, you'll see that there's a lovely backstory to psychopathy, which is really quite fun.
Like, well, I won't spoil the surprise for you, but it's a fun thing.
But this population of people, they cause a lot of pain.
Yeah.
It causes a lot of pain.
And I'm like, okay, I'm not a fan of pain.
I think pain is something you can reduce.
So this is a painkiller, right?
This is morphine.
It's not going to stop psychopathy,
but it's going to let you at least deal with it
in a way which reduces the pain.
You can actually be in the same situation
with the same difficult person,
but once you've understood what's going on
and once you've got tools for protecting your mind against this continuous attack,
then it stops. It stops. And I've done this. This is written from experience. It's not just me
fantasizing about stuff. It's written from experience. I had to make this book for myself.
And it works. It works. It doesn't solve the
problem of difficult people. They're still there, but their power goes away almost completely.
So you've written several books, obviously. A lot of them stem from either answering a question
that you've investigated or direct advice from your mind. And similar to the question that Jared
asked you earlier about mountains left unclimbed, let's kind of tee this question off to you in a similar vein,
but different. What advice can you give to the former self, to the developers out there,
to the community of open source, whomever you tend to speak to whenever you get into book mode
or you get into advice mode, what, what advice comes
to mind most for you to, to those who are trying to be in this rich community of either
creating software or pursuing creativity in software?
Um, I think camping out there simply because our audience is software developers, primarily
people who make things in and around software or even companies.
What's some good advice you can give that isn't in a book you haven't written yet?
Because you haven't written this.
What's some advice that comes from maybe a book you won't read or won't write?
Most things are bullshit in our industry.
That's my best advice.
I learned this as a young developer.
And, you know, don't trust the mass opinion seriously trust your own
intuition and look for you know areas where you can uh you can be special and where you you can
find yourself where you can learn where you have space to practice and just because things are
fashionable and everyone is doing it in fact that's the reason not to do that stuff do stuff
that's hard and it's different and that's that's freaky and of course
most of the time you'll be completely wrong and you'll be doing stuff that no one cares about
but you will learn and you will learn the hard way and you will learn well and you'll internalize
those lessons and to be like i consider myself a happy programmer. I program happily. I write code. It tends to work first time the way I want it to work.
And that took like 30 years to get there.
So you need to be patient as well.
You can't just learn something and then become a master in it.
You have to internalize so many little lessons.
And making mistakes, that's fun.
Getting it wrong is fun.
Just don't make huge mistakes.
Just don't bet your company or your house or your family
on something that you can't prove.
You take small steps, make small mistakes, you learn.
You make huge steps, you make mistakes, you die.
And then my last advice is to trust other people.
Not blindly, of course, as we've discussed,
but you're part of an ant's nest and
your power is other people it's not you it's how you can bring other people into your world and be
in their world and how you can share and build stuff together that's the real power of a human
being whether you're a programmer or a writer or a cook or a taxi driver, it's other people. And that's my inspirational thought for the day.
I love that.
Thank you.
Very good.
One last question for you here, Peter, before we close out.
First of all, this has been tons of fun.
I appreciate you giving us so much of your time.
If you were able to pick your own legacy, what you leave behind, what everybody remembers remembers about you what would your legacy be and
what would people say honestly i'm very content with what i've been able to do so far this was a
this was a decision i took maybe three or four years ago was to stop working for money
as a first goal and to go out into the world and become a writer i wasn't a
writer before that i hadn't written a few things but to actually publish books and do that as a as
a as a focused job to go to conferences and to meet people and to you know go out there and
i began writing my blog three years ago, four years ago.
Everything I've done since then is what I'm most proud of.
I think before that, it's mostly, there's good stuff,
but it's mostly lost to history.
And it deserves to be lost to history.
And I think that's all the lesson is, you know,
be yourself and be happy to be yourself and be proud of being yourself.
We all have unique stories to tell. And I think that's our power. I don't think that you have to be yourself and be proud of being yourself. We're all, we all have unique stories to tell.
And I think that's our power.
I don't think that you have to be an expert in someone else's story,
but we're experts in our own stories.
Well, Peter, it's, I think it's been, I don't know,
I want to say like maybe close to two hours, I bet.
It's definitely been roughly 50 minutes since the last break.
I didn't think we'd go this long, but I honestly couldn't hang up on you because you were just so much fun to have on this show
and i was telling jared in the back channel like i dig this guy like this guy's cool man and i
really i really enjoyed this conversation with you um i think you guys too you know this has
been this has been a real pleasure awesome and we we're glad to share you know even though we don't have a history with you we're glad to to pull out some
of your history and and what we do now yeah we yeah we definitely definitely do now that could
be our legacy peter hinchins digs us digs us yeah i dig you guys yes thank you peter um it's been a
pleasure to have you on the show uh any any final closing thoughts before we start tailing out
anything left unsaid that you want to make sure we bring up?
No, I just want to thank you very much for this.
I really do.
I feel very privileged to be on your podcast.
I have to go and listen to it because I never listened to ChangeLog before,
I have to admit.
And I'm a modest person, and this is very flattering,
and it makes me feel very happy.
So thank you very much.
Well, good.
It was an honor to have you on this show. And like I said, we almost didn't ask you
simply because I wasn't sure how sensitive the situation was, but we've been fans of yours for
a very long time. And now is the best time as ever to get you on. And in closing too,
I want to mention a couple of things. And you've mentioned And in closing too, I want to mention a couple of things and you've mentioned this in your blog,
the last couple of blog posts
around your death
and planning for it
and things like that,
that you've got a PayPal link
that we're going to share
in the show notes
and that kind people
have already sent you
over $10,000 in donations
and this money is going to go
to your children's lives
and making their lives easier when you're not there anymore.
And so we're going to put that link in the show notes,
but the link, just reading it out loud here,
is paypal.me slash hintgens.
That's H-I-N-T-J-E-N-S.
So that's Peter's last name.
So paypal.me slash Peter's last name, hintgens.
Go there.
And I think this is, maybe you can even step in here some, Peter, as well.
But ways that we can give back to you.
I mean, just this podcast alone, the advice you've given through your books,
the ways you've given back to the community.
C4, RabbitMQ, or not RabbitMQ.
I had that on the brain.
ZeromQ. There's a lot of MQs out there RabbitMQ, I had that on the brain, ZeromQ.
There's a lot of MQs out there,
but that's the one that came to mind
for when I was saying that,
so I'm sorry about that.
But, you know, all this thing,
all these things you've given back,
I just think like if somebody,
if anybody can give back anything,
whether it's buying a book,
whether it's donating to this PayPal fund for your kids,
you know, what are ways that you would like
to see some action in light of your obvious scenario? What what are ways that you would like to see some action
in light of your obvious scenario?
What are some ways that you want to ask the community to step in?
Buy the books.
I prefer you buy the books.
Actually, I get a good slice of the price on a paperback or an e-book.
Amazon is very good for that.
And when you buy the book, then I can give you something
which you can share with other people.
And that's the,
the nicest thing you can do.
Well,
you heard it.
You heard it here first by the books,
people buy the books,
people.
I think that's a good thing too,
because I'm,
I'm going to go and buy several of these.
There's a,
I don't know why I haven't bought it sooner.
I guess now getting from
the a bird's eye view on what the the psychopath code and your latest is about i mean to me there's
it it brings home for me there's a thread on reddit where i i did an ima on the psychopath code
and i was absolutely trashed well i was trashed it was i was I was I was I was murdered I was told by
people to kill myself it was what you don't have the qualifications for this
you know a psychologist how dare you write about this stuff and I'm like wow
wow wow and I got my bullshit sprayer and I was like I love reddit but
sometimes now I like that book I'm really actually very proud of that book
I think it works really, really well. charges that we try to help foster and lead is just encouragement, you know, like to fan
down the negativity.
And these are all things that you put in there, positivity being in that latest book, Social
Architecture.
You know, when you talk about people or things, don't talk about it in negative ways.
Just people don't have to be that way.
And I'm not on a soapbox here, so I'll get down.
But again, Peter, it's such a pleasure to have you on the show.
Thank you so much for all that you've done.
We definitely appreciate having you on the show today.
And with that, let's call the show done and say goodbye.
Adam, Jared, thank you so much.
Bye-bye, guys.
Thanks, Peter. Outro Music you you