The Changelog: Software Development, Open Source - How open source saved htop (Interview)

Episode Date: September 24, 2020

Today we welcome Hisham Muhammad into our Maintainer Spotlight. Hisham is the creator of htop - a well known cross-platform interactive process viewer. This conversation with Hisham covers the gamut of being an open source software maintainer. To set the stage, a new version of htop was announced, but not by Hisham -- it was a kind takeover of the project and needless to say Hisham was surprised, but ultimately relieved. Why? Well, that's what this episode it all about...

And on one hand, like your rational self thinks that, yeah, I've never gave, you know, an SLA for my open source users. Which is 100% true. Which is 100% true. And then on the other hand, you go like, yeah, but that's not how I would like to be treated myself. Yeah, exactly. But it doesn't scale.

You know, you're only one person. And if I was to give the kind of like prompt, in-depth, thoughtful response for every communication that comes my way on all of the projects that I'm involved with, at the ideal level, maybe that would take my entire weekend. Or that wouldn't even be enough hours in a day or something like that, right? If you're running a one- man show project like that or something. So you have to weigh those things. Right. So yeah, some things are just not scalable in that sense.

Being with her change log is provided by fastly learn more at facet.com. We move fast and fix things here at change law because of roll bar, check them out at robot.com and we move fast and fix things here at Changelog because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com slash changelog. Welcome back friends. This is the Changelog podcast featuring the hackers, the leaders, and the innovators in the world of software. I'm Adam Stachowiak, Editor-in-Chief here at ChangeLog. Today, we welcome Hisham Mohamed into our maintainer spotlight.

This is a special flavor of the ChangeLog where we focus on maintainers and their journey. And Hisham is the creator of Htop, a well-known cross-platform interactive process viewer. And today's conversation with Hisham covers the gamut of being an open source software maintainer. To set the stage, a new version of Htop was announced, but not by Hisham. It was a kind takeover of his project. And needless to say, Hisham was surprised, but ultimately relieved. Why? Well, that's what this episode is all about. Special thanks to Tidelift for partnering with us to co-produce this flavor of the change law thanks for coming on Maintainer Spotlight. We appreciate it.

Yeah, thanks for having me. And I guess I should start by saying thanks for HTOP. I've been using HTOP for years. And I even have a long-standing bash alias that says, if HTOP is on this machine, alias top to HTOP, because it aims to be a better top, and it certainly is in my opinion, so I'll use it whenever possible.

Oh, thanks. Yeah, it's appreciated. I have that alias as well. I think a lot of us do. So we're here to talk about not just HTOP. You have lots of things going on, Gobo Linux. You have other projects that you're interested in. Things still not announced quite yet, but coming soon. And H top just hit 3.0.

And this was a big release. Maybe not so much because of what was in the release. However, there are features in there. There are a lot of updates, a lot of bug fixes in there, but really how 3.0 came together because this was long time kind of unmaintained by you. Do you want to tell the story of your recent Htop history and what happened? Yeah, sure. I can go through that story. Well, yeah, Htop 3.0, like, after maintaining Htop for over 15 years, Htop 3.0 was a surprise to me. I literally woke up once Saturday morning and I had a Twitter notification

that someone like at tagged me in a news post about the release of H.3.0 which said H.3.0 released. The project has been taken over by a group of maintainers after prolonged inactivity from Hisham, from myself. So yeah, and well, I jumped out of the bed to check. This was a pleasant surprise. This was not like, oh no, someone hacked my machine.

No, yeah. So yeah, I jumped out of bed and I, okay, I need to, like, and I just straight ran into to write a reply on GitHub because there was an open GitHub issue that title is this project still maintained? I wrote a response there and I started the response with, I don't have it open here to get it literally, but I pretty much started it with, first of all, I'm very happy to see this because I'm sure that everyone who saw this news and who knew me as the maintainer would have like questions like what happened and so I ended up writing this long post on which I explained what

went on and yeah so in that post I say a lot of things and one of them was that my first reaction when I saw that tweet was, to be honest, one of relief. And so essentially, like at this moment, I'm like retiring from HW maintainership. And well, I saw all of the whole thread that was going in that GitHub issue about asking if the project was maintained. People were going like, I wonder if he's okay. Everything was super respectful and friendly. And someone said, well, his GitHub activity continues to be green. You know, the tiles with your activity. But I wasn't answering the emails and all of that. So yeah, what happened there was really that at one point,

HNOP has a long history. And as time went on, the project really started as a way for me to, like in a typical free open source software, like scratch your own itch, take something that you're annoyed by and try to fix it. So I started a like many years ago like and at one point i considered it pretty much like done like for my own use right and uh and then it was like maintenance that went on and i started to take longer and longer breaks from the project right and then sometimes it would take like six months and then i would look back at it, merge a couple of PRs, fix something here and there and make another release.

It was always like for a long time, it was kind of a low key maintenance thing for years on because if you have used H.Dub for a long time, you've noticed it pretty much looked the same for a long time and it has become right sort of like reliable dependable thing that's kind of like you know there and works the way i wanted to work so at one point i started looking at like you know like cp you know some other things like you don't expect the copy command to like change a lot overnight or something like that so yeah so it felt pretty much done to me and those breaks kept getting longer and longer and yeah eventually i at one point like i had a an email filter that just sent everything that mentioned htop to an email folder which i would like occasionally maybe like twice a year look at it and and do like batch maintenance

but as as you said like i'm like so involved in so many things and i and i think we're going to touch on the point of like burnout you know more deeply later because it's an interesting topic. But at one point, I realized that I hadn't looked at the project for a whole year. And I felt like a sort of sigh of relief, like, oh, that was refreshing. I wrote it down there in that post mentioning this. So, yeah, and then I realized, I started actually thinking about, like, handing over the project, but, like, I've never done that before for an open source project. Like, how do you like to find someone to pass it on?

And, like, if you find it to be, like, sort of a burden to you, like, do you want to instill that burden on others? And, like, you feel that mixed feelings about, like, your responsibility to the project. But then, like, along that mix of mix of emotions like months flew by again and i did nothing right and uh yeah and when i realized i think i guess it was like a year and a half and like yeah and then i'm just gonna like fast forward to that day when the when the announcement came so yeah so basically i told this whole story there and like to show like my side of what's going on what was happening and well

reaction has been like overall positive from all sides right the the new maintainers i got in touch with them and like i apologized to them for like having to have that one side decision of forking the project which i understand is like not the ideal way of doing things and i said like okay all is well when it ends well. And at the same time, I made myself available for any of the transition matters. My old website now redirects to the new one, which has a much cooler domain, by the way, htop.dev. Cool domain.

And that's what happened, basically. I'm very happy that it's an actual group of maintainers. It's a collective now like i think the project is in very good hands and uh yeah i'm very happy here from how it turned out to some people it might have been my reaction might have been a surprise but uh yeah but i think as i said like i think this is like free open source software working as intended you know and it has been like overall positive from all sides. Well, you got that email filter there that filters out H-top and coming back to it every once in a while is one thing because you need those natural breaks.

I mean, if you don't, I suppose, think something needs to be advanced or in your expression of it, you mentioned how it was sort of done to you, but you mentioned there's a couple other people that try to reach out to you about changes to it. And maybe that's sort of the feedback loop being broken, so to speak, of a maintainer or creator of a project not really hearing the community. And that's kind of what happened, right? Was like that you'd filter the emails and people are trying to reach out to you to sort of collaborate.

I suppose in this new GitHub fashion, new GitHub meaning like the last 10 years of GitHub or more of how it's become more and more collaborative, but they tried to reach out, and you just were in and out of the project as you saw fit. Because that's just how it worked for you. Is that right? Yeah, that's right. And I see different projects in very different ways.

In open source, like, everything always, like, falls under that same umbrella of, you know, based on the licenses and all of that. Yeah.

But the nature of, of projects is very different. Because for me, Htop was always like this, you know, kind of like one person hobby project that was never a developer team

and all, or any of that. And, I am part of other projects with our arm which are more collaborative and you feel that sense of you know uh responsibility and communicating and all of that but uh but for h top like back from the start there was like 2004 it was a very different world back then like you you're talking about like github in the last 10 years but this this was

this was started back in the days of sourcefororge. Those of us who are old enough to remember. And the dynamics were very different, right? It would have like a mailing list or something like that. And many of the licenses in free software, they come with the whole thing. This is offered as is with no warranty, you know, in capital letters. And again, like there, sometimes projects have this like implied notion that, you know, you're responsible to actually be available. And, you know, whenever someone reaches out to you or something like that, but in that sense, you know, that no warranty comes with that as well.

Right. Sometimes, sometimes people will just, you know, here here's like here's something i did here's the code well if it's useful for you right feel free to use it but you know i'm i'm not really on the hook for like you know answering the emails like in a timely fashion or or any of that of course i try to be like i i like it when i reach out to other free software maintainers and you know and they're responsive to me right and and, as I said, at the same time, I have another project where I'm the maintainer of the Lua language package manager, Lua Rocks, and that has an ecosystem around it of other developers who develop Lua modules and put it in the repository so that is very much a live project that needs

that tending to, that housekeeping. Whereas Htop was this kind of hobby weekend thing where sometimes I would hack on it for a while and sometimes not. And I treated those very differently. And I guess at one point I started that

email filtering thing as sort of an organizing thing. Like, oh, I can't look into this right now, but I don't want it to slip away in the inbox mess. I'm just set everything Htop aside so that next time I look at Htop in my next round of playing with the project, I will get to this. Well, it makes jumping back into things a little easier if you've got sort of a collective,

but you don't want to be distracted because it doesn't require daily maintenance or even monthly maintenance. It's more like in the meantime, when I want to look at it, this is where I go and it's easy. So you will procrastinate less and maybe there's a lot of psychological things you could do yourself to sort of be productive with what matters most today. This kind of reminds me, Jared, of something that Nadia said recently when we talked to her, which was like not really a one-to-one but more like if it's public it doesn't mean that it's participatory and so not so much that htop wasn't open source and available for others to participate but more like as you had said and Sean was like it's more like it's a one-person project doesn't need a lot of people involved in it it's not active like maybe little rocks might be so

you know just you weren't really sort of paying attention to the fact that others might want to share their opinions and interact and progress the project as natural open source kind of is today would you agree with that yeah and because in fact there were many releases like past releases of h stop it was which were made mostly based on me merging community contributions. And then I figured that, okay, now I'm going to do a round of maintenance on this in which I'm going to look at all of the PRS, which daytime sometimes, sometimes assessing the validity of a PR, like it actually takes longer than it takes to write it. Right. Because you want to, some people would just like send you like like a change okay i tried something out

and it kind of works for me and then i had to look at it like will it work for everybody right will this crash any other corner cases and and and things like that like insist what about the systems that i can't test myself like ever since ever since they stopped became portable and and wasn't a linux only thing right you You know, like that increased because, you know, someone might come out of the blue and send you like a huge patch with support for a new operating system, which, you know,

you have no way to test. This has happened. Like I've gotten like a huge patch which was like very intrusive with many like considerable changes for adding for IBM AIX support, right? Which like, I don't even... Did you merge that?

I don't even know what's the status of AIX is nowadays, right? It's kind of a legacy Unix by now, which like, and I looked at it like it was this huge patch. I have no way to test it and probably no way to maintain it myself. Like, okay, what if anything goes wrong and someone complains that something's not working out if this breaks something else so at that point you know it's like for that one patch i just let it slide and say like okay like

if folks like want to run it on ai x like i guess you guys can maintain your own fork with it you know and essentially do the maintenance of that like On the other hand, the Solaris one I did merge in because there's the whole open Solaris and that's like Illumos and the subsequent systems that do follow on that tradition and that I said, okay, that will have more of a living open source community around it and I might be able to ping more people to help me out on that one so yeah so so even that baseline maintenance work you know that you do in batches you know it's like sometimes it is significant it can be significant but yes of course there were lots of people who had reached out to me over time and like it's amazing getting those patches and at one point i joked around and said, yeah, the software is writing itself because all I'm doing is clicking the merge button.

There were H-top releases. Even as far as 10 years ago, which had not a lot of code from myself, which I like lines of code which I've written myself. It was mostly merging patches. For stable projects, that sort of becomes the norm over time. What I found interesting about this thread, it probably wasn't interesting to the people involved,

might have been frustrating to them, but interesting to me was how they were kind of becoming H-top archaeologists, like what's going on with this project? Because the initial is this maintained issue was open all the way back in March. Here we are in September. I think you replied a couple weeks ago. So the story began in March on this thread, ended at the end of August.

And in the meantime, like you said, you were doing other work. You have your Lua Rocks. You have GoboLinux. You have, I think, some of your work is also open source or public on GitHub. So like you said, the GitHub contributions were happening. Maybe you're posting to your blog. I don't know if you were tweeting or whatever. In the meantime, because you had that email filter set up, like you weren't seeing any of this activity. And it's fun to read, they're like, well, it looks like the guy's alive and doing well,

he's just not replying here. So you know, they're starting to figure out like, okay, what's going on here. And eventually, people stepped up, which like you said, this seems like very much an open source success story because you've had this desire to hand it off over the years. It's kind of done to you. It does what you want it to do. Maintenance is a burden, but one that you're willing to do every once in a while for a while. But if you could have had a reliable person or people maybe

five years ago maybe three years ago whenever it was you probably would have handed it off then and by a matter of circumstance here you just kind of ignored it you know filtered it into a place you weren't looking and people just popped up out of nowhere and said hey i can maintain this and that's it's somewhat rare it happens but it doesn't happen all that often. And there were a lot of people who just stepped up and took over. Pretty cool. Yeah, it's funny to think about it. Because yes, I was working at GitHub. I was tweeting.

It was just like life was going on as normal. The way you described it, I was thinking in the back of my head, it's so funny. Because it's kind of an observation of the state of technology and the way we communicate nowadays and all of that because like if it wasn't the 90s someone would have called me on the phone i guess about this you know and i think what happened specifically in my case like it also had a lot to do with the the burnout angle of things because one of the things that I mentioned in that message that I posted that Saturday morning was that at one point in time, I had a conversation where I was talking about burnout and open source maintenance, in which I said, well, if a project is like, and maintained

and like important enough, you know, then it will be eventually forked, you know, if people actually need, like if there's a concrete need for the fixes to get in and for things to move on, they will. And I wrote that and then after writing that, after posting it, I recalled what was the context in which I actually said that. And it was actually at one of the Google Summer of Code

Mentor Summit sessions for which I have been for a number of years a mentor at that event, mentoring students on open source for the Lurox project. And we had that session where it was all like a room full of open source maintainers and we were all talking about burnout and people were talking about like, you know, that feeling of responsibility that they had for their projects and, and how they felt that they could never take time off and, and all of that, you know, and, and, and how pressured they felt about that and how that was one of the things leading to, to burnout. And at that point, I mentioned, like I said,

I'm actually, I'm here for LOROX, but I also maintain H-Stop. And I kind of take regular breaks from that project, you know, and people like stared at me, you know, with, you know, wide eyes and go like, what, like you don't look at it like every single day. And like, you know, what if someone, you know, has something like, I go like, well, if something is critical,

I'll probably get to know it sooner or later. Which did happen at one point. There was this crazy bug hitting Apple Mac systems in which there was a kernel bug in Mac OS, which Htop, by trying to read the state of threads, it was consuming some resource from the kernel. And it turns out that opening Htop managed to crash the Mac OS kernel. So that was actually- Congratulations. Yeah. So like-

Quite a feat. Yeah. At first I looked at it and I was like, well, that's Apple's problem. They should fix their kernel. Right. Right. But that was like the most commented thread on the history of the project with like more than like, you know, I don't know, there were like 200 messages and something like that.

And I was like, okay, I'm going to put a mitigation for this, you know, until like Apple, you know, gets its acts together and uh yeah so so so so i did that like so so i kept that in mind i felt like okay if something gets critical to that point i will know about it like even between my breaks and uh i guess this htop3.0 was kind of a similar event but um but otherwise i said like you know like we we can't let ourselves, it can get super overwhelming. And I understand that as H-Top is important for me and for lots of people. As I said, the fact that it was forked meant that it was really important for someone, important enough for people to put in that effort.

And I feel super flattered. And I feel like it's one of like, you know, probably like one of the things like in my professional career, like as a software developer that I'm like most proud of, like having, you know, written this project and having see it become like an open source success in that sense. Like as important that software is, you know, like people's like mental health and all of that, you know,

it's more important than that, right? So like when I was in that session and I was talking to other open source developers, which, you know, they felt like super pressured, you know, and super unhappy with that situation and not knowing what to do. Like I told them, you know, what I told them that day was like, Hey, take breaks. And, you know, if, if you need and, you know, and if something happens, like things will, you know if you need and you know and if something happens like things will you know eventually sort out by themselves because you know the software it's out there it's in the open you know it's kind of like that's something that we can do you know in open source you know if if the

project was you know proprietary even if it had like a huge user-based community that would like and i decided to disappear there would be nothing that could be done. People would have to, I don't know, try to write it again from scratch or something like that, but then it would not have been an option. Open source does give you that option, so maintainers should keep that

in mind, that they should take breaks if they need. Right. It might also be in a sweet spot in scope. I didn't look at the lines of code and all that and how complex of a project HTOP is. Conceptually, at least, it seems like the kind of thing where if you're going to hop in and maintain,

if you have pull requests, you have bug fixes, you have minor things, is it the kind of project where you could grok it after a little while and kind of start to dink around? Or is it is it massively complex? Because sometimes, sometimes a piece of software has the value, like you said, if it's valuable, somebody will fork it, somebody will maintain it. Sometimes the people that are relying on it don't have the skills, and they're not going to have the time or the money to acquire the skills, they would love to maintain it if they could. But they

can't. In this case, it seems like there's some people who are willing to and able to maintain Htop. Yeah, that's a great question. And yes, that does factor in. And yeah, I think like, for Htop's case, well, everyone likes to believe that they write clean code, right? And I always felt that the code base was approachable, given that, you know, the number of PRs I got over the years, and that, you know, that people would add features and send in stuff. Right. So I was never like too worried on that regard. say, oh my god, this is super important, but the code base is like inexcludable, either because it's... Not because it's badly written, but because it's ultra complex and things like that.

In the case of Htop, I don't think it to be super complicated. And I actually, I was complimented by the new maintainers. I said, okay, thanks for maintaining it. In one of our out-of-band emails after the 3.0 release, they said, okay, thanks for having it. It's been fun to work on the code base. I was like, okay, glad you like it and enjoy it. It's always nice. As I said, it's a matter of how needed the maintenance is to people and what are the

stakes, right? Right. Because at the same time, when I look at the new maintainers, some of their emails were at redhat.com, at debian.org. They were coming from established organizations, established names in the open source world. That feels reassuring and it feels like, okay, like these folks will have like, you know, will have the time and the resources to keep maintaining it. You know, so that feels good.

Like that, as I said, like, as you said, even for complex projects, that reminded me of the OpenSSL situation after Heartbleed, you know, and people were like, okay, everyone's using this. And, you know, large companies, you know, at one point, like they stepped in.'s using this. And large companies, at one point, they stepped in. Stepped up. Yeah. So I guess it's a matter of that.

Well, I guess a shout out is due to a few folks. Nathan Scott is one of the people who stepped up and kickstarted the fork. There's an account called Faster IT out of Germany, another GitHub account who seems like they were involved in the takeover. Anybody else? The cool thing is because HTOP is so prevalent and been around for so long is that it's in all the repos, right? Like you can apt-get it, you can yum install it.

And I did see people from Debian hopping into that thread saying, yeah, if you guys want to maintain this, because one of the problems is if you hadn't showed up, taking over maintenance would have been much more difficult, I imagine, because it's not just the source code, it's the distribution. And I'm sure that you have some sort of interaction with the distributors, is that right?

Yeah, that's right. And let's give a shout out to Nathan Scott and also to Graham Ings and Daniel Lang, or Daniel Lange. I don't know where in the world he's from. So whether I should like pronounce that in an English or German way. And yes, I did.

I got in touch with maintainers of a few distros to confirm, yes, like moving on for the next releases, you know, get the upstream from here, like htop.dev is the new upstream. And I made the point, that very morning I made a point of redirecting the website to the new one so that the URL pointed to, so that would be a clear indication. And also now under my personal GitHub account the the repo is now archived and the readme points to the new one because we didn't transfer over the repo because the new one had

already been kick-started and already had new issues, new PRs and we didn't want to override any of the new activity. And at the same time I think it's good to give the new maintainers a clean slate where they can... And as I said, same again. If any of the old PRs is super important, they will be ported over. So they got in touch and left comments in the old PRs and telling people to consider reopening there in the new repo

if you still want this merge and all of that. I understand that it's important that I got involved. And I think it does help a lot. And as I said, if something like this happened, which I never actually foreseen this would happen, but if something of this kind would have happened, eventually I would expect to get involved. And yeah. details about the Tidelift subscription. It is a managed open source subscription backed by Project Maintainers. And as you may know, this series, Maintainer Spotlight, is produced in partnership with Tidelift because we love what they're doing and we believe in them. So if you're

building applications with open source, Tidelift helps you to ensure that you have components that just work, including comprehensive security updates, active maintenance, and accurate licensing, which is so helpful. Tidelift helps you to speed up application development, save money, and reduce risk when building apps using open source. And best of all, with the Tidelift subscription, you help open source maintainers that we feature on this show to get paid for their work. Learn more and get a demo at Tidelift.com.

Again, Tidelift.com. Again, Tidelift.com. So you seem to have a pretty healthy relationship with your open source projects. The ability to step away, I think, is a skill or maybe a personality trait, which you have. I'm curious on the other end, when you hand off something which you've maintained for very long, which you created, right? Like this was your brainchild. This was your baby for all these years. And you've given it to a bunch of strangers. Does that concern you? Like what if they

destroy each top and it sucks after this or like is there any sort of those feelings of like maybe it's gonna not be maintained the way that i would want to or as well as i would do it any of that uh yeah that's an interesting question and probably something that you know a long time ago like a younger version of me might have worried about. But I think over time, you rethink things. Well, the older versions, they will always be there. If that's my open source legacy. With your name on the commit messages. Yeah, if people want to check out what was it like

when I was involved, it will always be there. Right, true. Yeah, but at the same time, I'm, like I'm happier to see it live on, right? In that sense, right? Because I think that's the ultimate success of an open source project, you know? Outlives its original maintainers and, you know, and carries on.

So in that sense, I think like the way I look at it nowadays, like I think that's like the ultimate success, at it nowadays, like, I think that's, like, the ultimate success, right, to be able to reach that point. And, like, I have other projects which I would love to retire from, but which I don't know if there are, if there would be new maintainers. Actually, I'm, like, I'm actively thinking about this now. Like, should I, you know, now that I've seen this happen once, should I be proactive on this?

It can happen. Yeah, it can happen. Should I be proactive and say, okay, we're looking for new maintainers? What do you got? What projects are you looking for new maintainers? We happen to have a podcast with software developers who listen to it.

Are you looking for maintainers? Put up a sign. As of now like lower rocks itself like it's a project that i'm i'm sort of overwhelmed by nowadays because i've been like lead maintaining it for a long long time that project started in 2006 i guess it's a long time is there a team around you for that one? Yeah, so around Lower Rocks, there are other maintainers, but it's very much on a whenever I'm available basis as well. They're not like regular maintainers.

And I keep Lower Rocks going as part of my work as well. So this is something that I am involved with because we use it at work. But I would love to see someone step up and take the lead in the project. So I could still be involved, you know, but not in a leading position. And I'm trying to figure out how to do it, you know,

because sometimes like in projects, you know, inertia is very strong. Like people will not, you know, people will not say like, hey, I would like to lead, you know, if there is someone else leading, you know, and all those kinds of things. And regarding my personal relationship with my projects, I've realized that, well, GoboLinux, which is this super experimental alternative Linux distro that I started back in college with my college friends,

which I use it to this day you know my my sister like if you look at my laptop it's this like super hacky running like all sorts of experimental stuff like including like the distro like this it's super hack including this distro that we maintain around on like with a very small group of people and that that that kind of project is one that we also maintain like in this like every couple of years you know we get together and we cook up a new version like it's it's not for general use like at all when we look at it like i had this bunch of projects which i started like in the early 2000s and which i have been maintaining for a long time and at one point in time i was like

yeah mostly of what i do now is maintenance. And I haven't started a new project in a long time. And I started to feel this itch of starting new projects and having more freedom to be creative again. And as life goes by, there's work and you have a lot less time to do that. And the few time that I had for open source projects I wanted to spend in more creative ways so so that that had to do with me like stepping away from from htop for so long because that's the time that I had to like to try new things and all that well if you want to find a new maintainer for lua rocks what you do is you create a folder in your email and you send all your lua rocks stuff to that then you ignore it for six months and then

boom, someone's going to pop up. There you go. That's the formula. We found it. As funny as that sounds, that comes to what I was saying earlier about the different nature of open source projects, right? Because Luarox has this live server, Luarox.org

with people uploading things, downloading things. It's a different beast. Exactly. I maintain the CLI tool. The server component has a different maintainer. Shout out to Leif, who maintains the lorox.org server. So that project has a community around it and it has that different you know maintenance style as a necessity right so for that one i could never do that you know and and nothing of that sort ever

happened with low rocks because you know things fall things go into my inbox and i do check it out and and have a different maintenance style for for that project you know depending on the nature of what you do like some different maintenance styles come with it. So for Lurox it was kind of different and I assume it will be somewhat more difficult. But then again, there's bus factor and all of that. If I disappeared for real tomorrow, I'm sure eventually things would be sorted out because Lurox is used quite a lot by the Lua community.

Right. So now I think it's now seeing what happened with H-Dop, I think like, OK, it's possible to do this. You know, I just have to figure out the way to, you know, to adapt to a new role. Yeah. You mentioned relief in particular, though, right? And your response, even at the start of the show, you mentioned that your response, you know, while you got this notification of 3.0 for HTOP, you know, via Twitter. So this not normal way to be told about changes to your project. You weren't pissed off. You weren't upset. You were not relieved. Happy.

Yeah. Yes. you were not relieved happy yeah yes well maybe juxtaposition is like with lua rocks if you got a twitter notification tomorrow there was a new lead maintainer of the cli tool for lua rocks would you be relieved or would you be upset no i would be happy for that as i said like i i would consider that to be a um to be a mark of success for the project. And in the end, we have to keep in mind, in open source, people throw around that word community a lot as this kind of random concept.

But in the end, it's really people. It's really about people. And recently in the Loo community, people it's really about people and you know recently in the lua community like i i had a like a someone who i consider a friend even though i never met him personally uh who passed away like suddenly and you know and and we had to some of my projects i had handed over to him like in the past like had made the coverage analysis tool for Lua, like LuaCov. And he had been maintaining it for a few years after I had started the project and maintained it myself

for a few years. And so, whoa, there was this whole commotion in the community with his passing. And we had to scramble, OK, what happens with his projects now? I sort of inherited back the ones that I had passed to him and took it on myself as a personal matter to figure out what to do with those projects moving on. And then some of the other projects the community got

together to get necessary patches merged and new releases out and all of that, right? And it's a pretty strong wake-up call, you know, like that's like to you about like, you know, like what does really matter here, right? It's really a community of people, right? Who care about each other and who care about like, who got to know each other through the work that they do, right?

But in the end, right, when you build a community of people, you have to keep that in mind. So to me, because of the pandemic, Lua Workshop was supposed to be held in Germany this year, it was canceled. And I remember a couple of years ago, Lua Workshop was in Lithuania and a bunch of us flew over to Lithuania from Brazil, from France us flew over to Lithuania, like from Brazil, from France, from the U S from the Netherlands, you know, from all over. And like, at one point I said like, yeah, like I come to this workshop, you know, because of you folks, you know, like to meet

the people who like, who have become my friends over this year, over the years, you know, like not so much about like Lua itself or, you know, the, the the talks. We go there and we give talks to each other. We present the talks and there's always new people who are getting into the community and getting to know folks. So yeah, this is super important. And in the end, it's a huge part of why I do open source nowadays, why I keep doing it. You seem to have a deep connection to other people. And going back to the relief, the relief was from what, though?

Yeah, because you kind of have that sense of lingering guilt when you're not responsive. Right? Right. not responsive. If you take the time to you would have to balance okay, now I'm taking this break for myself, but okay, but what if I was on the other side and I had this bug that

was hitting me every day. At one point I get so annoyed that I decide to hack on it and fix it. Then I fix it, then I open a PR and I send it over and say, okay, please get into the next version. And I go months without an answer.

I've been on that side of a relationship before with a project, a couple of times. Not where it's so, I can live off of my own fork or i can just you know depend on that and i'm okay like it's not like a huge my life is ruined or my business is ruined scenario but like you said you take the time you're usually somewhat angry because it's not working the way you need it to and you have there's ramp up time right like figure out how do i contribute to this thing? Where, you know, where do I even put this in?

Am I going to break it? There's all the questions like, how do I talk nicely? And you do all the work and it can take hours and submit a patch. And then it just never goes anywhere. You know, I've been on that side and maybe you have as well as a member of the open source world, you kind of go on both sides. Sometimes you're the maintainer, sometimes you're the, the submitter. And I wouldn't say like hurts, but it definitely weighs, you know, it sucks when that happens. And so I could see from your perspective, the guilt of saying like, knowing like, well, I got all these PRs out there, I got these open

issues and I can't look at them because I'm living my life and all the reasons that we've stated, but they're still there. And those are real people on the other end of those PRs. Yeah, it comes from acknowledging that it's real people, like, as you said, like on both sides. And then yeah, and on one hand, like you, like your, your, your rational self thinks that, yeah, I've never gave, you know, an SLA for my open source users. Which is 100% true. Which is 100% true. And then on the other hand, you go like, yeah,

but that's not how I would like to be treated myself. Yeah, exactly. But it doesn't scale. You're only one person. And if I was to give the kind of prompt, in-depth, thoughtful response for every communication that comes my way you know on all of the projects that i'm involved with you know like in the ideal at the ideal level maybe that would take like my entire weekend or you know that i would or that wouldn't even be like enough

hours in a day or something like that right if if you're running like a one-man show project like that or something so you you have to weigh those things. So yeah, some things are just not scalable in that sense. But you feel guilty for not responding or not showing up. But then you're sort of trapped, so to speak. Because there is no playbook that I'm aware of, at least. And if there is one, let's put it in the show notes or highlight it more. Of how to be a one-person-ish maintainerer maybe take contributions as you mentioned have you been doing before like

the code writing writing itself so to speak as you'd mentioned but how do you care about something like this be the creator and maintainer of it for so long but then be able to hand it off in a way that lets it have life doesn't ruin it as jared mentioned before. And there's just no, it's kind of icky, right? Like there is no easy button for it. So you just never do it, right? You just sort of procrastinate. You kick the can down the road a little bit further because there's no. Yeah, because you don't know how.

Yeah, it's not easy to do. There's no right or wrong way. There's no playbook for saying, hey, this is how you begin to hand off a project. No, yeah, 100% agree. And even if you're not wanting to hand it off or you just want it to keep going, but add in a maintenance style that makes it sustainable. One thing to keep in mind is that there are a lot more open source projects

which are one-person shows like this. Because when we think of open source, you can like one person shows like this and like then because when we think of open source you can think of like the huge projects i don't know like gcc the linux kernel and all of that and when people go like how do i contribute to open source yeah you start you know with a tiny patch to you know to some project you know and people usually point folks to like huge projects and all that but you know if you look at you know like i don't know like the github archive you know and things like that like there are so many so But, you know, if you look at, you know, like, I don't know, like the GitHub archive, you know, and things like that, like there are so many, so many projects,

you know, like which are, you know, most of the repos are like one person only. And even some of like many of the ones that are used a lot, like I see a lot of that in the Lua community because of the Lua Rocks repo, right? It's a lot of modules. Or if you look at like something like,

you know, a lot of times, many times look at like something like, you know, a lot of times, many times bigger, like NPM, you know, like how many of those packages are maintained by a single person. And yeah, and there is no playbook on how to deal with that. And I think what we're doing here in this, you know, in this conversation helps, right? Because even, you know, like, because I think what part of it is for people on both sides of the table to understand the social contract that's established between maintainers and users

and especially for things like developer tools and things like that, we have this additional angle in which the user is also a maintainer of something else. But in many cases, for many types of software, that's not the case. And so, yeah, I think step one is an understanding of what goes on on both sides.

And in sharing experiences, you know, like, okay, this is what happened for this project. These were the circumstances. These are the things that went well. These were the things that went not so well, right? Like, I do not consider like this to be like a, you know, like a fully like smooth transition, right? As you said, like there was this whole like thread that went on from March until now, right? Which, you know, that could have been like easily avoided, you know, like, and I totally take the responsibility for that. We're happy that the outcome was positive.

But yeah, it's important to share those experiences within the open source community at large. That's why I'm happy to be here talking to you guys about this because we need to learn how things happen and slowly try to improve them. Well, that's the reason for this series, this Maintainer Spotlight series, is that we can share the stories of maintainers,

the wins, the losses, the accidental successes, and all the things because there's so much that goes into it. And because open source is such a broad thing, dental successes and all the things because there's so much that goes into it and because open source is such a broad thing you know we're starting to have some formalization via nadia eggball to work with working in public about how to like even talk about the different types of projects right because it's so diverse that you can't just say well this is how you hand off a project because there is no this is how you hand off a project right yeah it's so diverse that you can't just say well this is how you hand off a project because there is no this is how you hand off a project right yeah it's different every time yeah and sometimes it goes

great and sometimes it goes terrible you know so yeah i think sharing creating community around maintainers providing more conversations that say oh i listened to that uh what hisham did that actually helped me out in my project because X, Y, or Z. So hopefully we have some of that effect, but you'd think over time we could maybe aggregate some sort of a knowledge base, like a starting point, like, hey, I'm burnt out.

I need to hand this off. Where do I start? Maybe we can start to create those kind of resources, but it's difficult and lacking at the moment. Yeah, fully agree. And it sounds like, yeah, it sounds like a great idea. Like, I think one good first step is recognizing like the different types of projects, the different types of communities, different types of, you know, maintenance styles.

Like, as I said, you know, I've been like engaging in like multiple different maintenance styles at the same time, right? You know, like for H-Stop, I was taking this hobbyist, once-in-a-while approach. For Lurox, Lurox is also this old project which is also in maintenance mode. We're not making any earth-shattering changes on it at this time, but it's important enough for a community

that we want to keep it working. And if anything breaks, it gets immediately noticed. And the whole server-side aspect has this very much online effect. It has to be up. So that requires a different maintenance style. At the same time, I do open source at work. So when you have an open source project that's backed by a company you know that has a completely different maintenance style right so

um and sometimes you have projects that it's like you know you do it once and you know and throw it away and then say like okay like you know i don't expect to give like any kind of maintenance to it but i'm gonna just put it out there there in case it's useful for someone. In which case, you might even mark it on the readme. Say, yeah, it's just here, but don't use it in production or whatever. I've seen lots of projects like that. All of these types exist. Maybe we should start talking more about this, giving those styles, I don't know, names,

so that this is a project of type X that this is a project of type X, this is a project of type Y, and then people would immediately understand and not have a stigma of the kind of interactions that they will or will not get from it. I'm all for having those types of conversations. Nadia Ekbal has come up with a taxonomy that has four types of projects

and it's based on the relationship with user growth to contributor growth and so your project age top would be what she calls a stadium because you have one or very few maintainers and the user growth grows dramatically but the contributor growth stays relatively small then there's a project like lua rocks which i'm not sure if that would be a club or what was the other two types i don't know i'm going right off the top of my head yeah um there's clubs there's stadiums foundations play into it what's that one called anyways i need to go read the book again do my studying but that particular taxonomy i'm sure our listeners listeners are yelling into their AirPods, like, yeah, I get it right. It's derivative, right? It's based on observing a project and saying, based on the project's

relationship with its users and its contributors, it's X, Y, or Z. But what I think would be also useful, and I mentioned this to her she seemed to have some excitement around it is what if you could explicitly state what you want your project to be like not this is what it is because this is how it worked out but like give my give this project a name this is a a tool for me or you know this is a a club like come join my club or this is uh you know you we have to come up with the names, but you could actually like, people are using their readme to set some expectations, right? Like pull requests, welcome, or hey, you can look at my code, but it's not really participatory.

Like people say those kinds of things. But I agree with you that if we started getting, giving names to these styles of projects, not what they end up being, but what they want to be up front, that could be like your step, add that to the list of things I do when I open source something, is I give it a name of what style it is, because then you come to their source code and you know immediately what to expect

from that particular project. Let me close the loop here, because I happen to have a PDF of her book right here. So I just went ahead and researched it. So to quote the book, it says, focusing on the relationship between contributors and users, we can think of projects

in terms of their contributor growth and user growth. And this gives us four production models, federations, clubs, toys, and stadiums. Toys. Yeah. That's all I can remember. Yeah, so the federation is yeah it's pretty close yeah you're pretty close so it's a start yeah i think it's important too because

naming things something we say on brain science and a lot of this is why i find this interesting is there are a lot of psychological tie-overs to what we talk about on brain science where we really focus on like what we know about the brain to kind of transform our lives whether it's habits or working in teams or dealing with burnout or stepping away to get unstuck, all these fun things or the power of our choices in our lives. We got to think about naming things to tame things, and that's what happens here because Nadia is able to give a text on me. And an example, naming them helps us all as a collection, a community, to tame the idea of what does it mean to be a federation or a club or a toy or a stadium as an example of open source. And so names really give us all something to anchor to. And that's kind of a great thing.

But it takes somebody to take that first step. And in this case, it was Nadia to give us those models and those names to anchor to. I fully agree with that and I think it makes a lot of sense. Usually what happens is that people realize after the fact what the project end up turning into. Because often when you start, whenever they always start they're always a one-person show anyway. And sometimes going like, okay, I'm starting this

and I want this to be a federation can sound like a super lofty goal and you don't know if anyone's going to pay attention to a project or care about it or if you're going to be able to build a community. So people might be reluctant on that. There are times like when Kubernetes came out of Google, for example. Yeah, but then it's coming from Google. I know, but again, that would be an opportunity for them to state what this project is, right?

Like current status, essentially. We aspire to be a stadium, or we aspire to be a federation, but right now we're a toy. Currently a toy, but it might become a... Yeah, you make a good point. Not all projects start from a single person. They might start from a company and they might start with a huge backing from day one.

So it's part of that. It ties into not only on the relationship between users and maintainers but also funding, things like that. Like, Lorocts has had like this varied history over time. Like, the project was actually started back when I was doing my master's. And we had this, you know, industry academy, like joint research project for the development of Lua because I was doing my master's over at the University PUC in Rio, which was where

the Lua language was created. And so for that, I had funding to kickstart the project. So I was not doing it as a hobby, as a part-time thing. So that was my day job for two years as I was doing that. After the research project was gone, there was no more funding, for a few years I kept maintaining it on a volunteer, kind of hobbyist basis, but just because of my attachment to the project and all that, and we were talking about that sense of responsibility, like okay now this is out in the open, people are relying on it, and every now

and then I would actually do a consulting gig that was coming from corporate users of it. And now I'm working at a company which uses it. So I'm, again, effectively being paid to maintain it. And so my relationship, the funding story for it has changed over time. My relationship with the project has changed over time. These things are very dynamic, which can be a complicating factor. But yes, having names for things.

I remember in the early days of open source, when people were talking about the cathedral and the bazaar, to compare different development styles. And they would call it, oh, open source is like this. And proprietary software is like this, you know, like the cathedral. And even that is not precise because, for example, the Lua language itself, my advisor who was like my master's and PhD advisor, who was the creator of the language, Roberto Yeruzalimski,

like they have at Pucreio a team of like three professors who single-handedly develop and maintain the Lua language, like the reference implementation of it, like the language and its implementation. And it's open source, it's MIT licensed, and they've been doing this since the mid-90s. They do it to this day. They take no contributions or patches. All of the code is written by those three people. They have a mailing list. They take feedback. They help run the Lua workshops. They're super friendly to the community and all of that.

But they say, no, no thanks, because we're academics. The result of our work goes in papers. And they're subject to that whole academic lifestyle don't know, like academic lifestyle, which is like very different from open source community. But and then they go like, OK, like, you know, I don't know if it's because, you know, they want to state in the papers that like this is our original work, you know, because of like academic restrictions or whatnot. Or if it's just the style that they personally prefer. Right. Because they time and again, they say like, we'll take ideas like suggestions and then we'll implement it ourselves but thanks, we don't want any contributions so it's 100% a cathedral

but it has been open source from day one so it's whatever works for the maintainers What that makes me think of though is this clear communication and the need for a good feedback loop. So as a maintainer, as a contributor, as a user, that feedback loop and the clarity required. So those three, this example you give for – as an example, their community is probably more cool with, hey, they don't take contributions because they've been pretty clear with the expectation as a user or as a would-be or a desire-to-be contributor. So not so much to call you out, but more like if that feedback loop was there for your users to say, hey, I take frequent sabbaticals.

Don't be offended. The SLA wasn't there, of course, but more so less of pointing that out, but more so to say like what my takeaway is a couple of things. It's clear communication and that feedback loop that we all desire because it helps us, I guess, to reduce our anxiety or our concerns or whatever. And then the other side is like the kindness that's required because, as you had mentioned, it's software, sure, but it's really a community of people. And what I see that played out with this HTOP 3.0 and all that played out was those folks were very kind to you. They didn't think that Hisham was ignoring them and he's a bad person. In many ways, they were regarded and concerned. I hope it seems like he's okay because of these reasons, as Jared had said earlier.

And I think that's what we all need to lead with is this nature of kindness rather than what could have played out, which is this person's terrible because they just had this repo or this project that's very useful. They're not responsive and they could have said a bunch of nasty things about you, but they led with kindness.

And I think that's my takeaway is the need for clear communication and lead with kindness. Yeah, for sure. This was a super positive experience in which I feel like I've come out a better person out of it communication and lead with kindness yeah for sure you know like that's like this was like a super positive experience in which like i feel like i've come out a better person out of it like from what i've learned like as you said like in terms of communication and all that yeah and also to participate in this episode in which like the kindness really went both ways in the sense as i

said like you know they didn't think i was a terrible person and at the same time you know i didn't think they like oh my god they stole my project or anything like that. So in that sense, the kindness really went both ways. And I think it becomes a multiplier. I've never had so many heart emojis in a GitHub comment on my life. And I probably don't expect to have it again. Well, on that that front let's give

a shout out to a fontanette the person who opened this particular issue back in March is this project maintained because a lot of times that very first nudge that first comment that first issue sets a tone right and they start off by saying I want to emphasize right at the outset that I don't believe project maintainers have any kind of obligation at the outset that I don't believe project maintainers have any kind of obligation to the community to continue working on a project. I'm very grateful for HTOP and all the work that has been put into this thus far.

And then they go on to say, is this project maintained? But that's a great way to set a precedent of kindness and respect and empathy before you go ahead and say, is anybody maintaining this? Instead of just saying, WTF, why aren't you maintaining this? set a precedent of kindness and respect and empathy before you go ahead and say, is anybody maintaining this? Instead of just saying like, WTF, why aren't you maintaining this?

You know, they really did set a tone. And that tone remained the entire, I mean, I read most of the thread. Almost everybody was positive and they were adding to the conversation. Like I said, it became this archaeology, where's Hisham, you know? And then once you hopped in, it was kind of a love fest so really uh we see a lot of drama in github issues this is like the opposite case this was pretty cool yeah yeah i felt like it and yeah it's a great point that you made like how it is like setting the tone was something that kind of like guided the conversation and also like that and that first

person said that you know like oh i i don't recall the exact words i don't have it open like in front of me but it says like i'm not volunteering to become the maintainer or something like that right because yeah because something you mentioned earlier right that you know not everybody you know has like skills time inclination whatever totally right but that was a huge contribution to the project you know that one message right some some right sometimes you know sometimes it's all you need to do and this is what i see a lot That was a huge contribution to the project. That one message. Sometimes it's all you need to do. This is what I see a lot.

As I said, sometimes there's a lot of drama in GitHub or on the internet at large, but generally my relationship with open source, with free software, those communities, has been positive in that sense. I feel this thread is actually representative of my life with free and open source software, really. If I fly out for a conference, I don't know whether it was like Fizzle International Free Software Forum that happened. Here in my neighborhood, like in Porto Alegre, southern Brazil,

every year where people from all over Latin America would fly here. Or in Brussels at FOSDEM, where the European free software folks get together. It's always generally positive like that. We see a lot of, sometimes the drama gets over-amplified on the internet. But when it comes to people meeting face-to-face, and I know that meeting face-to-face is not something available to everybody, right? So we have to be mindful of that.

We have to, you know, whenever we get the chance to meet people face-to-face or even, you know, even like over, you know, a video call as I'm talking to you folks. So that's like a way of having like a more personal connection. We have to carry that, you know, to our like online our online written communications and interactions. I feel that I kind of have sort of a pet peeve with the whole code review culture and what it is. Because I noticed even the style of writing that people use when they're doing code reviews is kind of different when they're sending a message on Slack or something. And when I noticed that i decided to become like more like consciously informal you know and and you go like like hey looks like something like this is missing blah blah emoji whatever like but not in a forced way but like

okay i'm gonna i'm gonna talk to this person the way i would talk to them in person rather than you know this seems inappropriate you know know, at line 375. Right. Like, if you're like the schoolmaster, you know, like grading someone's PR, like, you know, this is acceptable. No, this is unacceptable, right? So it's kind of like I think it boils down to this as well, you know, how we communicate, you know, how we interact with people.

It makes a difference, right? So as you said, like the way the conversation starts is the way the conversation flows right and to that yeah for sure you know that that was super important and i'm so grateful that that's that's the case here you know i think i mentioned it before my takeaway is like you know leave a kindness in these scenarios and because of that you know we can point back to this you know not as a hostile takeover but as a kind takeover, you know, of this project. And we can all look back at this as an open source community and say, this was an example

of things working out well and people being treated with respect and with empathy and having compassion. And rather than the drama that can sometimes ensue, because I think we forget that we are literally talking to other humans. It seems so logical, I suppose illogical potentially to think like that, but other humans be authentic, talk to people with kindness and act as if they're there on the other line, reading your words rather than just simply, I don't know, like a machine. I don't know. It's easier said than done sometimes, but that's my hope, this conversation

and this example gives me hope that we have this opportunity in open source but Hesham, thanks so much for one, coming on the show and being open and honest and sharing your side of the story and all that it is and even what this project meant to you

Lil Rocks as well and your contribution to open source and being a person that can point back to doing things in a kind way and caring about actual people and showing up and whatnot. But thank you so much, Sean, for this time you've shared with us. We appreciate it. Thank you, folks, for this space

where we can talk about this. I know this has a huge audience and I know this has like a huge audience and, and, you know, I hope this, this conversation, you know, becomes like, you know, useful for, uh, for more people who are like perhaps in similar situations or really that we can, you know, spread this kindness, I guess, like, you know, and make sure that this is the norm, you know, and free and open source software and software in general, life in general, you know, like it goes beyond software right yep that's right thanks a lot yeah thank you fantastic work hanging in for the whole show that means you are a super fan that means you're a prime candidate for being a changelog plus plus member and if you haven't heard about this yet this is

the membership we launched so that our awesome audience can directly support us, get closer to the metal and make the assets appear on our podcasts. Check it out at changelog.com slash plus plus. And if that isn't cool enough, the irony of this episode being 413 and the error code 413 standing for payload too large. I don't know. That's super cool. Once again, huge thanks to Tyler for being an awesome partner in this podcast series. Also huge thanks to Fastly, Linode, and Rollbar for having our back. And of course, Brake Master Cylinder 2 for making all those awesome beats for us. That's it for this week.

We'll see you next week. Thank you. Bye.