CoRecursive: Coding Stories - Story: Leaving Debian

Episode Date: November 2, 2021

Today’s story is an insider view of Debian. One of the oldest Linux distributions and probably one of the longest-running volunteer-based open-source projects. Joey Hess is my guest, and he dedicated significant parts of his adult life to working on Debian. He’s going to share what that was like. The good and the bad, and it’s almost all good. It’s a story about open source software, but it’s also about community and teamwork Links: Episode Page Support The Show Subscribe To The Podcast Join The Newsletter

Did you ever wonder how open source software got started? When I first got a computer, I used to buy shareware games from a local computer store. The discs were in this spinning carousel thing, like you'd see books in at the airport. And they all had this picture of a friendly cartoon wizard on them, and then the title of the game. They were $5 for a disc, and the first one I got was Duke Nukem. That was shareware. But my friend Eric, his computer had a CD-ROM, and you could fit a lot

more games on a CD-ROM. And eventually he got a CD for something called Slackware, which was a Linux operating system. This was a complete replacement for the operating system of your computer. And this was in the days when Microsoft was charging serious money for Windows 95. And this was totally free. Not only was it free, but according to Eric's uncle, it was so much more powerful than Windows. This would be my introduction to Linux. Well, it would have been, but Eric never managed to get it installed, and I didn't even have a CD-ROM on my computer, so I just actually stayed with my shareware games. Oh yeah, this is Co-Recursive, and I'm Adam Gordon-Bell. Each episode, a guest shares

the story of a piece of software being built. And yeah, I always wondered who built these Linux distributions. Did you know that 96% of the top million servers run Linux? And many of those are using a Debian-based distribution. Debian is one of the oldest Linux distributions, and it's maintained and supported totally by a group of volunteers. I mean, this is what blew my mind as a kid when I heard about the idea of open source, and it still kind of does. That there are these people working hard on a voluntary basis to build software that powers important parts of the world.

It's almost like if important parts of our electrical grid were built by a volunteer group of electrical engineers, you know, who were just really passionate about what they did. Like, that would be amazing, right? I mean, I know open source isn't quite like that, but it kind of is. And it is an amazing thing. So my guest today is this guy. Yeah. Hi, I'm Joey Hess. I'm kind of a crowdfunded free software developer.

And I can't see you now. Like often I have a video call, but like describe your surroundings. Well, I'm in my office right here at home. I live in a kind of a cabin. You can say a cabin and you get the right idea in the woods here in Tennessee, just surrounded by trees. Joey dedicated a large part of his adult life to working on Debian, and he's going to give us a peek into that world. He at times worked for years without pay to push Debian forward. And then something changed, and he decided to leave it all behind. So that's today's story, an insider view of Debian, the good and the bad, and it's almost all good. It's a story that's about open source software, but it's also about community and about teamwork. Joey's introduction to Linux started a similar way to my story with trying to install Slackware. He was at

Cornell, staying in residence, and someone lent him 15 floppy disks. I got it installed, and I'm sure I ran into problems. I probably destroyed, you know, five or ten systems before I figured out what not to do. I remember at some point, I moved on to Red Hat, and I had to, like, email, you know, somebody at Red Hat just to figure out which floppy driver I needed. And that was actually the first time I talked to somebody

in like the computer community who was actually a developer. That developer was Donnie Barnes, Red Hat employee number two. Red Hat would go on to become a giant company and eventually be acquired by IBM. But at that point, they were small. And Donnie told Joey what driver he needed to get things working, which is great customer service. It's amazing customer service, especially since I wasn't paying for it.

Yeah, exactly. Once you had Linux running on your machine, what did you do with it? Well, mostly what I did with Linux was what I had been doing with Windows, which was building websites and CGI scripts. This was back in like 95, 94, 95. And I had done that on Windows and quickly found that Windows was not suited for it back then. So I just moved all that stuff over to Linux

and kept on with that path. Yeah, and I started writing command line utilities and things too, but a lot of it was web-focused. What websites did you build? you know, command line utilities and things too, but a lot of it was web focused. What websites did you build? Oh, a bunch of things that have been lost to history, more or less. You know, there was this thing called the personalized Shakespearean insult list, which was just floating around the net.

It was a list of, you know, several lists of words. You pick one from each and you get your insult. So I made a web version of that. You would type in your name and it would say thy pox ridden, whatever, cock swing. I don't know. I don't remember them, which was very briefly a big deal because there wasn't really much going on on the web. So having a website that insulted you was novel. So were these running like on a server in your dorm room? It was running on my desktop in my dorm room 24-7. You know, people would have struggled to put up websites, right?

But you were hooked right up. You had a good connection to the internet. You had Linux running. You were set. Yeah, I had pretty much everything that you could possibly need to put up a website in 1995. That's awesome. What did the internet look like in 1995?

That's pre my time of internet access. Yeah, I mean, you know, it was very text-based with occasional images and pretty sparse content when you get right down to it. You know, you really wouldn't spend a lot of, you would browse around the web. Browsing, of course, was, you know, much different back then

because you would, you know, jump around a lot from site to site up somewhere random but but you know it wasn't there wasn't enough really you couldn't have a conversation with someone on the web for example you know that was all limited to other mediums i don't know what year this was but there was like the netscape site of the day or something. Do you remember this? Oh, yeah. Yeah, there'd be a site of the day because like, wow, there's a new website. Yeah.

My Shakespearean insult thing was probably on, I don't know if it was the Netscape one, but whatever came before that. And your little computer handled that, like being the site of the day. You know, it was probably a few thousand people who looked at it. So, but I mean, I was, you know, the web quickly grew. This was 94.

It had been around for a few years and it was well on its inflection point upward at that point. So that was a brief period in time. It's wild that Joey's dorm room computer and some small script he wrote could handle being the site of the day for the whole internet. The internet was smaller then, but it was growing quickly. But anyways, one of the problems of switching your computer to Linux in the 90s was there was very few computer games. Joey thought he could address that by finding free games and packaging them up to run on Red Hat Linux. But while Red Hat Linux was free, Red Hat was a commercial business and they had no interest

in accepting game contributions or any contributions from outsiders, really. I was free, Red Hat was a commercial business, and they had no interest in accepting game contributions, or any contributions from outsiders, really. I was like, well, that seems kind of silly. And then I was like, oh, here's this Debian, which somebody showed me in a users group meeting. Well, I'm going to switch over to Debian. That way I can make packages, and they'll land right in the distribution. Debian had a different model than Red Hat.

It was built in the open by volunteers who coordinated on email mailing lists and on internet relay chat. And if the licensing was right, anyone could add a package to Debian, which meant that now all the Debian users could easily install and run that software. So I pretty much became a Debian user to become a Debian developer. You know, I was, you know, 20 or whatever, 18, 19, something like that. I was, I was exploring all the Linux games, which there

weren't very many, but you know, there were, there were various Tetrises and stuff like that. So I was like, well, let's just package all these up that way. When somebody installs Linux, they can play Tetris. And then did you start like socializing with people in Debian? Were you on IRC? Did you make friends or? Yeah, I was, I was on the Debian mailing list. I assume I was on IRC, though I can't really remember it. And you know, it was all remote. And I, I don't know if making friends is the right word for what happens when you're on a technical mailing list with other people, but you're collaborating and you're getting to know them. Joey finishes school and gets a job in the Bay Area.

I was running a place in Berkeley and Bruce Perrins showed up at my door. Bruce Perrins was the leader of the Debian Project at the time. He also lived in Berkeley, up in the hills. And he was like, I'm going to take you on a tour of the Bay Area.

I had only ever, you know, been in Berkeley briefly. How did he know that you were in the area like did you i guess i must have came up on irc or something i i don't remember and then that was the first time you met him when he showed up at your door like rang your doorbell absolutely yeah and what did he show you i remember he drove like we were driving down what was the 580 or whatever i can't remember now i haven't been in the bay area in a while but anyway we were driving down he what was it, the 580 or whatever. I can't remember now. I haven't been in the Bay Area in a while. But anyway, we were driving down. He's like, well, this is the Stanford particle accelerator we just crossed.

And then, you know, we went down into San Jose. We probably went to Fry's or something. You know, it was basically just a Bay Area tour. We didn't really stop at a lot of, you know, institutions of computers in the Bay Area. But he did probably point out, here's this company and that company and that kind of thing. That was, you know, kind of the first,

yeah, the first person in Debian, probably the first real free software in-person interaction outside of maybe a user group or something. It was pretty neat. That's kind of amazing. I totally agree. I mean, all i was doing was

packaging up tetris for debian and here's the debian project leader like hey let me show you around you know again like kind of like the donnie barnes thing there's a openness there which is i think pretty incredible and did you feel like this is a place for me? Yeah, I mean, I certainly felt, you know, I got sucked into the Bay Area to a decent extent. Although, you know, I was spending most of my time sitting in front of a computer. So how much can you really be sucked into it? But, you know, there were Linux users group meetings in the Bay Area probably every other week or so. So I started to go to those and I met a lot of other people who were users and developers and people and companies

and all that stuff in the Bay Area that way. So Joey's going to these Linux meetups and meanwhile, the internet is growing. The dot-com boom is underway and all these crazy buy cat food on the web startups need web servers to run their websites on. And so Joey's Debian experience is pretty valuable, and he got recruited by a company called VA Linux. So VA wasn't a Linux distribution.

They were actually a hardware vendor. So their thing was they were selling rack-mount computers with Linux on it and then would support it for people. They were kind of trying to pull in a collection of people from different free software communities that were all involved in Linux. So they had X developers and they had, you know, they had me as a Debian developer and they had this, you know, just a whole bunch of collection of talent. So they could then turn around and say to somebody, well, look, you need this. We've probably got somebody who knows about that or knows somebody who can help you with that or that kind of thing. They would occasionally complain to me that it was too hard to install Debian.

And I'd be like, yeah, it's too hard to install Debian. I'm thinking about it. But mostly my day job was just to go into work, work on whatever I was working on in Debian that day, and then go home and probably do the same thing. So it seems like you had a soft landing in the Bay Area. Like you got there, you met like the head of the Debian project, and then you got a job where you just got to do whatever you wanted. Yeah, I don't think you can really have a better experience than that. I'm pretty sure that I wouldn't have even went to work for a different company if it hadn't been something like that.

I was pretty skeptical about just going to some IPO and selling cat food or whatever was the big deal at the time. Oh, yeah, because it set the stage, right? Yeah, this is like 1998, 1999, and the dot-com boom is ramping up. And that's why, of course, VA was selling all these rack-mount computers, because everybody needed lots of them. So were they growing at some astronomical rate as you were there? Yeah, I think I was probably employee number, I don't know, 30 or something. And then there were probably, you know, within a year there were 200 or 400, or I don't know how many employees, plus not even counting all the fulfillment. This was just, you know, in the office. So it ramped up pretty quick. It had an IPO after I'd been there

for maybe a year and a half, which I think is still on record as the biggest IPO in history, or maybe the second biggest as far as the opening price or the opening increase or something. Do you have stocks? I had a little bit, but they did have a generous friends and family program, which was nice. I was able to get my dad some stock at least. So you had unvested stock options? Yeah, sure. How much were they worth?

I don't know, a couple million dollars at some point or other, I'm sure. I really tried not to pay any attention to it because, you know, it was clear that I was never going to see that money. Was it clear, though? Like, did you know at the time? Well, I mean, I could I can look at the trajectory of a line on a graph and make a guess as well as the next guy. I don't think I would feel it like this. Like, OK, I have several million dollars, once this vests, and yes, the stock market is starting to go down, but maybe it'll be back by the time I vest.

Maybe it'll be even higher. That's what I would have told myself. But Joey, he has a different focus. All that was kind of, in a way, a big distraction from what I was really interested in. Which was like packaging or what? Well, when I first joined Debbie and debian you know i did package up you know 70 or 80 games or whatever but then the first thing that i did is i'm like wow it's really

hard to package software in debian how can i make this easier and so i went off and built something called deb helper which made it easy to make debian packages you You know, of course, then people were like, whoa, here's this thing that makes it much easier to do our work. So everybody in Debian started switching to DevHelper. So within a year or so, probably 50% of the distribution was built with DevHelper. But still, I mean, DevHelper was a fairly simple program. And then I went to VA, you know,

while VA was talking to me about how hard Debian was to install. So I'm like, well, I should try to make Debian easier to install. At this point, the Debian community really starts growing. It was around 100 or 200 people for a while. It got bigger during the dot-com boom and probably at some point went up to more like a thousand total people. It was a whole lot of hobbyist stuff and college students and, you know, occasional somebody in the industry who was using it or whatever. But yeah, it was a very, very wide array of different people. And very few of them were paid to work on it like I was.

I was very much the anomaly at that point. Did you rise in stature? Like if there's like a leaderboard of like important people in Debian, is it like? Yeah, I guess so. You know, like I said, everybody was using Dev Helper. And so that meant that everybody had, you know, ideas for how to make Dev Helper better and I was the person that came to them with those ideas. So I was constantly talking to pretty much everybody in the project about their pet peeve,

and then trying to get them to develop a patch or developing a patch for it. There was a group of people who had done a whole lot of very hard and important work to build Debian, like building package managers and coming up with the entire idea of a Linux distribution and things like that. So yeah, I was certainly in that group.

Being an important Debian maintainer and also being a tool creator for Debian, it's a hugely impactful role. And now Joey was planning to write a new installer that would improve everything. But first, disaster strikes. The dot-com crash happened. VA imploded. I decided it was time to leave before I got stuck doing something that wasn't just go develop Debian all day. We don't care because it was clear that

eventually the money would run out. People were like, well, we really need you to go install Debian all day on these machines here that we're going to ship out tomorrow. Nice, yeah. Or something like that. Yeah, so I decided to leave.

I bailed and moved back to Tennessee where I grew up. This was a common story of that era. The bubble burst and people left the Bay Area to go back home and look for new work. But Joey didn't look for new work. I mean, Debian was still there and there was a lot to do. Recently, the Debian project leader, Jonathan Carter, said, Debian is a bottomless pit of problems.

It happens often that our lives are completely consumed by Debian. For Joey, the thing that would consume him was his new idea for a Debian installer. So he found a way to live with very few distractions and at a low cost. I was really lucky that I was able to move into this little cabin down in a valley that was basically free for me to stay at for as long as I wanted to, which, you know, how many people have that opportunity? But we had friends who owned the place. It was kind of a ex-hippie commune and it was nice to have somebody there to mow the yard and stuff. You know, I'm used to just working on whatever I want to work on.

And somehow I had money, so I guess I'll just keep doing it, right? The thing he wanted to work on was a new Debian installer. He had come up with a design for it, and now he just needed long periods of uninterrupted work to work on it. And the place in the woods was perfect for this, although it was more of a shack than a house.

It was like a one-room cabin with a stove, with a wood stove in the middle, and like an unheated kitchen in another building. You know, it was pretty rough. It was fun, though. That was some of the, you know, I was like, I really enjoy living this way. It's such a change from being in the Bay Area. That would be a big change to go from living in the bay area to like a one room yeah the hauling and hauling water jugs down a half mile yeah well i mean what spoke to you about about that process i mean it's kind of you know i kind

of finding a way to be almost bored because because if you're bored, then you're going to think about something. And it's not going to be like what's on TV or where should I go to dinner or something. It's you know, 1230 actually opening up my laptop and just working and not being interrupted all day long and just entering a flow state and staying in it, you know, pretty much continuously. I find that I really get good work done that way. It's just, you know, it's almost an order of magnitude improvement. And so, you know, I much prefer to work that way a lot of the time and so having you know being out in a quiet place this really facilitates that and i find that it doesn't really you know i don't i don't miss out on a lot because when i go into town i enjoy that

then when i'm in the city i enjoy the city as the city. But I'm not, you know, I may be getting some work done in a coffee shop, you know, in Cambridge or something. But, you know, it's always, it's a temporary thing. And I enjoy coming back to calm and just, you know, recentering and just, you know, having the space to work on what I want to work on at home. So Joey spent around six years in that one-room cabin. He worked a lot on Debian Installer and got paid for it when he could. And a couple times a year, he went to user groups. Eventually, Debian started having a conference for developers, Debian Developers Conference, DebConf. DebConf gave Joey a chance to see the world. DebConf moves around every chance to see the world. DebConf moves around every year

to a different country. So I ended up visiting, I probably went to 10 or 12 or something DebConfs and I, in all in different countries. So I mean, DebConf often would be, you know, a small amount of talks and a whole lot of hanging out and socializing and also trying to get probably far too ambitious amounts of coding done. And often people would be up all night doing it. In fact, I've done that more than once at DevConf and other Deian get togethers i remember one time in spain i was uh up all night and i think catharists or some place uh trying to convert like a massive subversion repository into like a thousand get repositories and it kept failing and it was like all night long with somebody else like about alternating you know things like that eventually i realized that i

really wasn't doing good work at these conferences. So I kind of stopped even trying to get anything technical done. Giving up on technical work meant he could just focus on socializing and bonding with other Debian developers. And he has a lot of good memories from these times. Sleeping in Norway on the floor of, I think it was a grade school gymnasium with basically half the Debian project all in all, you know, because we were all poor back then. This was all we could afford for the accommodations for the conference. A few people brought tents and most people just had a sleeping bag or something.

The year after Norway, the conference was in Brazil, and it really stands out in Joey's mind. Because I went over like a month before the conference with some Debian folks, and we met. We were like staying with local Debian users, and we traveled around, and yeah, I really enjoyed that. I remember at one point in Brazil, I sat down for breakfast next to somebody who I'd never met before. And it turned out it was Jim Geddes, who was basically one of the main developers of the X-Window system back in the 80s.

I had no idea. He was not involved in Debi. He had just come down for the conference probably because he was like, this is a cool conference in a cool country. In 2011, the conference was in bosnia well me and a bunch of friends from the uk ended up driving with three cars full of people all the way across europe over to bosnia they were kind of reenacting top gear like zooming down the autobahn with three three automobiles all trying to. That's awesome. Yeah, and I'll communicate by radio. Oh, you guys had like little walkie-talkie radios? Yeah, yeah, or something.

I think these occasional adventures that the Debian core people went on, it was a big part of what bonded them together. Everyone was excited to see each other. And when they got together, big ideas started knocking around. Yeah, there were a lot of things that started

just as a conversation at a conference somewhere. A big one that I remember is the idea of source dependencies, which when Debian started, you would have the source that builds a package, but it wouldn't declare what it depended on to build it. Originally, Debian and all Linuxes just got bootstrapped through some trial and error process. So they ended up with a compiler that worked.

And what do you do then? Well, you ship it. And, you know, throughout my time in Debian, there was a lot of going back and fixing, you know, shortcuts or just, you know, rethinking a problem and having to do a whole lot of work to change it. It's a whole lot of what Debian's about really, or probably any Linux distribution is about, you know, continually making these big changes that touch a lot of different stuff. These big projects to refactor Debian and the fact that these improvements would affect like millions and millions of machines,

that was a big motivating factor for Joey, but they would also be why he ultimately left Debian. But I think I'm getting ahead of myself. I mean, Joey's life still revolved around Debian at this point. I was doing Debian stuff pretty much full time for about 10 years, maybe a little bit longer, somewhere in there. I mean, I did have other projects that also happened in those times, but you know, a whole lot of time was spent reading mailing lists, you know, arguing with people

mailing lists, going off and coding, answering bug reports, you know, trying to prepare releases. It was pretty much a full-time job, often unpaid for many years. I mean, that's intense, I guess. You must have enjoyed it. Oh, yeah. I mean, I enjoyed it because you could see that people were using it. You would make some improvement and then see your users either complain about it or hopefully not even notice that it happened, but you could see signs that it had, or you would just do something because it really

needed to happen to allow other things to happen yeah i i was really happy to put that time in although it did eventually become too much for me to do it was just you know but but but for many years i was very happy uh with that being my thing and was it was it like the impact of your work that you liked like you felt like i'm improving this for a lot of people or was it like the impact of your work that you liked? Like you felt like I'm improving this for a lot of people or? Yeah, it's a lot of, it was a lot about the impact. You know, I, I, like I've mentioned a couple of times when somebody from the free software community had a big impact on me with something that, you know, was small, but like, why would they bother to do that? And I, you know, I've tried to do the same thing here and there,

although I'm probably not anywhere near as good at it as some of those people. I would also try to fix dev helper bugs, especially in like within an hour, because I knew that it was blocking somebody else from getting their work done. And I would, I would release a new version with like one bug fix in it because I'm like, well, let's get it out. And I think that kind of thing did, you know, people were like, Hey, that's cool. He's responsive. He's kind of thing did, you know, people were like, that's cool.

He's responsive. He's, you know, fixing it really quickly and getting out to me really quickly. But despite all this enthusiasm, Joey did start to get frustrated with Debian. I had noticed for years that it was too hard to make large changes in Debian. The first one that I noticed was it was a completely ridiculous thing. Debian was shipping documentation in slash user slash doc, but the file system standards said it should be in user share doc.

It's like nothing. Like the easiest change in the world. Guess how long it took for every single piece of software in Debian to be updated. It took, I believe, six years and a lot of work. Six years is so much time.

And all they were trying to do was have everybody move from putting something in one place to putting it in another. But I do get why this would take a long time. I mean, Debian is a volunteer project and people always probably had something more important to do. And this happens at big companies, too. If there's a low priority, big initiative, sometimes it can take years to get complete. Yeah, I bet this is not unique to Debian at all. It's just that it's all out in the open in Debian, right? And also, you know, Debian is probably the largest free software project I've contributed to. And so you're going to see, you know, the edge cases there. And it's much easier to contribute to something with 10 maintainers.

Somewhere in the midst of all this, Joey had to leave his one-room cabin. But the concentrated focus he got there was so important to him for achieving big projects like the new Debian installer. And he wanted a way to recapture that. I heard about this place that was off-grid. And the people who lived there were getting ready to move to Africa for a number of years and were looking for a caretaker. I was like, well, okay, I can go back out there and, you know, kind of get back to where I was earlier.

The place was much bigger than his old one-room cabin, but in other ways, it was a big challenge. It had basically no power whatsoever and no internet and also no running water. So yeah, nothing. When he moved to the cabin, Joey was paying bills by doing Debian work for a company that needed it. But then he quit his job and raised some money on Kickstarter. It was a point where even a minor thing like, you know, a person writing a piece of software that most people have never heard of, once it got to a certain number of contributions, was suddenly trending on Kickstarter, probably, or nearly trending. What was your goal? What did you ask for and what did you promise? I think I probably asked for $20,000 or something. It really wasn't a lot of money, but I was just thinking in terms of I'd like to work on this for a year and see where it goes. You had already adapted to kind of stretching out your savings so that you could do this

kind of work? But I've been really lucky in that way that I haven't been dragged down by, you know, the cost of living in some city. And I can't imagine it working if I had been living in like, you know, the Bay Area or really any major city. Around this time is when I first heard of Joey. And the thing that caught my interest about him was he looked like this platonic ideal of a hardcore software developer. He was just working on what he cared about and living out in the woods. At the time, everybody was talking about entrepreneurship and startups and how you could work crazy hard

as a software developer and make a whole bunch of money and then you'd be set for life. And here was Joey and he had been through the first bubble and it seemed like he had cracked the code. He said, you know, you're looking at the numerator, how much money you need to do what you want, but I'm looking at the denominator. I've just decreased my cost of living.

I made the Zen move. So instead of hitting a big score, I can just do what I want right now. At least this was my impression from the outside. So that's really the question I wanted to ask Joey. Was this idyllic life you've built in a cabin in the woods as great as it looks? There's always a backstory that might not live up to the romanticism, but I certainly do feel very lucky that I do have a lot of ability to, you know, to take some time and just think about an idea and then be like, okay, I'm going to go spend whatever amount of time it ends up taking because it's worth doing this. It's hills around me here and a few mountains, you know, in the distance. But in dense, dense forest and kind of down in a little bowl with, you know, basically completely isolated from whatever's going on, except for whatever noise might filter up from the distant road a mile away.

And yeah, it's a very calm and peaceful place. And for me, that's more of just a background thing. I just know that, you know, I can sit down and work for five hours and that's a really nice thing. And I know that if I take if I need a break, I can go go and easily take a walk and, you know, refresh my mind. And so, yeah, it's, you know, the little things really that make living, you know, living in a rural place nice. I think I wish more people had that ability. I feel very, very lucky to have it right now. Who knows how long it will continue. It sounds magical to me, but I've also never had to haul my own water. Anyway, hauling water was

less of a pain to Joey than this thing called System D. System D isn't a knit system. It's basically the thing that starts up everything else on your computer. It's not really important to the discussion how it works, but systemd was new and most people in Debian wanted to adopt it, but most just wasn't enough. Debian had this big fight internally about systemd because it raises people's temperatures. You know, there was a lot of fighting about it, a lot of technical arguing. Debian has all this, you know, constitutional framework and a technical committee that can make, you know, that can break, you know, log jams and that kind of thing. And all that got dragged into it. I mean, it took several years really for all that to work its way out. I was perfectly happy with systemd myself. I did, you know, I did empathize with people who didn't like it for

whatever reason but it was clear to me that every links distribution except for one or two would be switching to systemd eventually and and i felt that i was pretty good at prognosticating this kind of thing so i was like if debian can't make the obvious decision in, you know, without all this much drama, it's too much for me. I don't want to be, you know, be drug through that for every decision. I would rather things move quickly rather than slowly. Joey had been contributing to Debian for 18 years at this point. And for most of those years, it was a pretty major part of his life. But he was stressed and he felt like the way the community was structured

and the Debian constitution, they were making this decision process harder than it needed to be. It was more about taking a step back from the decision and saying, hey, this isn't really healthy. It's not healthy for a project to have this hard a time making a decision. Part of it is that Debian is a large project and Debian tries to operate by consensus, which is great,

except, you know, consensuses are never going to be 100%. There's always going to be somebody who's unhappy with it. And, you know, it's just kind of structurally hard for it to move quickly. So Joey sent an email to the Debian mailing list. It was a bit angry in

tone. It's been abundantly clear to me that this is no longer the project I originally joined in 1996. We've made some good things and I wish everyone well, but I'm out. He also took shots at the governance model. If I have one regret from my 18 years in Debian, it's that when the Debian constitution was originally proposed, despite seeming dubious, I neglected to speak out. It's clear to me now that it's a toxic document that has slowly but surely led Debian in very unhealthy directions. And the type of news organizations that track operating systems, they wrote headlines about this. Like more SystemD drama as long-time Debian developer quits. That was InfoWorld. Or, what's wrong with Debian from DevWatch? The problem with a totally open project is that all

the drama ends up right in the open. And you might expect based on that, a backlash from the people within Debian. Maybe they would close ranks and shun him. But it wasn't like that at all. It was a lot like if you imagine you went to your own funeral and everybody was saying good things about you, it was one of those days. It was really nice. I'll treasure those emails. And I really miss a lot of people in the project, which is a shame because, you know, it's just not going to be the same when you're not there working with them every day. But, you know, at the same time, it was probably also a lot like leaving some company that you've worked at for 15 years and you have a lot of friends there.

Like what do people say that you found touching? You know, just a lot of heartfelt, you know, well wishes and thanks for what you did and that kind of thing. Nothing. It was more of just hearing from all these people who, you know, they were friends of mine, but you know, you don't really know until, until you do something that's almost guaranteed to piss people off. And instead they turn around and say, oh, thank you. We appreciate your work. And, you know, when the conference time comes around, do you feel do you miss it do you think back to that oh i do yeah i've tried to find another conference that's that tight but it's

hard i'm sure there are a lot of them out there but you know there there really was something special about debbie and debbie really is a family in a lot of ways with the, with the good and the bad that comes from being a family. Right. So yeah, I, I, I've searched for quite a while for conferences like that and I still miss them. And every year when they have it, I'm like, Oh, I wish I was there. Have you gone back? I did go back.

I went back and spoke, actually. It was a good experience, but not one that I think I would want to do very often because it was also a painful experience because, you know, nice as it was to see everybody, they were all talking about stuff that really was of no concern of mine anymore. And so, you know, and then you realize, well, how much of this friendship is just, you know, sharing a technical context, you know? Yeah. Yeah. I've thought about this a lot before, not with Debbie and though with like, you know, I worked at a place and I forget the word they used. Oh, they're trying to build a culture. They're like, oh, we have to have this culture, right? And I had a problem with this word

because I wasn't sure there's such thing as culture, but I tried to make an argument to them that it's actually, it's like about shared experiences. Like a culture isn't something that you hand down or that it appears. It's more just like people who may have, like it's like people go to war and they might be in a platoon with people who they shared nothing with who have very different views

but they're they're bonded by yeah the horrible experience right yeah i think you're right i think especially since i was in debian as it grew up a whole lot there were a lot of horrible experiences and in one way or another things like like moving that user doc around or having to go back and add source dependencies to everything. Yeah, definitely. The people were there with you. If you've ever left a job, you have a great job, you leave for whatever reason, and you have a notice period. It's super weird like it sounds like what you're saying about the going back to

the conference because like you were so tight and everything whatever but once you give your notice it's kind of there's kind of like everybody kind of stepping away kind of process right and all of a sudden you're still there but it's like you're not like it's right yeah i mean you know it was you were definitely on the outside looking in, even though, you know, not, not intentionally like we're snubbing you. It was just like, what can you really contribute to this discussion? If Joey's story were a movie, I think that as he left that last Debian conference,

there would be sort of a montage, you know, it would show the time he got a tour of the Bay Area and the time he was working in his cabin on the new design for the Debian installer. Or staying up late at a Debian conference working on some project and sleeping on the floors of gymnasiums. But then the montage would end, and Joey would drive off into the next part of his life.

Debian was this great thing for you and this great community. Like, what's out there now? Like, you know, if you're pointing somebody part of his life. Debian was this great thing for you and this great community. Like what's out there now? Like, you know, if you're pointing somebody towards a community to grow in. I honestly have been mulling over that exact question for a year

and I haven't come up with the answers. So I would like to know. I know people who are involved in developing geeks, which is Nick's like distribution. They're clearly involved in that kind of thing. I have that same kind of feeling that I had back working on Debbie. You know, there's clearly somewhat in various language communities, like the Haskell community, you know, has, you know,

a community feel to it where, you know, we all have a lot of libraries that we collaborate on, and there are concepts that flow through the community and things like that. But I do feel that something a little bit is missing that was there when I was younger, maybe, but I would like to know where it is. That was the show. A big thanks to Joey for being on the show. Thanks for sharing so much about your life and thanks for all the great open source software that you've created. If you want to know more about Joey, you can go to his website, joeyh.name. And if this is your first time listening, be sure to subscribe to the podcast.

The episodes come out monthly. And if you want more than that, you can subscribe to my Patreon page. Next up on Patreon will be some bonus questions with Joey that I didn't get to here. Here we mainly focused on Debian, but Joey also built Git Annex and some cool solutions to off-the-grid living using FRP and Haskell.

We'll talk about why he likes Haskell so much and other stuff that didn't fit into this story. So until next time, thank you so much for listening.