Orchestrate all the Things - Open source software economics and community health analytics: Enter CHAOSS. Featuring CHAOSS Project co-founder Georg Link

Episode Date: May 3, 2021

Trying to capture the value open source software generates can be a bit chaotic. The CHAOSS project may lend a helping hand. By now, the tension between commercial interests and open source projects is well known.  Trying to balance building a sustainable business and a community around open source software, in a cloud-first world is not the easiest thing in the world. In the latest episodes of a long-winding saga, two more commercial open source vendors, Elastic and Grafana, changed their licenses. Their rationale was clearly communicated as trying to protect the business they have built around the respective open source projects from cloud vendors that they feel compete unfairly with them, without contributing as much as they do. Interestingly, all parties involved refer to "the community" as being front and center in what they do. While obviously important, however, what constitutes an open source community, how it's faring, and what value it generates all seem rather vaguely defined. The people working on the CHAOSS project under the auspices of the Linux Foundation want to change that. We caught up with Georg J.P. Link, CHAOSS project co-founder, to find out more. Article published on ZDNet

Welcome to the Orchestrate All the Things podcast. I'm George Amadiotis and we'll be connecting the dots together. Trying to capture the value open-source software generates can be a bit chaotic. The chaos project may lend a helping hand. By now, the tension between commercial interests and open-source projects is well known. Trying to balance building a sustainable business and a community around open open source software in a cloud-first world is not the easiest thing in the world. In the latest episodes of a long-winding saga, two more commercial open source vendors, Elastic and Grafana, changed their licenses.

Their rationale was clearly communicated as trying to protect the businesses they have built around the respective open-source projects from cloud vendors that they feel compete unfairly with them without contributing as much as they do. Interestingly, all parties involved refer to the community as being front and center in what they do. While obviously important, however, what constitutes an open-source community, how it's faring and what value it generates all seem rather vaguely defined. The people working in the chaos project under the auspices of the Linux Foundation want to change that. We caught up with Georg Link, Chaos Project co-founder, to find out more.

I hope you will enjoy the podcast. If you like my work, you can follow Link Data Orchestration on Twitter, LinkedIn. and well actually the topic of the conversation which is the chaos project and actually that's that's a little question to start with I wonder if I'm pronouncing it right because well that's what it looks like but maybe well you know you know better no yes chaos is the correct way to pronounce it the chaos project it is short for Community Health Analytics Open Source Software. And the goal of the Chaos Project is to bring together industry professionals, academics, and anyone else who has an interest in understanding the health of open source communities and using metrics to understand how healthy communities are. So putting some numbers to this understanding and having that transparency. Okay. And so for my background, I'm originally from Germany.

I grew up there. Fun fact, I grew up on the 800-year-old castle. Not in the castle, but on the premise of the castle. And after high school, I did an apprenticeship in a bank. I earned a bachelor's alongside with a focus on financial services, business administration. And then I went on to a master's degree in management information systems. I had the opportunity to go on an exchange program to Omaha, Nebraska, where I earned an MBA. The credits transferred seamlessly between the programs, but I fell in love here in Omaha,

Nebraska, and so I still live here. I came back to earn a PhD in information technology. And this is where the current research focus on corporate engagement and open source community health led to starting the Chaos Project. So today I co-lead the chaos project. I'm a director of sales at Vitergia. I teach courses on open source communities at Brandeis University and I lead the IEEE SA open community advisory group. Okay, well that's a lot to manage and well you sound like you have a busy schedule for sure. And for me personally, the first time I came across the project was I guess a few months or maybe a year back. But I've been a long time proponent of open source and I'm a user.

I sporadically contribute and I'm also very much interested in my writing revolves around data, metrics, analytics and all those things. So the chaos project is, I would say, at the confluence of those two things. So I kind of naturally gravitated towards that. And before I expand on mentioning the specific context that attracted me, maybe it's interesting if you can share a few words about the background of the project itself. So when was it initiated? Who were the first people that got involved? I should also mention that it's under the umbrella of the Linux Foundation.

So I wonder what the exact relationship there Carlos, who was trying to understand how software is being built in open source. And they didn't have any tooling for that that was open source. So they built their own, open sourced it. And this is the Grimoire lab tools. And they were building a community around this. Eight years ago, they started Biturgia as a company to provide services because the interest was pretty big to understand how open source communities work. And then a couple of years ago, they approached the Linux Foundation to move their GrimoireLab open source project under the umbrella of the Linux Foundation. At the same time, this is where I come in,

we were researching community health at the University of Nebraska at Omaha. And we went to the Open Source Leadership Summit, which is the annual Linux Foundation member summit now. And in the spur of a moment, we put up, hey, let's talk about community health as an unconference session. And the interest was so big, we had to pull in chairs from other rooms. And so after that session, we talked with the Linux Foundation

that, hey, your members are interested in this. We are interested in this. Can we continue the conversation? And they said, yes. And we also have Biturgia and we have this tool. Biturgia was providing metrics to the Linux Foundation already. So yeah, let's bring this conversation together. And that's how we started the Chaos Project in 2017.

So four years ago, and then it was officially announced at the open source summit North America with Jim Zemlin on the big stage calling out that, hey, community health is important, and now we have the Chaos Project. Okay, okay. That's interesting. And yeah, it sounds like you have captured some momentum there, all right. And just quickly browsing through publicly available material, like things you have on GitHub and the project website, I saw a few names actually there and I'll be honest with you, I didn't actually do the work of tracing back each and every one of those names, but it seems

like there's a number of people involved so, and you already mentioned the connection with the Linux Foundation, so I wonder plus the fact that there is Bitergia, which is in my understanding a commercial entity which is also involved so I So I wonder what the model is for the project

basically. So is it in some way sponsored by the Linux Foundation? Do the people who contribute work independently and are paid for by the organizations they work for? Are some people at least full-time on with the project? So how does it all work? The Chaos project, as all Linux Foundation projects, has a project charter.

And in the project charter, it says it's an unfunded project. So there is no money that members pay to be part of the project, unlike other Linux Foundation projects. And so we rely on donations, on securing sponsorships to run events and do other things. The members involved in the Chaos Project come from a variety of different companies, foundations. Over the years, we had Mozilla, Eclipse, Linux Foundation.

At our events, we had Software Freedom Conservancy, Open Source Initiative. we have different companies from google gitlab red hat vmware the the list is pretty long we have in our metrics release so one of the things we do is we define metrics we list 160 contributors who have helped shape the definition of these metrics. And on a weekly basis, we have about 30 to 40 active members. And we have currently sponsorship from the Sloan Foundation. This is through the University of Nebraska at Omaha and University of Missouri. Matt German-Pree and Sean Goggins, who I was working with during my PhD, have this grant that allows them to be dedicated to the Chaos Project. It also funded the community manager, Elizabeth Barron, who is amazing. We love her. And we also have sponsorship from the Sustain OSS group for our podcast.

Okay. Yeah, I did notice you have a podcast and you also seem to be having an active community. So I guess both the sponsorship and the work of Elizabeth, whom you mentioned, kind of so. And the other thing I noticed just by browsing around your material is that, well, A, it looks quite diverse and also quite well organized. So you have different working groups. You have software that you produce. You already mentioned in your initial introduction to how the project got started, the GreenMoir software. You have lots of activities such as the podcast, for example. So I wonder if you can quickly mention what the working groups are, because I think these

are pretty interesting and understanding and we're going to get to that, the specific metrics that you devise. You have divided them in areas and you have assigned a work group to each of these areas. So if you would like to introduce those by starting with a different work group to each of these areas. So if you would like to introduce those by starting with a different work group. The Chaos project is... We're doing two things in parallel. One thing is we built

software for metrics and the other thing is we define metrics. And we do those in parallel because they inform each other. As we are coming up with new ideas for metrics, we can prototype them in software, build them out. And then also as we are building them out, we learn about the feasibility of metrics, what else we can do, get feedback from people, and then we improve our metric definition.

When we started the project, we had four software projects. One is the Grimoala project, which was donated by Vitugia, and still to this day, Vitugia is the core maintainer behind this project. And so to your earlier question, there are employees of Biturgia who work full time on the Grimola project. Another software is Augur, which is a research project at the University of Missouri and the University of Nebraska at Omaha, to learn about the metrics and implementing them and having hands-on experience with them, which informs the research there. But then also it's already been used by Twitter and others. We have Cregit, which is a project by Daniel German, very targeted at the Linux kernel for understanding who contributed what, not at the line level like git blame would do, but at the lower level at the token level. So you can see this variable was introduced by this person.

And so if you want to touch it need background information you know who to talk to and then we have prospector which was donated by red hat which has a interesting the prospector has an interesting perspective of like saying hey these are the metrics I care about this is how I weight them and then you get like red green yellow for how healthy the project is so we have these software projects on the one side and then we have the metric definition work on the other side. And this is done in five working groups. And the working groups started to focus on the conversation because at first we had just a long laundry list of metrics. And we had like, I don't remember, really long list of metrics. And it was unmanageable. So we started a working group on diversity, equity, and inclusion.

Because that was an important topic that a lot of people cared about. So we started to have conversations about how do we capture that in metrics. Evolution, at first it was called growth, maturity, decline. Now it's called evolution to focus on just raw numbers, basically. How is the project evolving? We have the risk working group that looks at license risk, business risk. And then we have a value working group that looks at project velocity, project popularity, labor investment, to understand the communal value of open source communities or projects, the individual value to the contributors, organizational value to companies. And the fifth working group is common metrics that touch on all of these areas

or that just didn't really fit anywhere else. Like just basic what, when, who was involved, like types of contributions, activity dates and times, burstiness, time to first response. Those are examples for the common metrics. Okay, I see. That sounds very interesting. And yeah, I wouldn't blame you for not actually remembering how many of those metrics you had initially, even broken down in the way that you described,

they seem quite a lot. And the way that the groups have evolved actually also seems kind of natural, organic, I would say. Whoever is interested in a topic just kind of naturally gravitates towards that. And if you have enough people, you sort of organically break them down in groups, as you described.

I wonder how these working groups actually work in practice. Well, obviously not so much on the procedural side of things. Let's say I've already seen, if I'm not mistaken, that you have regular meetings of the people who work in the working groups. But I was more wondering about the evolution of the metrics themselves. So just to pick one that caught my attention from the examples that you gave, labor investment. I'm sure there are other metrics that are far more straightforward than this one. Like, for example, I don't know, number contributors or a number of commits or what have you some things are rather can be rather straightforward but for something like this labor investment i think it's a good

example because it sounds abstract enough that it i'm kind of guessing it may not even be a metric per se. Maybe it's a KPI, so a synthesis of other metrics. I'm guessing you have both. Would you like to tell us a little bit more maybe about this specific one and using that one as an example, how do you define metrics collectively, I guess. You're absolutely right. We have different kinds of metrics. Some are like raw numbers, counting number of commits, counting number of issues, counting number of people. And then we have more aggregate metrics, synthesized metrics that are pulling these threads together and taking them to the next level. And this is an example for the labor investment metric. The idea for this metric came from a conference session we had. We love to run conference sessions where we had a birds

of a feather. We said, hey, anyone interested in talk about value metrics, come and talk with us. And one of the participants was sharing how in their company, he had gone and said, okay, on average, when we get a commit or when we write a commit, this is how much it costs us. Basically, here's all the money we spend on our engineers. Here's the number of commits, issues, pull requests, whatever we do

in open source, we divide the number up and then we get a value for each type of contribution that we as a company make. And then let's turn this around and look at what are the contributions that we get from other companies, from the community. And then we can assign this dollar value to all of these contributions and we can turn it into a number where we say, hey, we have invested this much money into our open source project. And if we assume that the community has the same rate of cost to produce the contributions, or if we had to produce all of this ourselves,

this is what the community saved us. And that was quite interesting because now they can go back to their managers and say, open source is saving us this much money and investing in proper community management, investing in being present at conferences, talking to communities and doing things that are not directly code related now has value because we can see the return on investment there. And so what happened after this conference, we went back to our regular process of bi-weekly meetings, talking through the GitHub issue tracker. We defined the metric. We wrote it down, what we learned, what we'd heard,

improved it a little bit. And then it went through our review process where we have a public comment period before it entered the official release. Okay, I see. That's very interesting and I'm sure it will become obvious in a bit as we progress through the conversation why that particular metric piqued my interest because, well, some of

the reasons I mentioned already because it sounded abstract enough so thanks for providing the background on how it came to be basically and one kind of meta question let's say that I had was well which is also reinforced let's say by the background you gave, was how many of the metrics that you use, would you say are actually open source specific versus closed source, let's say? Because the fact that you do have a community and therefore, in a way, some of the effort is you can say outsourced even though as you already pointed out there is a kind of fine balance there because you may not be

paying for the contributions of those people directly by hiring them but you pay maybe indirectly because well you to nurture a community you have to do things such as going to conferences and producing documentation and so on and so forth so there is a kind of intangible let's say cost and benefit function at play there so I'm wondering if you have any any idea and insight as to how many of those metrics that you use are, would you say, are specifically exclusively open source related versus any kind of software versus how many of those could apply to any kind of software? It depends on the context here. So if we look at the metrics, many of them apply to open source projects just the same

as they would apply to an internal software development team, especially when you do inner source where you're trying to build a community, break down silos, create a culture of open collaboration within your organization, the metrics are very similar. The thing you want to do first is figure out what are the business goals that you have for your open source engagement, for your inner source projects, figure out what the business goals are, then ask questions about what does it take to reach our goal. So if our goal is to grow the community, then we have questions like how many developers do we have? How many other types of contributors do we have?

And then we can go into the metrics. We can say, okay, how many contributors do we have on the commit side? How many people are opening issues and pull requests? And now we can make a good rational argument for why we're looking at these metrics and how that helps us reach our business goal and make decisions for, hey, we're not actually there yet. So let's focus our attention here. One example is let's grow our community. How is the contributions from outside being handled? That's the question.

And then we can look at the metric for time to first response. How long does it take the core team of the project to respond to people who are making contributions from outside? And we know that if you don't get a response quickly, people just go away. Whereas if you engage them and say, oh, thank you for this issue. What are your thoughts? How can we help you? Then they're more likely to stay engaged and become even more engaged. And so having a difference there where, and this is totally a human nature, you like to talk to your friends more than you like to talk to strangers. And so we often find a disparity where the core team reacts quickly to other core team members and then outsiders are often ignored longer time so having that transparency helps

in both cases open source and inner source equally okay and yeah you also touched on an interesting point there. Like, for example, how people in general tend to respond faster to people they work with or already know as compared to others, outsiders, let's say. So I wonder how much, you know, that would be something I call intangible. So I wonder how many of those intangibles did you come across? And I'm guessing probably quite a few of those. And how successful do you feel you have been in capturing those intangibles in the metrics you came up with? We are still working on capturing all of this we have only released something like 50 metrics at

this point and every every release we keep adding more and more and more because we hear from people and the experiences i i don't have a hard and fast number. The challenge that we as a chaos project are working on right now is how do we inform people who want to do metrics to get started on their metric journey? So that is our current effort to lower that barrier and help everyone who wants to bring transparency to their projects, to have the software, to get started, to know how to interpret the metrics, and then

go through this goal question metric approach for deciding on what metrics you want to pay attention to. Because when you deploy any software and start measuring, there is so much data, so many metrics that you could be looking at and without a strategy and a focus, it's overwhelming. It's like opening a fire hose of data. And so that is where our effort is right now to help people focus on the ones on the metrics that are most relevant to them by the way just out of curiosity have you ever encountered any pushback in what you do and i'm

referring again especially to the intangibles so reactions of the type like oh you know we don't actually want to measure things like a community engagement or health because that would be i don't actually want to measure things like community engagement or health because that would be industrializing what we feel is just good work, basically. We definitely have conversations about how do we measure things without creating an environment that is going against the culture of the community? So when, when you start measuring, people will start to look at the numbers and you get what you measure. And so there have been examples for, Hey, if you review patches, then your reputation in the community increases.

And so what would happen is people would wait for an experienced person to review and then they pile on with their own review, just thumbs up again. And so this kind of gamification of metrics is something we've talked about time and time again and it just simply requires to be on top of it to have this reaction from the community watch it and then change your metric strategy The interesting thing is metrics are driven by what I see, two different perspectives. One, we've talked about organizations wanting to have transparency into the projects, foundations wanting to have transparency about how good they are stewarding the projects and showing to their members, hey, we are taking good care of your projects,

your contributions are valued. The other side is communities themselves want that transparency as well. Community members like to see, hey, I'm having an impact. Look, I'm on the leaderboard. My contribution matters. And so we see this from both sides.

Having these metrics helps the community. There are some other things. Gamification was one where we want to be careful. Another is when we talk about diversity, equity, and inclusion, where we start to deal with personal identifiable information that can be sensitive. Outing someone as being gay and they live in a country or environment where they can be targeted for that. So those are things we are mindful of as we go about this. But our goal as Chaos is to say, hey, here are the metrics you can be measuring. Careful, here's a place you might want to be more careful about going down, but here are the options.

And just to kind of tie it together, the metrics and the software that you referred to earlier. So, you know, suppose your evangelizing work is successful and you have a new adopter project or a community that comes to you and says, well, okay, we're on board, we're convinced we need to use those metrics. How would they go about adopting them? And I'm guessing they don't have to adopt them wholesale like all or nothing. They can maybe adopt some. But how would they, for example, actually tie, suppose they have a GitHub,

suppose they have an online forum or other software that they use to keep track of their activity and potentially also their internal communication, maybe their JIRA or other project manager software. So how do they tie these things together to the software that you provide as a project? And how can they use the metrics through that?

Do they have dashboards or something? The first step is to get off zero. You're absolutely right. You cannot look at all of the metrics at once. Start small. An easy way to get started is to use Cauldron. If you go to cauldron.io, it's a free platform. You can type in, here's my GitHub project or organization,

and it pulls all of the repositories from that organization. If it's spread out across different repos and organizations, then you can pull all those into a report. It's completely free. It's built on the Chaos software and you can start to have metrics. There's even a tab that has the Chaos logo with metrics that we are including in our community health report, which if you don't want to do this yourself, you can go to the Chaos website. There is a community health report link, and then you just tell us your information about the project and we send you a PDF with some metrics. So that is the easiest way to get started. And then you look at the metrics. If you have questions, you can come to the Chaos community.

We have regular meetings, office hours. We have a mailing list. We have a matrix IRC. I think we're just about to add a Slack channel. And then you can ask questions about, hey, what does this mean? What do you see? And what really matters here is to bring the knowledge that you have about the community in context with the metrics.

Because we can look at the metrics and say, oh, it seems to be going down. And then you say, yes, but we actually implemented a new process for code reviews. And so we have fewer commits, but each commit has gone through a more rigorous process. They're a little bit larger. There is more engagement from the community overall. But this one number just happens to go down,

but the quality, the engagement, everything else is going up. So it's actually a good thing. So we need to have this context from the community, the understanding of what's happening, and then we pair it with the metrics and we can start to tell a story. And that is how we get started with metrics. You start to tell the story and back it up with metrics. And then once you do that, we can start to add more and more metrics and become more and more sophisticated with that. And then if you want to run your own dashboard, your own platform, you can use any of the software we have in the project and deploy it yourself. Or, of course, you can hybrid church.

Okay, I see. Right, so this is the part that Viturgia plays into that. I guess you provide support services to projects that want to adopt the methodology, the metrics, the software and need some help with that. Correct. Viturgia provides the consultancy around metric strategy, around how do you implement it in your day-to-day processes. We go through the goal question metric approach with our customers.

We help with hosting, maintaining, updating, supporting the platform. Biturgia can run reports, say here you have this question, here's the answer. And that is what Biturgia does as a service on top of the metrics. But the tools being used are the same open source tools you can find in the Kiosk project. Yeah, that's a very common model same open source tools you can find in the chaos project yeah yeah that's a very common model in open source actually so it makes sense that you also apply it there so i mentioned earlier in the conversation that there was a specific thing that led me towards the the chaos project and that was the confluence of two of my interests, let's say.

So data analytics, metrics, all of this thing and open source and open source projects and actually also commercial open source. So the trigger for that was, well, as I'm sure you know, and many of the people who listen will know as well, there has been a kind of debate going on around commercial open source and specifically open source projects backed by commercial companies. And there was, let's say, mostly with cloud vendors and the context there is that many of those companies that build the open source software that actually most of them were started as open source projects they were successful, a company

was built around that to commercialize it and so they're having trouble with the way cloud vendors use the software and their claim is that well they're having trouble with the way cloud vendors use the software and their claim is that well they're not giving enough back or maybe in some cases they're not giving anything back. There have been a few analysis on that mostly in to the best of my knowledge the analysis that have been done to that day are focused on contributions in terms of code. And they say, for example, oh, you know, the main contributor to the code base of project X is actually the vendor that is built around that project.

And the cloud vendors that use that project don't contribute so much. And the counter argument in many cases from the vendors that use the project without actually contributing in terms of code that well yes but you know we shall spread the adoption of the project we contribute in terms of community and so on and there have been a few ways that companies have reacted to that situation one way was that some companies basically changed their licenses so that they would discourage or forbid cloud vendors to use their software. Others have suggested that maybe there's a better way.

Maybe we could actually measure the contributions, not just in terms of code, that much is clear, who contributes what, but actually in terms of value overall. So things like community health, adoption, resolving issues and all of those things. And it sounds like very, very much in sync with what you're doing. To me, it sounds like the closest thing,

the closest match to implementing something like this proposal which initially sounds over-ambitious. I think the closest match would be something similar to what you're doing. So I wonder from where you stand whether you think this is actually feasible and whether you've had anyone coming to you and saying, well, hey, we would like to use your metrics, your way of doing things to actually do that, measure value in a holistic way and actually give credit where credit is due. There are several conversations going on around this topic because, who creates it.

There are, I think, three perspectives that we need to differentiate here. One is the perspective of the open source project, where as an open source project, because of the open source license, we don't care about users, how many users, where they use it, because the open source definition says we do not discriminate against any users. What we do care about is contributions. And so as an open source project, we want to have transparency in who is contributing, how much is being contributed, how active is the development, where are things being neglected, do we have funding to run the events and do the advertising and whatnot that we want to do as a project. The other two perspectives are the commercial perspectives of companies that heavily do

research and development, create an open source project, or heavily invest in an open source project. And then we have the other perspective of users who don't really invest in it. They are the ones using the open source software and they don't have the expenses of investing and developing open source. And I think Dries Beiter did a really good job in his blog post on balancing the makers and takers to scale and sustain open source software

to explain how this dynamic is shaping up and what this actually means for open source projects. So I recommend the blog post by Dries Bieterich for this. The place metrics come in here is, and this is a conversation that we've had so far is takers, the companies who use open source software without heavily involved in the development, can use the metrics to see how much value do we get, how much should we give back if they are conscious about this, and having that transparency and the makers likewise can can use the transparency to make a similar assessment with how much are we investing how much does the community give back to us with the license change that is a story i think think, outside of the community because it the communities and see how has this license change affected the collaboration. I think when we looked at Elasticsearch, a lot of the development was done by Elastic. And so changing the license, you still have the Elastic employees

doing all of the work after that. So at least on the code development side, the majority of the work continued just the same as before. So having those kinds of metrics and insights really helps you to understand what kind of open source project you're looking at and what that community looks like. Yeah, I would say that in broad strokes, you know, just black and white things, I would classify most open source projects in one or two categories. There are those that are heavily reliant, let's say, kind of built around a single vendor that employs most of the contributors and, you know, organizes events and does all the community outreach. And then there are a few that are more spread out let's say

so you have more contributors for more organizations and for the former as you said for the example of Elastic well maybe changing licenses doesn't have that much of an impact because most of the contributions were coming from one place anyway but for the latter that would be different. Yeah, again, this is where context comes in. And as we are looking at the metrics, we need to know the story of the community and we can use the metrics to tell the story better and more objectively.

Yeah, I think that's a good takeaway point from all of this, which actually applies in a broader context, not just for open source, but in general metrics are good, but to tell a story you need to know the context, you need to know how to interpret them. Yes, absolutely. Great, thank you. It's been really interesting and thanks again for making the time for the conversation and if you have any closing thoughts or comments you'd like to share, feel free. Otherwise, thanks again for your time. closing thoughts or comments you'd like to share feel free otherwise thanks again for your time yeah I'm in closing if any of the listeners are interested

in community metrics join us at the chaos communities and up to our mailing list you can join any of our calls we We are pretty open and friendly, I want to say. So feel free to just drop us a line, listen to our podcast to learn more about how different companies and different users have used metrics and implemented them. There's a great many stories that are worth listening to. And thank you, George, for inviting me on to your podcast.

I really appreciate talking with you. Great. Thanks as well. And I may testify to the fact that, yes, you look very friendly indeed. So people should not be afraid to join. I hope you enjoyed the podcast. If you like my work, you can follow Link Data Orchestration on Twitter, LinkedIn, and Facebook.