The Changelog: Software Development, Open Source - Spotify's open platform for shipping at scale (Interview)

Episode Date: October 9, 2020

We're joined by Jim Haughwout (Head of Infrastructure and Operations) and Stefan Ålund (Principal Product Manager) from Spotify to talk about how they manage hundreds of teams producing code and shipping at scale. Thanks to their recently open sourced open platform for building developer portals called Backstage, Spotify is able to keep engineering squads connected and shipping high-quality code quickly — without compromising autonomy.

Within our culture, Spotify years ago rallied around the concept of giving teams autonomy and unblocking them to basically go as fast as possible and empowering them to move quickly. With the idea that more teams building software in an unblocked manner leads to more discovery and a better product and basically a better experience for all of our customers. In terms of scale, we've been growing at an incredible clip. We're now up to about 500 engineering squads in the company. And I say about because you go with those numbers change on a daily basis as teams reform and swarm on new opportunities. We do run at quite a high scale. Basically, we have over 2,000 microservices in production, over 4,000 data pipelines, which Backstage suits as well. We also ship thousands of machine learning models. And on average, we're pushing code to production about 2000 times a day.

And that's interesting given 10 years ago, there was the 10 releases a day for the big DevOps kickoff. Being With Your Changelog is provided by Fastlyly learn more at fastly.com we move fast and fix things here at changelog because of rollbar check them out at rollbar.com and we're hosted on linode cloud servers head to linode.com slash changelog what up friends you might not be aware but we've been partnering with Linode since 2016. That's a long time ago. Way back when we first launched our open source platform that you now see at changelog.com, Linode was there to help us, and we are so grateful.

Fast forward several years now, and Linode is still in our corner, behind the scenes, helping us to ensure we're running on the very best cloud infrastructure out there. We trust Linode. They keep it fast and they keep it simple. Check them out at linode.com slash changelog. What's up? Welcome back. This is the ChangeLog, a podcast featuring the hackers, the leaders, and the innovators in the world of software. I'm Adam Stachowiak, Editor-in-Chief here at ChangeLog. On today's show, we're joined by Jim Howitt and Stefano Loon from Spotify about how they manage hundreds of teams producing code and shipping at scale at Spotify. Thanks to their recently open-sourced, open platform for building developer portals called Backstage, Spotify is able to keep engineering squads connected and shipping high quality code quickly without compromising autonomy. And today we talk through all the

details. So we're happy to have a couple fellows from Spotify here. Jim Howitt, Spotify's Head of Infrastructure and Operations. And Stefan Alund, the Principal Product Manager and Head of the Backstage Open Source Project, which we're here to talk about. Guys, thanks so much for coming on the ChangeLog. Thanks for having us. Thank you. It's a pleasure to be here. We would love to get started by understanding a little bit about Spotify's engineering culture and the scale at which you're operating. Because Backstage, which we're here to talk about, is an open platform for building developer portals.

It's really an infrastructure tool, especially around organizations that have a lot of services or microservices. This particular problem that comes at scale, and you guys solved it at scale. So tell us a little bit about the scale of the company. How many teams? How did the orgs break out? Give us an idea. So I'll start off and hand to Stefan to talk about how Backstage has solved that. So within our culture, Spotify years ago

rallied around the concept of giving teams autonomy and unblocking them to basically go as fast as possible and empowering them to move quickly with the idea that more teams building software in an unblocked manner leads to more discovery and a better product and basically a better experience for all of our customers. In terms of scale, we've been growing at an incredible clip. We're now up to about 500 engineering squads in the company. And I say about because those numbers change on a daily basis as teams reform and swarm on new opportunities. We do run at quite a high scale.

Basically, we have over 2,000 microservices in production, over 4,000 data pipelines, which Backstage suits as well. We have several hundred websites that Backstage also produces. We run over 200 microfeatures in our client, if you use Spotify and Android or iOS, that are built as well. We also ship thousands of machine learning models. And on average, we're pushing code to production about 2000 times a day. And that's interesting given 10 years ago, there was the 10 releases a day for the big DevOps kickoff. So it gives you an idea of the scale and speed. So that's coordinating all of that and letting people move quickly is a key challenge. And Stefan, you've been instrumental in helping us tackle that. So perhaps you can talk about how we've tackled that.

So Backstage is kind of the center around all of our software development inside Spotify. And as Jim said in the beginning, we really want to have empowered and small software teams that own a piece of the Spotify experience. And while we value autonomy, we also don't want every single team to have to make a lot of different choices about the technology they use and how they build the software. So Backstage helps us really to kind of balance that autonomy and speed at Sustainably by sort of introducing standardization in a nice way. I mean, standardization has kind of a bad rep for many engineers, but the way we look at it is that a team shouldn't have to make some choices.

And Backstage kind of helps provide and drive towards fewer choices and more standardization without sort of sacrificing on the autonomy part. So recently open sourced, but we found that Backstage is not a new piece of software inside of Spotify. This is something that's been baking in the oven for many years.

Is that correct? Yeah, that's correct. We actually started, I think, the first versions were kind of put into production almost six years ago. We actually started in kind of the microservice domain where we had these small teams owning their own microservices, like typical DevOps practices today. And the first need Spotify saw was to have a central place where we had a registry of all the software that we had in production, like a service catalog. Interestingly, though, what happened was that some clever infrastructure engineers started

to add tooling on top of that read-only catalog. All of a sudden, you could click a button in Backstage to add capacity for your service or redirect traffic to another region if one of our data centers was failing or something like that. So it kind of became, it almost accidentally became a platform in that sense for the microservice ecosystem that we started to build out. Can you maybe speak to the internal tooling side of this thing? I think that a lot of engineering teams sometimes feel like working on internal tooling can be a waste because they're not building product

or they get kind of lost in the minutia. It seems like this was maybe an internal tooling endeavor gone right. Yeah, I'd say so. You know, what we saw was kind of interesting that we saw those patterns that, you know, infrastructure or platform engineers that we talk about them as kind of started to add more and more use cases on top of this this you know common common system and we sort of started seeing when we started our data infrastructure organization you know the same

we started to like you know sort of broaden the scope of the platform also to cover data engineering practices and but um what we really saw was that like there was a lot of innovation that started to happen just by having one one central place and that at that point we we kind of doubled down on the you know this platform aspect i said that it was almost became a accidentally became a platform so at that point we kind of saw that, hey, we could probably build a centralized platform for all of Spotify that covers all of our different infrastructure needs and all of our domains. So me and my team started building a more opinionated platform, more of like a plugin framework, basically, where rather than having infrastructure engineers that

wanted to really push the boundaries in their domains, what they did before Backstage was build their own island of infrastructure. And now we invited them to build a plugin instead in the central place. And that model exploded in terms of the number of use cases. So now we have more than know, there's more than 140 different plugins that have been built internally and more than 60 different teams have contributed like at least one of those plugins. I wouldn't say that you asked about, you know, internal teams,

not, you know, finding it interesting to work on internal tools. We have actually seen the opposite. Like there's like, it seems seems to be an infinite number of possible things that we can add and innovation that can happen on top of this central platform. And one of our guiding principles is we wanted to empower engineers to go quickly, so we made purposeful investments in teams like Stefan's and others to build this tooling.

But the plugin, the movement from backstage to the plugin architecture was pretty interesting. At that time, I was sitting at my standing desk by the team that built Backstage, and we'd come in on Mondays, and people over the weekends would build plugins. So I remember, Stefan, the day you turn, it's like, oh, we can now do machine learning model deployments on a Monday. So just bonus functionality that would come in for places. And that started to give us a sign that we had true community,

where you're not asking people to do things, but instead each day just new software shows up that's expanding your platform, giving new capabilities and people choosing, you know when platforms are winning, when people choose to use the tool, because it's an easier path for them from that perspective. Similar, every hack week that we have at Spotify, which is an amazing experience, everybody stops for a week and can just self-form and swarm on teams. There's always about half a dozen build this in backstage kind of teams. And Stefan, maybe any of the few of the Hack Week projects that pop out to you that are interesting, that kind of showed the strength of the community. Yeah, as you said, we typically

have a lot of different teams building stuff, which is really interesting is that it's not only the infrastructure and platform teams that are building the plugins. We're actually seeing our customers, the engineers at Spotify, who scratch their own itch and see a need for something. And it's so simple to add a plugin that they can do that over a week and ship it out into production and get people trying it out, which is pretty cool.

One of my favorite plugins that I was actually part of as well during a Hack Week project was we identified that keeping track of people's tech health was kind of something that teams did using massive spreadsheets. They were keeping track of what are we deploying with here, what versions of job are we running for these services, et cetera? So we came together, a team, and sort of built that in as a plugin in Backstage where you can get kind of a bird's-eye view, a heat map almost, of your services and software and kind of get to see

what's the fragmentation we have in our team, essentially. Yep, so as you build and update software, your scores update automatically and you can manage everything in one place. You had asked a little about how did we go from internal to external from an open source project. We had this vibrant internal community. We also, about two and a half years ago, joined the Cloud Native Computing Foundation and several other open source foundations. And in May of 2019, I was basically sitting at Barcelona at the end user SIG group talking about, you know, just basically

everything from what kind of CI engine do you use? How do you do observability? And one of our senior staff engineers sitting next to me said, well, I'll show you, here's how we do a build. And he popped open his laptop in front of other users and opened up Backstage. And they go, well, what's that? And it's like, oh, this is a tool called Backstage. And he clicks away, did a build. It's like, and what's this? Oh, here's Tech Insights. And we started to click along. And a few of the end user companies said, well, this is pretty interesting. You should maybe show this at one of the demo days, you know, for the rest of the SIG. This might be something that's, you know, worth open sourcing.

And that started us on a journey where Stefan got pulled in. We did additional demos with discussions. And after talking to about a dozen or so companies, we saw that there was an interesting need there. We didn't want to just throw software over the wall for open source. But we saw a purposeful need, started to work with those companies to get feedback. And when we decided to go open source, because we saw this need, we took this as a purposeful go to market. So we came out with a website, with demos, with a mailing list, with community engagement, and trying to kind of do things the right way of open source, contributor agreements, code of conducts, and the like from that perspective. And the community was so helpful to us. And we've been so excited with the reception from

that process. And that's really cool that you open sourced it. So the million-dollar question, I guess it might be a multi-million-dollar question over the course of multiple years with multiple engineers working on it, is here you have scratched this internal need, and you've built a system which is serving Spotify very well and is a platform inside of Spotify that most people would look at that as an incredible strategic competitive advantage because you've solved a really hard problem

in a way that is awesome. And giving that away to the world seems like maybe a bad idea. So were there conversations around, should we, should we not, or was it obvious? Of course we're going to open source it, because you could also just sell this thing to other companies, because there's a demographic of companies that would need this

that all have huge revenue streams. So curious of the thought process behind open sourcing it. And we did have, it was a discussion, and we have had companies approach us that have wanted to do managed service models around Backstage. We had this discussion and there's a few things as we have a fundamental desire

to improve the developer experience, basically across the entire tech industry. The belief there is we're competing on audio and music and podcasts. If we can create a great open source experience, we're going to attract talent. Yes, some people will use Backstage as well, but people will come to Spotify because it's a place where you can build amazing things like Backstage. You can open source them. You can get those open source credits from that perspective. Also, in working with him, we contribute to other open source projects.

We benefit from that. We know we're going to get more back than we essentially lose by other people adopting this. And as Stefan mentioned, Backstage is like a central nervous system for our developer community. It's entwined in so many of our systems. We're in a good place where we've got a head start. And as new plugins come in, as new capabilities come in, we can pull those in. And it's kind of a rising tide lifts all boats. We get people who come in on day one and they know how to push code in Backstage. People come to us for Backstage. And I think we've made our first hires through the open source project. And Stefan, maybe you've got a few examples of some pretty interesting pull requests that came in on the open source that helped us out maybe you could share yeah i mean we i think we were kind of

surprised by um and delighted obviously like by people coming in and starting to contribute to the like to the project almost from day one or actually on day one um and we we think that i mean we have solved a bunch of problems at spotify but we also think that collaborating with the broader ecosystem of infrastructure engineers across other companies and other open source projects, we will eventually build a better experience through Backstage and bring that back to our engineers. I think it's pretty cool already that a couple of weeks ago, they were actually the second company, third party company that adopted Backstage. They were really keen on having a better experience for API documentation.

And this is not something that we had a very good story on internally at Spotify, but they came in, had expertise in that area and contributed functionality that we are now bringing back into Spotify already now. So, yeah, we're reaping the benefits of collaborating with a broader set of developers. We firmly believe that this will give us a net good, and it lines up with our values of trying to create a great collaborative developer community that empowers developers everywhere. It's important to hear that too because I think that when you see why, the why of open source is often the mystery to some degree. And to see that your heart's in the right place, that one, you've gotten value from open

source and you recognize that value. And two, you want to give back. And I think going back to keeping the main thing the main thing you didn't say that we say that but you kind of said it through your words yeah audio music podcast is your main thing you compete on that level and rather than compete on this strategic advantage as jared mentioned you're giving it away in open source and in many ways propping up a community around that because you recognized how important it was inside of your organization in terms of community. But, Jim, you mentioned the terminology central nervous system. And as Stefan and you guys are describing this,

I'm thinking this is kind of like a brain for your organization. How often do you have, what do we have out there? Where is it at? What version is on? Whose teams are managing it? And this seems to solve that kind of problem. Stefan, you've demoed this. Please talk to it. That's definitely true. What we found was that by talking to other companies

who have tried to build a developer portal and tried to build a central repository of all their software, is that sometimes that repository became kind of it goes stale it becomes like an accounting system where your teams have to go in and add you know here's our services etc they're really interesting so i think you know secret source of backstage is that we kind of integrate the tooling on top of that metadata that you have in your catalog. So we start off by, you know, you get features if you keep the metadata about your services, etc. up to date. We kind of have this really nice way of encouraging teams to keep the information

up to date. And that means that we can build even more interesting tooling on top of that rich and up to date information about our whole software ecosystem. So that kind of feeds the cycle of plugins and improving the discoverability of everything in the ecosystem. And there's a bit of a flywheel. So as you do a build, you get a prompt to update your documentation, You up your documentation. You show up on the what's new. You're exposed for service discovery and documentation discovery.

So if a new team comes in and they're looking for X, they can find it. Instead of building it themselves, they reuse yours to your question to try to find something. This is you would lose trying to figure out who owns this, how do I get them, is a button click away and moves right into a Slack channel. People either resolve items, and that's part of the reason that you see Spotify rarely interrupted from that perspective. So everything from developer reuse to production operations

to even supporting our audit and compliance is all in one place. And that service catalog is the memory store for that brain analogy that you had. To some degree, maybe even a social network. If you can see other teams solving problems that you haven't even met yet, if those engineers, they're solving similar problems that I haven't even met yet in Spotify or my org if I adopt backstage, then that gives me an opportunity to, as you had said before, community. And I think that's something that kind of is missing in large organizations. Jared, you mentioned at scale.

I think smaller orgs may have less of this problem. Like I know you. I kind of know what you're working on. I kind of probably know who owns it if it's just you and I primarily. But in large orgs where it's 500 squads, which I can imagine is more like 2,000 engineers or more, it's going to be hard to know everybody and connect. It becomes more important as you grow as a company.

One of the things we have seen is tracking the time it takes for an engineer, a new engineer that joins Spotify. How long does it take for that engineer to get productive? We measure that by a time until it takes to merge your 10th pull request. Just not a perfect metric, but a metric that we use to look at. And what we were seeing before Backstage was essentially that that number was going up steadily. So we got slower. For every new engineer that joins Spotify, we got slightly slower, like less efficient. And after rolling out Backstage and really doubling down on this centralized place to have all your technical information and all your tooling we've actually cut that number in like more than half like 55 percent it has gone down over the

last couple of years and even though like you know not all companies are onboarding engineers at the same pace as spotify, basically that is a fantastic proxy for kind of how complex your ecosystem is. So if you reduce that, it also has benefits for your other engineers because their lives are easier, they can find stuff easier and be more productive as well. Thank you. Instantly troubleshoot your applications on Kubernetes with no instrumentation, debug with scripts, and everything lives inside Kubernetes. But don't take it from me. Kelsey Hightower is pretty bullish on what Pixie brings to the table. Kelsey, do me a favor and let our listeners know what problems Pixie solves for you. Yeah, I did this keynote at KubeCon where we talked about this path to serverless.

And the whole serverless movement is really about making our applications simpler, removing the boilerplate and pushing it down into the platform. Now one of the most kind of prevalent platforms today is Kubernetes. It works on-prem, works on your laptop, works in the cloud, but it has this missing piece around data and observability. And this is where Pixie comes in to make that platform even better. So the more features we can get from our platform, things like instrumentation, ad hoc debugging, auto telemetry, I can keep all of that logic out of my code base and keep my app super simple. The simpler the app is, the easier it is to maintain. Well said. Thanks, Kelsey.

Well, Pixie is in private beta right now, but I'm here to tell you that you're invited to their launch event on October 8th, along with Kelsey, where they'll announce and demo what they're doing with Pixie. Check this show notes for a link to the event and the repo on GitHub, or head to pixielabs.ai to learn more. Once again, pixielabs.ai. so so stefan on the break you'd mentioned so the listeners don't get the break, so give us a little bit of what you just shared there in terms of how your orgs are structured. Jim, you'd mentioned earlier around 500 squads, quantifying that to product owners, managers, business folks,

a lot of people involved in this. So in many ways, it's to some degree a social network, but also very much a nervous system for your organization. But if you have orgs out there or businesses trying to give small squads like that autonomy, startup-like features, Backstage gives them that. Kind of go into further why that's important for your teams to have that autonomy and that kind of drive and sort of speed to innovation and maybe even speed to the 10th commit being coming faster rather than like a lot slower yeah so we have a i think a very interesting setup in

the way we organize our engineering or r&d teams at spotify we we call them squads and basically all those small squads have they have a mission and that mission could be you know to do podcast ingestions or recommendation systems or you know work on our cdns or or you know everything that makes up small pieces that makes up the spotify experience and all of those teams are almost set up like small startups where they have like one product manager who's set in direction, figuring out what to do. They have all the engineers that they need. They have front-end engineers, back-end engineers, machine learning engineers. Some teams have designers if they have user-facing features.

And they have engineering managers as well. So they're like one tight-knit team that is there to build one part of Spotify. And we want them to be able to iterate as fast as possible on their domain with ideally as few sort of dependencies as possible, because that's when they move fast and are empowered to kind of solve the problems because they know the domain, they know how to solve it. And the obvious drawback of something like that, that kind of autonomy, is that it's really hard to know what other teams are doing. And it's easy that you can reinvent the wheel and multiple teams building similar things. And that's, once again, where Backstage comes in and allows us to work that autonomously

because there is one central place where you can go and get a bird's eye view of all the different teams, what software exists, what libraries are out there, you know, what technologies are used in production so that you don't have to kind of reinvent things. You can build off of other people's what they've

already learned and you can you know use for example you know a treasure trove of like existing data that we have in our ecosystem like all that data is available to everyone in the company you can see who produces the data and you can sort of build more higher level algorithms on top of it without you know starting from the beginning and then kind of scaling that up if you you know in the days when we had a small number of squads you could keep everything in your head once we got more squads than dunbar's number that was very hard to keep all of that in your head to the social networking analogy backstage created that directory where you could see who's doing what, who do I need to connect to, how do I jump right into their Slack channel,

where do I see their documentation and get started. If you look at these benefits about enabling a small team to not have to rebuild tools, of being able to have a mobile engineer, a backend engineer, data scientists all work from the same tool set. You multiply that by 500 teams, that's a ton of productivity. You add product managers, other business owners that are looking at things, looking at insights and data. That's also another 20-25% benefit. And then if you think about the onboarding, Spotify is rapidly growing. In any given year, 30% of our company has been here less than a year just because we're just growing so quickly. So if you can come in on day one, get your training and education on our tech stack, start building things in Backstage, get to your 10th pull request 25 or 30 days less,

those productivity gains are even faster. And in talking to other companies, talking to people who invest in tooling we found that once you hit about 100 microservices you need something like this you can keep it all under your head until things get kind of crazy you just answered one of our questions which is what size orgs need something like Backstage

so let's skip that question altogether 100 microservices or more, there's your key and let's talk about how it does what it does we've been talking about the benefits how it's helped you how it's helping other orgs as they adopt it in the open source world and the ecosystems built around it but peel back the covers how does backstage do what it what it does and then on the other side of that coin is like how do people interact with it and developers build what they want to build with backstage but first of all stephan maybe tell us how it all works. So as Adam described it,

there's a central nervous system, like the brain. And the brain is what we call the service catalog or the software catalog, where we keep track of all the software and the teams and who owns what, essentially. And that model, what we do is that we keep a YAML file, a small configuration file, a metadata file, that regardless of where your software is stored, you keep that information together with your code. And then Backstage harvests that information and makes it available

in a centralized repository so that you can build on top of it. And that allows you to model and keep track of microservices. It also keeps track of bigger monolithic applications where the application can be divided into multiple logical parts owned by different teams. But it still keeps track of, with that metadata

YAML file that stores the source of truth for information. The same goes for data pipelines and machine learning models, et cetera. So that's the starting point. And then on top of that service catalog, what we do is that we integrate all these various tools that you need. So it could be rather than going into your Jenkins machine and looking at your builds, you start from the service catalog in Backstage and find your service. And when you click into your service, there's a plugin there, a UI for showing your builds.

So it's kind of an information architecture. We want engineers to reason about the stuff that they own, the services that they own, rather than the tools themselves. So this is a very different approach to how many organizations adopt infrastructure. They add infrastructure to their organization and then engineers need to figure out like how do i wire these things together how is like you know tool a connected to tool b we take a pretty different approach we basically build plugins then for all of those

different infrastructure tools and so integrate them into one place and the plugin is essentially a small web application that one team can build and so iterate on by themselves. So for example, we have a team at Spotify who keeps track of our deployments and runs our Kubernetes clusters for their customers. Not only run the Kubernetes clusters, they also build a UI, a plugin in Backstage to make it simple for engineers to kind of see what services are running and do rollbacks and those kind of things. So those are sort of the key parts to the architecture.

So Kubernetes on the backend, and then is that container orchestration like agnostic to where you go ahead and get that deployed into the world like how does it hook into a some sort of a cloud so backstage kind of abstracts away all the different infrastructure pieces that you have so it could be your cloud tooling it could be your ci environment it could be your security scanning like all of those different tools that you normally interact with directly, they are integrated into one experience instead. And those kind of plugins, they are expressed and showed as a plugin.

So basically the team that manages Kubernetes has a web service that's invoked by Backstage that essentially, once it detects a build is done, can pull the code over and start a deploy process. And that allows that team to work unblocked. So first it was deploying into basically our own orchestration system. Then later, as we moved to Kubernetes, we basically that then triggered the deployments in the Kubernetes APIs based on the YAML that you had set up in Backstage that you had set using standard templates. So you had consistency. And now that team is working on things like doing automated canary analysis,

safe deployments, and they can work and build that capability. And it just gets extended into Backstage. So essentially, you've got our Kubernetes team working unblocked, you've got the backstage team surfacing, and then you have feature teams, people actually building features and deploying. And for them, it's just simply one day they deploy the next day, it's like, oh, I'm getting automated canary analysis up. The automated canary analysis is getting even better. And for them, the ecosystem just gets richer and richer from that point of view. Okay, maybe walk us through an end user who's an engineer's experience. So say I'm working for Spotify,

and I've been tasked with creating a new service that recommends the best developer podcasts in Spotify's catalog. And so I write this little Go program, which every time you ask it, it just returns the changelog, no matter what you send it, just the best, obviously, the correct answer. I have my little Golang binary, I have my repo. How do I get it plugged in? How do I say, okay, this is a service in

Backstage that anybody can call? Where do I go from there? Just get the YAML metadata and I'm done? Great question. I think we have a concept that we call golden paths at spotify which is basically like a standardized way to create you know a piece of software so we have one back-end golden path we have one data golden path machine learning golden path you know web golden path okay so what you typically do is that you follow that path and a lot of the steps in that path are done to Backstage. So before you even decide how you want to build your service, you basically go into Backstage. Rather than picking a language and picking the framework and deciding how to set up the CI, etc. You don't have to do that at all. You just pick a predefined Golden

Path template, and that template basically gives you a Hello World application with Kubernetes deployment information, CI configuration set up. It uses the best practices that we have at Spotify for how you build a microservice and it removes that choices and I talked about standardization before and how

backstage drives standardization. That's the primary way. We go in and we make the experience of using the preferred path better and it's like everything is automated for you. I love that because instead of standardization slowing you down

standardization is actually speeding you up. Exactly. So you could literally do our onboarding, go to your Hello World rename it to podcast recommendations. Your code in there is going to be relatively simplistic. It's going to be

on one return change log know, change log, deploy it, roll it to production. You'd automatically have your Grafana integration, your tracing and observability integration. You'd check if it was a compliant-oriented or non-compliant type of a feature right out of the box. If it was a tier one service, needed global deployment versus a single region deployment, that would be covered as well. And if you wanted to

expand or create a new golden path, then you would work at a golden path together and basically build that as a software template that would be added to, and that's what we're working on basically people right now as they adopt be added to it. And that's what we're working with basically people right now as they adopt Backstage to use. Yeah, because eventually somebody's going to be like, that golden path is not my golden path. I have problems with that golden path.

So I'd imagine as we talk about Backstage, there's a difference between maybe how Spotify uses it and maybe what's baked into Backstage that's open source. And so maybe help us toe that line or at least define it as we bump up against it. And so whenever you have these golden paths, I imagine it's since you're an organization that cares about autonomy and ownership and stuff like that, there's a way to push back on that and say, like, I think this is the wrong golden path. We make a new one. And this is a consensus that you come to.

So this is organization-based, less baked into backstage, Backstage enables it for anybody who uses it, right? This GoldenPath analogy or this GoldenPath opportunity. So if a company comes to Backstage open source, I mean, we don't expect them to kind of use our GoldenPath because we have our opinions. We have our way of building stuff at Spotify, but the Backstage open source software templates, as we call them,

the point of them is that you make your golden pot. You set up what is the preferred templates that you want to have in your organization. And then you start to drive towards having that opinionated way of building your software. What we have found, though, is that it's really important for standardization not to become this top-down thing that we don't really want. It's important to have those opinions and golden pot recommendations be very strong, but still have them held very loosely. So that they're essentially code. And what happens internally at Spotify is that if an engineer feels like, hey, this is actually not the best practices right now,

they can go in and make a pull request towards that software template that challenges the standard way. And if they can motivate that in a good enough way through community discussions, that becomes a new standard. And from there on, everyone uses that version.

So we don't aim for these golden pots to cover everyone's use case. We want to try to cover, let's say, 80% and then leave 20% for teams that either don't want to or can't. You're trying to enable speed. You're trying to enable speed. So if I want to do what Jared's doing here, which is create this awesome new Go service

that recommends only the changelog for best developer podcasts, just saying that one more time for everybody to know, I want to be able to follow this golden path. I don't want to have to rebake all the things. I might have opinions, but I can redefine those opinions as you just mentioned. But any new engineer can come and take that first step or those first few steps into a new thing with so much more assurance that the right steps is the right steps because they're not having to think, well, is

my opinion different than everybody else? Is this going to be accepted? Will I be shamed? You know, will I get a, you know, a PR against my code deleting it all essentially, you know, like these things can all be avoided by giving this consensus, the standard path, this golden path that you mentioned. Yep. And if you think about onboarding and you're walking into a platform with 8 million transactions a second at the perimeter, 299 plus million monthly active users, and you're going to deploy a feature on day 10 to production. That could be terrifying. But Backstage essentially gives you the armor, the guardrails. You know you're going to go out and things are going to be working well.

On the flip side, we are always innovating. A team that is pre-MVP exploring a brand new area that the golden path doesn't work, they build their own framework in place and extend it in. And then they can get all the benefits of still the tooling, the community, the alerts, the monitoring, instrumentation, collecting events for basically observability and data analysis, but they're not heavily constrained. They can create a new path. And if that MVP takes off, then as Stefan mentioned, that becomes something that gets adopted and grows in our own community and helps us evolve. And if you are a new company using Backstage, you're going to have your own opinionation. Backstage will let you make that opinionation friendly, easy,

fast and with a UI that's pleasing, makes developers and product managers and the like able to move basically quickly and easily from that perspective. And we talked about, since we released Backstage open source, we talked to a lot of companies and this way of kind of having standards seems to be very compelling to people. It's basically many companies, when they grow to a big enough scale, they see the need to kind of reduce fragmentation

and insert a few standards and make sure that people are solving the right kind of problems. And I think Backstage has this nice way of introducing standardization in a way that it's actually a benefit for the engineer, not something that ties your hands behind your back. And that seems to be resonating well with people that are looking at Backstage. I really like that style of there's one place that you go,

and that place has the standards baked into it and it hands them to you versus it dictates in some sort of documentation of course we'll talk about the docs too which is a cool aspect of what's in there but like the templates are there for you to get started and you plug your your secret sauce into what we already know as the standard versus you having to go and read a spec or follow a thing, a tutorial. I think that's really cool. So there's two concepts in Backstage that you guys have on the website that I'm trying to plug our conversation into. The first one is plugins.

And I think that's like the backend kind of thing, like the Kubernetes plugin. There's maybe like a Postgres plugin. There's like CI. Is that the kind of thing that a plugin is? And then the templates, is that the golden path, is the templates? Yeah, the templates are the golden path. The plugins are kind of how you extend, and I talked about before,

how you build, how you integrate different pieces of infrastructure into your platform. So the long-term vision here is that we want there to be a thriving ecosystem of plugins, and those plugins ideally should be for every project that's out there. So a company that walks up to Backstage has a gallery of existing plugin integrations to whatever infrastructure they're using. So if they're running on Amazon, if they're running on Azure DevOps pipelines,

if they're using Lighthouse to track their accessibility scores for their website, if they're using Grafana, if they're using Kubernetes, etc. Our hope, and what we're starting to see now, is that there is a plug-in for that, and they

can pick and mix the plug-ins in the ecosystem and sort of install them into their version of backstage and make it their own and if it if the tools you know everyone has you know homegrown uh you know infrastructure in their basements right not everyone is running on like latest cloud-native stuff and the coolest stuff. And for those people, you can basically build your own plugins, custom ones. So your Backstage deployment becomes a combination of open-source plugins that you pick off the shelf, and then you build your own custom ones

and roll them into your version of Backstage. And a key aspect of the plugin architecture is it basically followed the Spotify model of autonomous teams. Without the plugin architecture, you would need this portal team to build things or do integrations. With the plugin architecture, essentially anyone can build a plugin. And that's when Backstage moved to plugins, we saw the explosion of contributions internally. And it essentially lets everyone just move at whatever pace they want.

What's up, friends? When was the last time you considered how much time your team is spending building and maintaining internal tooling? And I bet if you looked at the way your team spends its time, you're probably building and maintaining those tools way more often than you thought, and you probably shouldn't have to do that. I mean, there is such a thing as retool. Have you heard about retool yet? Well, companies like DoorDash, Braggs, Plaid, and even Amazon, they use retool to build internal tools super fast,

and the idea is that almost all internal tools look the same they're made up of tables drop downs buttons text inputs search and all this is very similar and retool gives you a point click drag and drop interface that makes it super simple to build internal uis like that in hours not days so stop wasting your time and use Retool. Check them out at retool.com slash changelog. Again, retool.com slash changelog. so guys we've been talking around the docs let's talk directly about the docs backstage has tech docs built right in with markdown based free documentation you get as part of this awesome infrastructure. Tell us about how this came about

and how it fits into the Backstage story. We did a survey of our internal engineers some years ago. One of the main blockers for productivity. One of the main things that came back from that was that it was really hard to find technical information at Spotify. And some teams put their documentation in markdown files, some put them in Google Docs, in spreadsheets, in GitHub wikis, etc. So what we found was that engineers didn't even know

where to start looking for documentation. There was no starting point. So during another one of those Hack Week projects basically, a small team of a few engineers and some of our tech writers came together and looked at that problem.

And what they built was a plugin in Backstage or basically two parts to it. came together and sort of looked at that problem and what they built was a plug-in in backstage or basically two parts to it so we adopted a docs like code approach where you where engineers keep documentation together with their code so it's really easy to basically keep your documentation up to date because you know if you change your code you can change documentation in the same pull request so you don't have to go into a separate system to kind of you know update the documentation as well and so go out of the flow so engineers really love that what we also the other dimension of solving this problem was that you know we have all these teams building

documentation together with the code what we did was to bring all that, like we basically built documentation sites during the CI process, integrated nicely with our CI system and then published all those documentation in one central place in Backstage. So it's kind of a really nice combination of

engineers having a, like can keep their existing workflows like Git and code, while still making all the documentation centrally available to everyone in the company in one place. And that plugin or that system that we call TechDocs I think was one of the most successful internal projects that we've ever done,

ever rolled out. We didn't have to do any marketing of it internally. Engineers just loved it from day one, and it just had tremendous adoption. Can you give us maybe a primer on the DOS-like code methodology? What does that mean? When you say that, for those who are not familiar with that, what does that really mean?

So it essentially means that you treat documentation as code. And what we mean by that is you write your documentation as markdown files. And they typically have a documentation together with your code for your website or for your service or for your data pipeline. In the same repository, you have a docs folder. And in that docs folder, there are markdown files, basically, that are different chapters. And what happens then, during the CI process is that we use an open source project called MK docs, or make docs, to basically translate those markdown files into HTML, like web content. And it's those web, you know, the resulting websites, the content that we then make available integrated in Backstage.

So let's say that I go to a website or data pipeline, I can see, you know, all the characteristics of it. And then the documentation for that system is just one click away for the person who wants to consume it. And it's the same documentation that's in your GitHub repo. So you don't have things getting out of sync. When you go to a site, you'll see that this documentation was updated eight minutes ago or something like that. So you know it's current, you're not going, is this the documentation or is there something else?

And the adoption rate, I think the thousand documented component, which is the name of something in our software catalog, was in under six months. Which was just the fastest level of adoption I've seen in 25 years of trying to figure out how to solve this. What about contribution to those documentation? Is it only from an engineering side where it happens in code only, or is there an opportunity for other folks in the squad or whatnot that may not be so much in the code? Is there a way from inside of Backstage to contributors

or is it just a one-way path where you just extract via mkdocs? So basically, when you read the documentation and you want to make a contribution to it or edit it, etc., there's a simple click of a button and then you end up editing that file in GitHub, GitHub Enterprise in our case. So it's still an engineer-focused experience of writing the code. So it's not for everyone. It's pretty opinionated in that sense. It's primarily for engineers. But what is really cool, I think,

is that once you treat documentation as code, there's a bunch of other sort of code-related tooling things that you can build on top. So the team that built this out, they have innovated a lot on top of the documentation. One example is, let's say you want to, you know, someone reads documentation and there's a sort of error in it and you want to change it. What you can do is you can highlight the documentation and then a pop-up appears and clicking a button and you can create an issue, a GitHub issue. So the documentation problem is then treated as a bug for the owners of that. And they can triage the bugs and sort of squash those bugs as they would with any software code.

And all of those things kind of contribute to documentation being kept up to date. There's a feedback loop between people who read the documentation and the ones who own it. And if you think about, you know, one of the mantras is fix bugs before you build new features. If your doc's out of date, someone flags it. It's a quick bug fix from that perspective. And this model's taken on so quickly that our architecture documentation or expectations of teams for how they operate software,

how we respond to incidents, are now all in tech docs as well. So it's all in one place. And Jim, you mentioned this was something that was adopted inside of an organization faster than you've ever seen in the last 20 years or something like that. Can you quantify that some more?

Yes, so basically whenever you do doc, and there's been tons of documentation tools and open API standard I've used over the years, but you're always nagging people. And it's like, well, I want to ship a feature, I don't want to stop and do docs. From this case, once we rolled this out and we actually built a squad around this team,

as Stefan was referencing, how do we roll this out. And we actually built a squad around this team. We, as Stefan was referencing, it's like, how do we roll this out? We just started to see in our logs and in our dashboards, it was basically an exponential line of going up. And within six months, we had a thousand different components, websites, microservices that were documented. And as you clicked on each, you could see the documentation was up to date with the last build, which was amazing. I've just never seen that even at startups, at scale, at companies that are compliance oriented, you have to hire whole teams of people to go do documentation. And this was something as we started to look at open source, and we were talking to other companies, it's a problem in the open source world of trying to

keep documentation up to date. You'll find a cool repo and then the readme and you go to the docs page and it's an empty folder and there's nothing to do. And then you're looking around and stack overflow to try to figure out how to use it. And then you wind up going to some other open source projects. So this was something as we open sourced this component, we had thousands of hits on our blog about it on Backstage. We had a demo video and an earlier demo video of Backstage in general on the Spotify R&D channel on YouTube. And people were bookmarking the minute and second in the video where we were showing docs and going on LinkedIn and saying, take a look at this. So it is kind of one of those just amazing kind of adoptions and it shows what comes out of Hack Week and then what an ecosystem like Backstage can raise to the front.

Now you get it as part of natively when you adopt Backstage open source as well. And you know the software templates that we talked about before, they come of course like pre-wired with all the documentation so an engineer doesn't have to do anything to get documentation. It's just there, scaffolded and ready to go. Super cool. I see why the adoption is so high inside of Spotify, because you're really making people's jobs much easier to do. And instead of it being like friction to get in your documentation,

written and read, it's just like right there along with everything else. How about adoption outside of the org? So it's been open source for a little while now. You all have a vision for this, which I thought was interesting, very intentional with your open source, you actually have a vision for the project, which you state is to become the trusted standard toolbox for the open source infrastructure landscape. How's it going? It's been out there for a little while we We have some CNCF stuff going on, but it's a big ask to say, hey, it's not like try my library, right? It's not like here's a cool command line tool

that you should try out. It's like, hey, run your company around this piece of software. So I'm curious about adoption and if there's been any struggles or challenges you've had to overcome to get other organizations involved. I can speak to that.

Yeah, so we have, from day one, I think people were pretty excited about the vision. It looked like we had done our due diligence and talked to a bunch of companies in similar situations as Spotify. And I kind of identified that this infrastructure fragmentation and proliferation of tooling is a problem that not only Spotify is solved or challenged

with. So from day one, we kind of had fantastic reception. However, people didn't really understand what it was. They bought into the vision and sort of got excited about it. But then since Backstage is so many different things, and as you said, Gerald, it's like

a complex piece of software, we were kind of struggling to explain what it does. It's a toolbox. It can be anything you want by this plugin framework. But too much choice and too much ambiguity, people need to get something that they can relate to so we had to you know write some blog posts about like what what this is really and try to do some videos demonstrating sort of the longer term vision of how it could look you know further down the line because you know the open source project is you know just getting started and there's a long way to go until you get something that looks like what we have at spotify and yeah so the kind of of the main thing that we heard when we released it was that, so

where's the software catalog, the service catalog? That was kind of the main thing that came off when we talked to a bunch of companies. So they saw our videos and we demonstrated like how the service catalog and the central nervous system, as you said, Adam, how that was crucial to starting building your developer portal. And that wasn't part of the initial release. So I wouldn't say we pivoted, but we doubled down on building out that service catalog and tried to, as quickly as possible, plug that gap in the story. So now when you have Backstage,

there's a service catalog and you get value sort of out of the box. Whereas when we launched Backstage, it was kind of like, hey, here's this open framework and it can become anything. And that was, you know, people didn't really understand that.

And we were lucky we had the videos that we could show what we were doing internally. So it was real software. It wasn't mockware from that case. But go ahead, Stefan. Yeah, I was just about to say that once we shipped the first version of that service catalog, then we started to get real adoption. People started to use it.

They started to set it up internally at their organizations, do internal demos. And some teams even started building a bunch of different plugins as part of kicking the tires and trying out the platform. So I think the pivot towards focusing on releasing that service catalog was kind of the key enabler for companies to kind of come on board and start using it. This reminds me a lot, not so much, bear with my analogy, but it reminds me a little bit of like

the integration of SAP. I live in Houston, which is the energy capital pretty much of the US at least. And a lot of people here work in oil and gas and everybody uses SAP. I have friends who are just simply product managers, project managers of integrations. They take a long time. So this is very similar in the way that it has those benefits. It can run an org essentially. It's very much like that central nervous system. And I don't know if it would be a benefit to you, but maybe study the wrongs they've done in SAP to not integrate well. Maybe that might be a homework item for you all to avoid, I suppose.

Because it has so much power, but only one done right. And understand the value of it. It's a big, as Jared said, a big ask to integrate this. And it can have such benefit, but it can be so massive in both its adoption as well as its ability to progress an organization forward. I've been on a few multi-hundred million dollar SAP implementations. And I think it's big, it's a big investment, it's a lot of change. What we're trying to do backstage is make it easy and start with one service, add the next.

As Stefan had mentioned, in our product marketing team, we're trying to treat this as a true go-to-market. So when we got that feedback, we're trying to make the templates, the getting started pack, the documentation easier so you can get out of the box. You can even install it very quickly, get started just like any open source. And as we've been doing that, as Stefan mentioned, our adoption has gone up. We've talked to over 200 companies now. We have 15 basically committed adopters on our list. That includes some very interesting companies that have shared what they're doing from that perspective. And at this point, I guess, Stefan, how many external contributors do we now have?

Well over 100 right now. And I think we're at a point where somewhere between 40 and 50% of all pull requests are coming from engineers outside of Spotify. And I think we're also, with this kind of increased adoption now, what we're seeing, which is tremendous, is that companies are putting together a team inside their company to be

sort of the backstage team, to be the evangelist of their platform. And they're starting to build their plugins. They're starting to evangelize the platform internally. While we had people coming in and helping and contributing to the project already from day one, we're seeing now a shift in the kinds of contributions that we're getting. When people's jobs are actually to build the backstage and to be the backstage team inside different organizations, they also share back substantial improvements to the platform to fully working plugins and just a completely different level of maturity when

it comes to the contributions we're getting now. And to the question you asked earlier about why open source and what's the return, 100 contributors is like 20 squads. So we have one squad on Backstage that's working it, and now we have 20 open source squads equivalents that are contributing software to us. So it's a pretty good payback. Have we mentioned being sandboxed with C and CF yet? Not probably in a clear way. Let's maybe quantify that, because my question really is now that you're at this stage

with the CNCF, what's the support from them? What's the inertia that they bring to the table to help adoption with this? Our CNCF discussions started two and a half years ago. At Spotify, our big fans of open source, we had aspirations to give back

to the community. We were looking for the right projects. Backstage was the, we're hoping, the first of many that's had community interest, community adoption. Many CNCF companies helped us shape the case, understand the offering. A key step to going to CNCF is when we spoke to companies, especially bigger ones, they had a fear. It's like, what if we adopt Backstage and Stefan wins the lottery and he retires? Or Spotify decides to turn it off from that case. It's like, what could happen to the code? So basically, one of our guiding principles is by bringing it to the CNCF, it becomes a community project. It is bigger and broader than us or any other company. We're showing it's a permanent commitment.

It's going to continue to live and have contributions. Sandbox is the first stage, but it's a key milestone because now it is officially part of the CNCF ecosystem. You can adopt it without fear that you're picking up something that could be a proprietary or eventually closed software. You have the appropriate licenses in place so you know someone's not going to come along and say you're using software that has a copyright infringement on someone else's intellectual property that would get you in trouble. So that means it can be picked up by banks, pharmaceutical companies, airlines, tech startups that get bought out and there's no particular ownership or IP strain that could get you from that perspective.

It also for us gives an endorsement of trust, of community, of adoption that will lead other people and that will let us go to the next stage, which is incubation. With some of the large scale adopters we have, as they move along, we move to incubation. The rate of change will slow, will go more to an incremental improvement. That now shows it's an even bigger, more serious project will drive adoption. And our goal is to bring it to graduated status, just like projects like Envoy and Kubernetes and Prometheus. And then it truly does become that vision, that standard of a developer portal for a great developer experience that can take the CNCF technologies

or what technologies you have. And that's the vision around them. And that was just announced this week and it was a very proud moment. Congratulations. That's an awesome achievement for sure. As you were describing the kind of businesses that could use this, I was thinking of, I suppose, the way the

world is now and the fact that we live and run on top of software. And so if something like this can be deployed and used as it has been, as you've demonstrated with Spotify elsewhere in the world, how much better may be me getting on an airplane, feeling more confident that it's not going to crash or more confident that, you know, that I will actually have a seat and I, you know, I get there or I, you know, what, you know, rinse, repeat, my bank is more stable. My money is more secure, you know, whatever it might be, you know, that, you know, as you sort of draw back on the bigger picture of what open source is and why give this back to the world, I think that's it is that if we can all live on and build better software, that's a better thing for the

world. Yep. And it's better software, faster, you're comfortable with compliance, you're comfortable logging and monitoring and traceability. So if it's an airplane, you know that they're able to deploy software and fixes and manage reservations faster. And even during COVID, we had a company, Weaveworks, that used Backstage to help get radiographs to doctors faster, which is just a really kind of inspiring use we never expected. So all of those giving back and wonderful cases are really, we're looking forward to hearing more. Also, we have a lot of engineers that are working from home

now and you know stressful situations and what Backstage ultimately does is like improve the lives of engineers and improves the developer experience and so if we can help people enjoy their work and enjoy their

distributed work in a better way that's a win as well. And especially if you're new and you're joining and it's like, who owns this? You can actually find the library. You can find the owner. You can find the Slack channel. You can find the documentation. So it's an important social connective tissue in bringing on new developers at this time. What's the best URL to share via audio for our audience to bake into their mind to check this out? Is it backstage.io? Is that the place to go? Yeah, that's a great starting point.

You got links to videos, the source code, everything there. We also try to keep a very, you know, we try to engage with the community. So we have a Discord chat channel that everyone can pop in and ask questions. And we're really trying to encourage like a vibrant community. We spend a lot of time thinking about

how we can make it inclusive and how we can sort of anyone participate. If we come back to that vision part of like making Backstage like an ecosystem of, you know of tools, we, Spotify, cannot build that ourselves. So for Backstage to become the product that we want it to become, there needs to be a thriving ecosystem

where many companies contribute, not only Spotify. Spotify is just one out of hundreds of companies contributing. How active is the newsletter you have there? So we can tell the audience to check that out because it's active. They should subscribe to that, pay attention. Absolutely. And we have a fantastic marketing team that makes sure that the communication goes out and that it's high quality and relatable as well.

Very cool. Awesome, guys. Well, is there anything else we haven't asked you that you're like, man, I really wish Jared and Adam asked us that question or talked about that scenario? What have we not asked you that you can share? We've talked about the volume of our deployments. We've talked about how we went to market and the lessons we've learned.

The end piece, definitely, in the community and an inclusive community is key to us. So I think, you know, we've got some of the... We've covered it all? Yeah. All the major points. Awesome. Thank you.

Got it all, I think. Well, fellas, thank you so much for your time. It's been an awesome conversation. Thank you for, I suppose, the four years worth of work, plus all the effort into this and the examples you've given from Spotify's point of view in terms of how this helps engineering teams. And having that position of thinking that this should be open source versus paid product to other organizations, that's something we

obviously enjoy as individuals, but here's to what this shows because that gives more teams out there software they can use. It's like this. So thank you so much for your time today. It's been awesome. Well, thank you for giving us a chance to talk about use. It's like this. So thank you so much for your time today. And it's been awesome. Well, thank you for giving us a chance to talk about this. It was a great conversation. For sure. Thanks a lot.

That's it for this episode of The Change Law. Thank you for tuning in. If you haven't heard yet, we have launched Change Law Plus Plus. It is our membership program that lets you get closer to the metal, remove the ads, make them disappear, as we say, and enjoy supporting us. It's the best way to directly support this show and our other podcasts here on changelog.com. And if you've never been to changelog.com, you should go there now. Again, join Changelog++ to directly support our work and make the ads disappear. Check it out at changelog.com slash plus plus. Of course,

huge thanks to our partners who get it, Fastly, Linode, and Rollbar. Also, thanks to Breakmaster Cylinder for making all of our beats. And thank you to you for listening. We appreciate you. That's it for this week. We'll see you next week. Thank you. Bye.