Software Huddle - Building + Evolving Sentry's Architecture and Funding Open Source with David Cramer

Episode Date: November 12, 2024

Today, we have David Cramer on the show. David is one of the co-founders of Sentry, an application monitoring tool that's one of the most widely-adopted tools for developers. Sentry does over 300,000 events per second on average, and there's a lot of fancy work to process these application errors, from rate limiting to fingerprinting to counting to source map unminifying. We walk through some of the architectural changes and systems design work here, including some of David's thoughts on shipping. David and Sentry also have a unique approach to developer marketing. They do some cool things -- sponsoring and then buying the amazing SyntaxFM podcast, sending $100k of free gifts to developers, and launching the Open Source Pledge with $500k donated to open source developers.

But what I do need is Redis or whatever that looks like Redis to always be free. And I'd love if they had a way to sustain the funding on that to develop it and all these other things. And there's going to be a variety of approaches. Sentry's approach is one approach that works great if you can do a SaaS service. But a lot of things, there's no way to actually commercialize them in any tangible way. Marketing developers, you say you do a lot of marketing. You've been sort of all over the C-suite within Sentry now. What works and what doesn't work with marketing developers?

What doesn't work is pretty easy. What's interesting is if you just ask yourself, am I actually genuinely providing value? It's a pretty easy litmus test. How do you see Sentry within the larger observability ecosystem? Who are the competitors who are more like the compliments and things like that? What's up, everybody? Today, we have David Kramer on the show. He is one of the co-founders of Sentry, which is just one of my favorite tools to use. I always want to use it on a new project or a new company that I start at. So great discussion with him. He's a systems guy,

so talked a lot about the architecture of Sentry, how that's evolved over the years. As they scale up, they're now doing 300,000 events per second on average across a month. So a lot of cool stuff there. We also talk about open source. We talk about marketing to developers and some great stuff. So feel free to check it out. As always, if you have any questions, feel free to reach out to me or to Sean. With that, let's get to the show. David, welcome to the show. Thanks for having me, Alex. Yeah. So I'm excited to have you here. You are a whiskey aficionado. You're a Porsche owner, but most importantly for the podcast, you are the co-founder and

CPO at Sentry. Can you tell us a little bit about your background and about Sentry? Yeah, kind of stereotype of Silicon Valley, self-taught software engineer. I've been out here building tech for quite a long time. Generally speaking, I was just attracted to bigger system scale problems. And so it's usually consumer companies, something like that. But, you know, a lot of my career has been infrastructure or particularly trying to make my job easier, which generally meant doing like internal infrastructure or developer tools or whatever you call it these days.

It's been a long time since I've been a full-time engineer, probably almost a decade. I still hack on things in my spare time, but we spun up the business in a significant way with Century about 10 years ago. And I've transitioned to a number of roles since then, but I've always been really focused on the product and engineering side of things. Yep. Awesome. Well, I love Century. I think it's one of the first things I always want set up at a place or on a new project or something like that. I guess like maybe just position like, I think of Sentry, I'm like, hey, this is a great way to do like application monitoring, error tracking type things. How do you see Sentry within like the larger observability ecosystem? Like who are the competitors who are more like the compliments and things like that?

Yeah. So we're kind of, I would say a little bit unique in that our approach to the market. So when you start a company, I invest in a bunch of startups these days, but the advice I always give people is just do whatever you want. Kind of like do the thing that's interesting to you, because if you fail and success is, I don't know, it feels a little bit random, but I think it's all driven by just like hard work and the right opportunity. And so if you're going to succeed, it's probably going to take a very long time. So you might as well do it the way that's interesting or spend your time in a way that's rewarding, right? And so for me, that was always, I wanted to build software that was used by as many of my peers as possible because I have great memories.

Early career, going to conferences, like the hallway track of like just hacking on on stuff with people talking about what we worked on, and then open source, being able to use each other's work, right. And so when we really focused on the business of century was designed to be a really large scale thing, you know, I, I'm sure there's a time in the future, hopefully, where I can't say this, but the goal was to monopolize the entire category. Not with any like shady tactics, right, but by building the best software that was affordable, accessible, all these things, right. And so unlike traditional enterprise companies, we kind of have this focus on make sure every software team can use Sentry and then try really, really hard to make it actually useful so they want to use Sentry. And the core of that was we started as

this error monitoring concept, which honestly, in hindsight, was one of those areas that like, duh, it's like we all break shit every day. It's probably more important than a lot of other tooling. Uh, but nobody really focused on it. And so back then, and I'd actually even argue still today, cause we, we do focus on startups as competition in the sense of we don't allow anybody to like kind of compete with us. Um, but the goal was really like to compete with a lot of the similar stage companies like smaller companies venture funded not the big enterprise players so like not app dynamics new relic think of any big observability monitoring company you can you can name uh

obviously we're much bigger now so it's a little bit more nuanced and i would say now our competition is probably every monitoring vendor in the space even if we don't actually build the functionality that they have, which it's a very different thing. I will tell you from an engineering landscape, it's much less interesting now because the way you succeed is no longer by building the best product, even though that's, that continues to be the thing we try to focus on. It just by doing clever business tactics. But broadly, tell me more about that. Like, what do you, what do you mean by clever?

You have any examples you can give there? Well, it's like, if you think about big vendors, like just the biggest companies in the world that live in a space, they're all like a lot of checkboxes is how I always describe it. And I mean that in a negative way. It's not like nobody wants like a, like a bad product experience. And that's actually like a really effective way to compete is like you offer a bunch of the functionality your customer wants, so they'll use more of your product, right? And that just seems uninteresting, because often that means you're building the second or third best product,

and it's much more inferior. And so Sentry has really strived to do that one thing, do it extraordinarily well. But as we've gotten bigger and bigger, and we have many, many more customers these days, there's this demand for do more things. And then on the counter side, doing those at scale is hard um and we have a lot of customers so doing the scale of like different kinds of customers is hard uh and so we're kind of in this middle

ground right now where we actually we do really well in our core market which is call air monitoring whatever it is um and that's great but we don't yet really compete and say enterprise application monitoring or even from the the point of view of like like a breadth of product, even though we have a little bit more today. And so we're on the cusp of trying to go from one product that's really good to say And do you think that's something that has changed just like in Silicon Valley, in the tech industry more broadly? Or it's like, hey, Sentry has moved to a different size and that's happened? Or is there like this larger, like, hey, we want to standardize on something. We want one vendor to rule them all. And what happens is that never works because problems are too complicated, especially these days. Like the internet's much bigger, data's much larger. And so like we just want vendors to solve everything, you know. And somebody tries, they do mediocre, a new startup comes in, does one thing really well, displaces them.

It's a cycle, rinse and repeat. And we're very conscious of that. And so I don't think it's a new thing. I think it just, it's a cycle rinse and repeat and we're very conscious of that um and so i don't think it's a new thing i think it just it's inevitable but it's also more present for us now because we've solved that one problem so well and we've almost like even though we're pseudo open source it's not really a standard but we've almost created a standard for every competitor just looks like century now that tries to do this um which you know it's kind of a good thing. But it also means, okay, what's next? And I think our side from the stage of the business that we're at, because we're now like a hundred million plus revenue business and you got to do more.

You can't have one product. And that's what separates from almost like a startup and a true long lasting independent enterprise company. even though I find the word enterprise a little dirty. So it's more our stage, I think, than the category or anything else in the industry. Yeah. Yeah. Interesting. Tell me a little bit about the origin story of Sentry. Because I was reading some blog posts and it looks like the initial commit was 2008. But you were working at some other big places between that and 2015 when you went full-time on it. I guess, what happened in that seven-year period?

Yeah. Century, going back to my core thing, I just wanted to build software that all my peers used that made it more rewarding. Open source was a good vector for that. And so Century was just open source. It's not quite open source. There's some nuance around it today. source it's it's not quite open source there's some nuance around it today um but it's generally free software still and that's always been a big part of my career and then combine that with going back to like i was in an infrastructure i was into making

my job easier and stuff and so century is just a side project for a very long period of time uh it was useful to folks and so they adopted it so it encouraged me to keep working on it eventually we decided to monetize it with a cloud service and then over even even that was 2012 i think when we bootstrapped it and so we did that for three years almost before raising venture and then you know last 10 years have been very different but they're they're all these little like stages right and it's actually interesting because if you think about technology or startups usually they're not a what 16 year horizon or something, and we're not even done. We're still solving the same problem the project started as, like that air monitoring and that core aggregation concept.

That's still a big investment for us. That's kind of wild to think about because if you look in the industry, there's very few technology projects that actually iterate on that long-of-time horizon. Actually, talking about infrastructure, I think Apache is kind of iterate on that long of time horizon like i actually think talking about infrastructure i think like apache is kind of the counter example of this there's a lot of projects that go there and they're very stable right but zookeeper is probably my favorite one to bash on but like zookeeper could be a lot better than it is today right but it doesn't evolve and so the the only way it evolves is competition a new product comes in and so it's it's kind of interesting because I think people

underestimate how complex the change of technology is and the demands for technology become over time. And that's, you know, we started supporting small use cases, one programming language to many programming languages to bigger use cases to now, I would say almost any size of company and almost any technology stack for our core product. And that is a lot of engineering work. Yeah. What sort of, I guess, led you to actually raise funding there going into 2015? I guess like, so you've sort of been hosting stuff for a couple of years then.

Did you have enough where you and your co-founder and whoever could have just kept bootstrapping and been full-time on that? Or did you need the money to actually go full-time? Or what was your thought process there? Yeah, it's interesting because I look back and we successfully bootstrapped the business. When we raised our seed, I think we had like 600K in revenue. We were profitable. It was myself, my co-founder. I think we had just hired somebody. And we had projections. It was growing okay. And so we could hire a few more people, but it was always tight. You're almost zeroing out the bank account over and over and over and over. There's two problems with that. One, we're an infrastructure company, so potentially you have to overspend on the hardware costs for capacity, especially because of the volatile nature of errors, and you needed to scale up to some degree.

The second challenge was just, it's fundamentally, because we have this idea that we want it to be as affordable as possible, that means maintaining a very, especially back then, actually, a very low margin. It means you're bleeding even more. I think we could have bootstrapped it, I guess, is what I would say. But the reality is we saw an opportunity. You get very few opportunities, I think, in. And especially if you can take an opportunity to build a venture backed company and you actually are going to work your ass off to make it succeed, it's probably the best thing you can do from a, from an individual career point of view, because the reward is the financial reward is so outsized. It's how you create wealth. Right.

Um, but beyond that, like I'm very ambitious in the, in the sense of like the driving factor for me at century is not, I don't care about technology. I never have. I don't care about, um, the academic correctness. There's a lot of things that motivate people. Mine was always like, I wanted to build the best thing that existed. And I wanted to sort of prove a point that it's not that hard to do a bunch of things. And for me, that was always just like, yeah, I can ship this random thing that I don't

understand back in the day. And now it's like, I can build a bigger business than all of the competitors. And even today, that's still the motivating factor for me at the company. And it doesn't mean we sacrifice on all these other things, but you kind of have to have that thing that drives you, especially this late into the game. Yeah, for sure. I don't know. It's a fun situation, and certainly Century's done well,

which is the good version of the story. Makes it more fun. Yeah, yeah. It would be a lot less fun if we were struggling to grow or to reach the stage we're at now. But yeah, it's very rewarding in that regard. But yeah, just seize the opportunity. Yep. You mentioned early on that you tell people, hey, go work on what you want because you

sort of need some luck around the way. Do you have one or two or three instances of like, man, that made a huge difference for us or was lucky or inflection points or anything like that? Yeah, I think there's a couple things we recognize. So because our goal is market share, we're always like, we're the most developers, right? And that's actually a tricky equation because I have to explain this all new hires. It's like, well, aren't the most developers like Java developers or something? And we actually deprioritize things like Java and.NET because they're generally not the kinds of software teams

that are looking for new solutions or looking for new technology, right? On the counter side, JavaScript, which actually is where most developers are these days, is the exact polar opposite. They're like, let's adopt a new thing every six months. It doesn't matter if it's stable or good or bad, you know? And so we recognized that early on, not in the exact same context that we see it now today, because, you know? And so we recognized that early on, not in the exact same context that we see it

now today, because, you know, hindsight. But we knew that there were a lot of JavaScript developers. So we went after that market. We're like, let's make our technology work there really well, make that basically the P0 of the company. And so that was really good. I think there were other things we figured out along the way that were just better businesses, because, you know, centuries old, and not everything we believe in now was kind of black and white back then so another good example is even today actually but but more so back then there was this um opposition to cloud services so cloud versus on-prem software right and century was open source um stills free self-host right i did not want to build a company that

just provided support and services for like an open source technology it's just and and that's the right business move for what it's worth nobody should be doing that if you want to start a business these days great way to support like a lifestyle thing but it's it's not it's not rewarding you don't get to build a product you build support you know you build logistics and so i was very anti doing anything around that and so and and that turned out to be the right move uh but it was not it was not black and white back then because this is like where get labs up and coming and there's still this unclear future of will like everything go to cloud services or is it going to be you know a little different and um and so that i think that was like a pivotal

thing it also allowed us to focus more on the product development versus the business and then we'll we'll see how this works out. Find me in 10 years. We'll see what the company becomes. But I think strategically, at least for Century, the open source, the self-hosting, the low price point, the market share, emerging technology teams focus. When we fundraised, it was like $300 ACV or something. Just the worst enterprise business you've ever seen in your life, right? But we had this slide.

And these were all paid logos. It was this paid logo slide. And you had to make them so small to fit them on there. But you recognized every single logo. And that was in our investment deck. And then every future investment, that slide just got better and better with even more iconic brands. And the ACV got better and better, too, iconic brands. And ACV got better and better too, to be fair. That's hard to create.

And that means, I tell people this at the company because the benefit I have is I don't actually care about immediate business results or anything. But I always tell people the best thing we can do as Century is focus more on net adoption, less on revenue, like in the near term. Like revenue will come, we'll build good products, we'll make sure we extract the value. But like the strategic

value of century is the fact that we have something like, I don't know what it is these days, but like on the paid cloud, it's something like 65,000 logos and like very few companies ever achieved something like that. And I'm like, that is the value of this business. It's not the growth curve or the revenue or anything like that. And that is, that is is exciting to me because that's also a fun kind of business to build. It's really different. If you're an engineer, you really think through problems all the time. And it's not playbooks, right? I guess because if there are playbooks, people insist you must use the playbook.

And in this case, there is no real playbook. And so it's like, well, you got to argue it out and figure it out. And I don't know. It's quite interesting. I actually like to think if I ever did another company, I would try to be even more, maybe not controversial, but like unique in the approach of like, well, what else could you push the boundaries on? What else could you first principles think about how you would approach your problem? Because sales and marketing, which I do a lot more marketing type stuff these days. I didn't

know anything about this 10 years ago. So it's quite... You picked it up. In that regard, it's fulfilling. Yep, for sure. Okay, you've said a few things now. You're very ambitious. If I do another company, or you also said, check back on me in 10 years on Sentry. Will Sentry be your last job? Do you think you'll have another company at some point in year? Or where are you at with that? I think I'll do something else after this. It's just, I don't know. I'm off for two months on my first break ever. And I'm like, fuck, I don't know.

What am I going to do? Maybe I'll go build something, right? Like, it's always like I want to work on something. And I feel like a lot of people, not everybody, but a lot of people get like satisfaction in life. Like, I actually think of this as like work-life balance. I actually enjoy the things I work on, you know? And that's the way I balance the sort of stress of also those same things I work on. And so I don't know if I would ever start another like big company venture business or if I would ever start another company.

But I think it would be rewarding to always be doing something for quite a long period in life. Grant, I don't have kids yet. And so we'll see if that that equation changes when I have kids. But I don't think it will. And so we'll see. You know, you can never predict where things go. And I didn't think I'd be doing century 10 years ago, you know, 10 years later. I don't know if I'll be doing it in 10 years.

I don't know if century will exist in 10 years. Right. But I'm sure it will. It's, it's hard to lose, you know, 65,000 logos, even, even over 10 years. So I assume century will still be around, but yeah, I agree with you on that work-life balancing. Like, I think like for a lot of people, myself included, it sounds like you, like, you you know life is just more fulfilling when you have good hard work like throw yourself into and that that helps balance the rest of your life as well so yeah cool okay i want to talk about century architecture and tech stuff because like you're saying hey on your website actually says 100 000 organizations 4 million developers numbers that jumped out to me is 790 billion

events you process a month, which if you spread that evenly is 300,000 per second. And I'm sure it's not spread evenly. So some like big tech challenges there, I guess maybe walk me through the architecture of Sentry. You can start with the original architecture if you want and say how it's evolved or I guess, yeah, wherever you want to go there. Yeah. So high level, the way to think about Sentry is we collect a bunch of events. Those events might be errors. We do more than just errors these days, but we'll just simplify it with just errors for the sake of this. So we collect an error.

We have these SDKs in all these languages. So it's all client-side, runs inside the application, which has a number of challenges because agent-based versus in-app-based, usually network I.O. is a problem. I need to submit this request to a server. It needs to wait on it. Fortunately, with errors, that's less of a problem because you hit an error path. It's already slow.

Performance hit is irrelevant at that point. So we're able to get away with a lot of stuff there. But generally speaking, some kind of client telemetry pushes to a server. The server is actually very complex these days. But the simple version is, imagine just a simple API service. You post to the endpoint. It writes that thing to a database or a queue. It's been a imagine just a simple API service. You post to the endpoint. It writes that thing to a database or a queue. It's been a queue for a very long time, some sort of queue at least,

because you need to keep that low latency, right? And I'll go a little bit deeper, but we'll start with high level. And then from there, we basically batch everything. And so Sentry's core, even still today, a lot of our stuff is in SQL. But the original version of Sentry was all SQL. And then we would patch little parts here and here as they needed scale. But it was all SQL. And so every event that came in wrote to a row in the database,

which once upon a time was awful because these events are not small events. It's not a log. It is a 50K minimum, 50 kilobyte minimum payload of error data, stack trace, context, all this stuff, right? And so we're like, geez, zipping that into a Postgres database. Probably even, actually 10 years ago, we might have already changed this. But the original thing was just like,

load this up in a Postgres database, which has a lot of negative consequences, by the way. But we also ran a job that would, because you have counters. And so the fundamental thing that sentry does with errors is we take all these unique errors and we dedupe them we we try to like fingerprint them use some real rule-based heuristics that are now becoming a little bit ml plus rule-based heuristics but so we try to dedupe those we write a row to the database for the event um we write a row for what we call the issue which is the aggregate at the end of the day. And what we'd actually do is we'd sample the events.

So we wouldn't store that many of them back in the day, but we wanted an accurate representation. So we'd actually try to roll these up into counters. And a fun challenge, if you're in infrastructure, is do all this with Postgres. It's not actually that hard, but it actually makes you, I guess, use your brain. It's like, well, how do you deal with concurrency

and write locks and all these other things? It's a pretty straightforward problem, to be honest with you, I guess, use your brain. It's like, well, how do you deal with like concurrency and, and write locks and all these other things? It's like a, it's a pretty straightforward problem, to be honest with you. But I would imagine most people can't figure out how to solve it in 2024. But like, it's a very simple architecture back in the day. And then it's just a dashboard of all these things, a bunch of notifications. And that was a simple version. And what's interesting is the core flow of data is almost identical today. There's some more nuance and there's a lot of things that we put in place to scale it and to offset issues or improve performance or something like this. But the core of it is still roughly the same. The biggest changes we've made over time is client collection is basically still the same.

But now what we do is we run an aggregation service. It's an unfortunately very complicated service. But what it does is you can run it like an agent locally on your server. We also run it on edges throughout the, like, global edges. And then we use geo DNS routing. I can't even remember what it's called these days. But basically, like, you're a customer visiting, I don't know visiting some website in Japan. We have a local pop, I think, in Japan.

The routes of this aggregator. And then it basically repeats. It's like, reduce if you can. I don't actually think we reduce that much these days because it turns out to not be effective. But reduce the data points there. Dedupe if you can there. Pass it on to the next one. And so it might be, first one's inside somebody's firewall

for a variety of reasons. Next one's our edges, our points of presence. And then the next one is like our core infrastructure, say in Iowa or wherever the hell we host. And then from there, so that part's changed, but they're all these like segmented. So in like, let's say that pop in Japan, is it writing to the queue in Japan only

and then returning to the client? Or is that pop forwarding that data actually to a centralized location? Yeah, so it writes and then lets go as fast as possible because the goal is to release the connection back as quickly as humanly possible. So I believe all it does right now is it checks rate limits, which is like a globally distributed thing, which is a little bit tricky. It checks rate limits to make sure you're not over quota and also to implement backoffs and things like that. And then it normalizes the data, which basically is validation to some degree,

and it returns a status code. And that's it. So it doesn't actually check a lot of things. It's very basic because it has to be fast. A lot of it's in memory. And then it will forward it from there asynchronously. And that's just like some rust service we built that uses i think it also unfortunately couples the redis these days not i love redis for what

it's worth but so it's not unfortunate that it is redis it's unfortunate that it requires a stateful store that's probably more distributed um but yeah so we call that like the ingestion layer which is just one part of it and that's you know we kind of went from here's an api endpoint that it does that to here's a service that just does that to like box it out as a separate service. And that's a bunch of our infrastructure these days, because one of the challenges with Sentry is it has to be able to handle like 10,000x like throughput at any given moment. I'm not saying we do that these days. It actually, the economy of scale helps here. But back in the day, you'd have a big customer like GitHub,

and they'd have an outage. And it cascades to us because we just couldn't handle the massive increase in error volume. And you have one of those customers, and obviously it's a significant increase, but you have hundreds of those customers. And actually, you're provisioned much better over time. And so that's the good side of cloud services.

So that's the ingestion layer. And is that all just one know, the good side of cloud services. But so that's the ingestion layer. And then from there. And is that all like just like one giant multi-tenant app? Are there like any segment, like are you segmenting tenants into like, you know, shards or clusters or anything like that? Or how does that look? Yeah, so it's gotten very complex these days because data residency and all these other concerns, right? And so we have, I would say, two primary versions of that because Sentry sells two services.

We sell a multi-tenant that has, to be fair, different data residency regions. It's a little bit more nuanced than just multi-tenant, but it's effectively multi-tenant. We can almost share everything in some layers. Then we sell what we call single-tenant, which is like you're a big media streaming company. You need everything to be logically isolated. And so we have to run the same thing. And that one actually is like painful because if you are a small customer, we can't

afford to run it. It just, it doesn't make sense because again, we need that capacity to be able to scale it out and we'd have to charge you through the roof for it. And so we've actually, we used to, we were questioning if that was ever going to be a good thing to invest in that single tenant concept. And for one customer of ours, it's great. They pay us a lot of money. They have a huge volume of data. It's worked itself out. For every single other customer so far, it's been a terrible investment.

And so we've been investing a lot in complexity. We call it hybrid cloud to basically split off data residency on multi-tenant so we can actually isolate that event, that error, all the way through one region or one zone or something like this. Even that single tenant, that's still running all on your infrastructure. It's not like you're running Sentry on there. Yeah, everything's on our infrastructure.

And so the single tenant basically is like, we took our SaaS and we're like, let's run the entire SaaS for one customer, which abstractly seems like, oh, that's an easy thing to do. Practically speaking, you've scaled up this global multi-tenant situation. It doesn't scale down that easy. You're basically proprietary art, in this case for us, is just how we've scaled it up. It's five years in now.

We're still working on those projects. Even when we launched, we split US into two regions. We did US and EU, which is Germany, where it's like, we'll store all the data there. We will route the data through there. So we'll skip US effectively or whatever. We'll store the metadata wherever we want to. It doesn't matter.

Like basically, like say, I'm going to use a hypothetical, but like say you're Uber and I don't know what, pick a company that's in Germany, Lufthansa airline that I know. I'm like, they're probably mostly German. So they probably care about this. It's like, if you're Lufthansa, what you care about is your customer's data, right? Not so much your data in the sense of like where your employee information lives doesn't actually matter as much or your projects live or whatever. It doesn't matter as much, but where the errors of your customers live matters a lot for like gdpr and all these other compliance things so we we kind of separate those into like here's the storage layer which is

like the customer data the events and whatnot and here's the century layer which is like the metadata that allows the whole system to operate and and for the most part metadata actually we distribute everywhere but it doesn't matter where it lives and that storage layer must only live in certain regions and so i would say the infrastructure has gotten to the point where it's such a complicated beast that no single person knows how it works anymore which is not great by the way you should do everything in your power to like minimize complexity forever you know and reduce complexity when you can so yep do you think it's all sort of necessary complexity or you're like man we made some mistakes

back in 2018 that I wish we could undo or something like that? It's both. I think some of it's necessary. I think the way software is like this wave where it's like, we invest in a bunch, it's kind of a disaster, but we got the shit done, now we need to go back and fix it.

Or you don't know what you don't know kind of thing. When we did single tenant, it was just clone SaaS. It kind of worked, but it was not a scalable thing. It was not a repeatable thing. And then we did eight more of these, and they were not a repeatable thing. They were not remotely scalable. And now we're like, hang on, hang on, hang on. We got to dial this back. We got to now invest the time to do it a little bit more correctly, a little bit more correctly. And it's the same thing for our overall hybrid cloud investment. That's just an extension of that early investment of multi-safs.

And now it's like, okay, how do we make it cheaper, less complex? It's the same thing for old projects. You have a giant monorepo with a bunch of tightly coupled services. Okay, that scales for a very long period of time. Now you need to break things out. And so I just think that's software. And it's totally fine. That's the value of it, you know?

Yeah. In terms of cloud, what are you all using for cloud? Are you using multiple different ones or just one? We're primarily on GCP. When we switched, we did a soft layer, which doesn't really exist anymore. I think it's owned by IBM. It might be called something else.

But we started on just like hosting hardware, dedicated servers, right? And then we had an okay deal, but it was kind of a pain in the ass. They weren't the most reputable. Well, let's just say once in a while they would trip over power cables, and that's not a good situation to be in. That's fine. That's the price you pay for that kind of thing.

And we switched to GCP pretty early days. And I think it was primarily because GCP was simpler back then. There was not 800 AWS services. And I think it was simpler back then. There was not 800 AWS services. And I think it was like monetarily better and their performance was better. Compute at the time and network like was unbelievable. And I don't know how it benchmarks these days, but that must be eight-ish years ago

that we did that migration. And so I don't know if we'll be dcp forever but yep and did did you do actual testing to be like hey check out the compute in the network performance of these different clouds before making that switch we did at we looked abstractly at the compute that was available in the pricing because they all offer roughly the same services but some might or like at the time i think google had um better chips or better cpus i forget exactly what it was. We did do basic testing on the network and Google's network was like so phenomenal in the point of where like,

it was a point of pride that centuries latency request hits our, our load balancer to responses serve from that ingestion endpoint was like 30 milliseconds. That's like a Python application, 30 milliseconds. And I was, I was very proud of that back in the day, wants to serve from that ingestion endpoint was like 30 milliseconds. That's like a Python application, 30 milliseconds. And I was very proud of that back in the day.

Switching to Google with no other changes dropped it to 20 milliseconds. Wow. And that was just like internal networking. And external networking, I don't actually remember what the numbers were, but I think we saw like a 40% dip or something, just because they have phenomenal networks, right? I'm sure AWS has caught up in all this, but also everything's even more complex with both providers these days. Yeah, interesting. And so all those different pops that you have around, are those all in Google data centers or

do you use Cloudflare or anything like EdgeLac or anything like that? Yeah, I think it's all Google data centers primarily because it's just easier for us, which the devil's in the details here because the problem with all of this stuff and something I didn't think about when we did this is the pricing variability of all these regions is quite drastic. Like, if you look at, I think Japan is one of the more expensive ones. And I think it's something like, I'm winging it with this number, but I remember it being significantly more. I'm going to say it's like four or five times more than, say, U.S., just for computer, network, all the things. And I'm like, hmm that's uh good to know

you know it's it's not a great situation but um that'll probably not be the case forever like we we designed sentries hardware footprint to not ever be vendor lock-in now there's some nuance to that over time like we actually do use some google proprietary services like uh um big query and and big table and and stuff but for the most part, the core application runs on a set of semantics. So it's like, you need Postgres. It doesn't matter from there. It's got to be Postgres though. You need a key value store. You need something that looks like Redis, but particularly, actually we use a lot of features of Redis, but some of that's changing. And then you need ClickHouse, which

yeah, everybody has a sort of, it kind of looks like ClickHouse, but then you need ClickHouse, which, yeah, everybody has a sort of, it kind of looks like ClickHouse, but isn't actually ClickHouse. But we basically, we run, for the most part, our own services on top of their compute. And the only thing we use from vendors, which is a great thing, is every cloud provider has these these days,

is like a file storage service, key value service, maybe a load balancer, but we actually might, I don't know if we might still run our own, I'm not sure, because we run a bunch of these layers of basically DOS protection. That's all like in-house. So yeah. Interesting. Yeah. And then, okay. So for core data storage, like when, when an error

comes in, is that going to Postgres still, or is that going to like ClickHouse now, or is it going to both? Like where, where is one of those things going? So what happens? Hits our endpoints, validation. All this stuff is done. We write it to Kafka. Another example of a technology that has not changed a lot in like a decade.

But it is good software to be fair, other than the Zookeeper bits going on. Right into Kafka, we batch that stuff. And then I don't actually remember, because we've been changing a lot of this in recent history. What used to happen is we would write this stuff to RabbitMQ. We'd persist it. So the great story about Century, this is really important, is know the constraints of your business. Century's constraints was it's okay to lose an error. Nobody cares. Practically speaking, nobody cares. There's like one in 10,000 customers that do,

but who cares? Don't build for them. And so a lot of stuff meant it was ephemeral. It was like, if we had to, we could drop data. Um, but so we'd write it to Redis. And we did this particularly because the network overhead of writing to like RabbitMQ because there were really large payloads was problematic. Redis isn't like the best for it. Like Kafka is definitely a better solution, but Kafka implementation of something like Kafka is a lot more operationally complex. So we, it was several years before we did that. But we'd write it to Redis,

and then we would actually increment a bunch of counters in Redis, which is still phenomenal for this kind of thing, especially if you can tolerate data loss. And then we'd run these batch jobs, which would just take all those counters and reduce them and write them to a SQL database, basically to remove all write concurrency. And I think most of that is gone these days,

but that lasted for, I mean, that got Century tens of millions in revenue, like this duct tape architecture. These days, it writes to Kafka, we batch that, we write it into a bunch of data stores, but ClickHouse being a primary store for sort of our event log, we write it to Key Value Store, which is like the big JSON blob of the event. And then we're actually removing a lot of the counters.

I think counters, I don't know where they were, but either way, they're kind of getting removed in lieu of just doing a column store and running real-time aggregations on top of it. But generally speaking, the primary storage is like SQL stores a ton of metadata still today. I think every aggregated issue in Sentry is still a row in a SQL database. And actually, I think it has its own SQL cluster these days just for that one table because it's so high throughput and so large. Because you're basically, you can only scale memory so far, and the rule

of thumb with SQL is memory is your limitation. But then ClickHouse has been phenomenal for us. And so that was a big game changer when we migrated to that. Interesting, yeah. So, okay. And are you running your own ClickHouse? Does GCP have its ClickHouse thing or what's it? I think, I don't remember which service it is. One of their services looks a little like ClickHouse, but it does not have the same performance characteristics. And so we run ClickHouse on top of GCP. I think we run it on native compute, but I know we're trying to move

it to Kubernetes. I don't know if that's a good idea or bad. We'll see. But yeah, we pretty much run everything. I actually don't think we use any managed services unless they're proprietary services. So like Bigtable, for example. Yeah. Okay. What's your sort of, I guess, happiness with the cloud right now or any wish lists or requests

for changes or services or anything like that? My only complaint about the cloud is it's gotten way too expensive. And they don't actually, they don't progressively discount. And so a good example is, we're going through this with Google right now, and we refuse to come to agreement on a longer term contract. But like the way they discount is they discount the hardware you don't want. That's also like ancient. And so we want high performance hardware because it allows us to scale much more simply, right? Like the more you can pack on a machine, the simpler the architecture becomes, things like that. And so my biggest issue with the cloud is like, it's just like the overhead, the cost overhead is like, it is extraordinarily high.

It's like a meme to talk about it these days. It's fine to fucking use the cloud. Just charge people money. Who cares? But the operational overhead from people and from just like this total cost is is quite drastic um and then i think there's like just multiply that out if you have to go multi-cloud which we haven't had to yet we might in the future but um otherwise like what would cause you to go multi-cloud there's a bunch of teachers so okay yeah so it might be we've

considered offering like a vpc specific thing so where we could run sentry for you on your own vpc i don't know if we'll actually have to do that it's again we're trying to not do on-prem that's a little bit closer um the other is like from a disaster scenario it actually might be better in a lot of ways to be able to run multi-cloud i don't think it matters that much. The other is from a pure financial scenario. That's probably the more likely scenario. You're like, you know what? Over a 10-year horizon, we're going to save hundreds of millions of dollars if we do this thing. That would probably be the forcing function for us because it is a lot of complexity to do that.

But otherwise, I think the only other thing would be opportunistic if we could find some really good value-add that used multi-cloud. Yeah. So you mentioned pricing, and it seems like there aren't pricing wars like there were 10 years ago in the cloud. Do you think we've hit a steady state where that just isn't a factor given all the other stuff now? I don't know, actually. It's really hard to say what's going to happen because I think all the cloud providers are struggling this year in particular, and that's not going to go away. And it's like, you've saturated the market.

It's like Apple. Apple's like, wait, can we make more money on iPhones? They're like, wait, maybe we can't. Maybe we've peaked. What else can we do? And I think cloud providers, infrastructure was their second thing, like these managed services. But it costs even more. You're not really getting this sort of – because if you racked a bunch of service, it's going to be night and day cheaper. And there's a scale where everything at that

point is cheaper to do it yourself. And there's an argument of like, should you do it? Or should you just make more money and pay the money? But at some point, if like the cost of the team to manage it and everything else just adds up, it's a dollars and cents thing. And we're not quite there yet. Certainly, at least it's not our focus to cost cut yet. But you can see the fact that there's so many of these companies that exist just to help you figure out and fix your bill on cloud services. It's kind of telling. And I don't know.

I don't know what that means. Like over time, it might be we just keep paying overhead after overhead on like abstractions, you know, or they have to become competitive once again. I don't know. It's really hard to say because it's a cultural thing, if anything, you know. Yep. Yep. Absolutely.

Okay. You've talked about the system stuff and the difficulty of the data ingestion. You also mentioned earlier, just like, hey, we have to maintain SDKs for all these different languages, different frameworks, things like that. What I guess, like, what's a harder challenge? Is it the infrastructure stuff or is it just like maintaining 30 different SDKsks i don't know it's they're all hard i will tell you so figuring out how to build the sdk so like we need to collect this data whatever actually dumb simple like you can be a competent engineer maybe not like a you got to be pretty competent

let's just say in a language or a framework you've never used before and you can fucking figure it out it's not that hard. The hard part is actually the nuances of each framework or the culture of a language. It's like, this is how people write code. This is the syntax sugar, like all these little things. So that's hard. Call it developer experience, right?

That said, it is also a lot of SDKs, and especially in JavaScript, it is brutal. I joke about this, but I mean it fairly seriously. It's like JavaScript is beta software nonstop. And it's always like some beta alpha thing that people are just adopting. You're like, we got to support this. We have to funnel a bunch of engineering into it. It probably won't exist in six months or people will not actually be using it in six months.

But you have to make every single bet. The only way we went into space is by betting on every single technology, no matter if it's academically bad or whatever. And so that is not without its challenges, right? way we went into space is by like betting on every single technology no matter if it's academically bad or whatever right and so that that is not without its challenges right if nothing else from resourcing because like we're bigger we have a lot of customers sdks can never break uh we don't always nail that we've had some issues lately but um so it's it is it is hard i wouldn't say it's the harder problem to be fair because our infrastructure is very complicated. We actually have this challenge now where probably in the last like three years, I don't know what our, I would have to look it up.

I don't have it on hand, but like our scale of data, let's just say it's two X right. And that means infrastructure should sub two X because of the economies of scale. I think our infrastructure has three X. And so there's some challenges there. And at the scale we have, I forget the stats and we don't run like the largest infrastructure of all time, but it's not small. And somebody was giving me the container stats the other day. And the worst version of the container stats, because it's all Kubernetes, which I have strong opinions about,

but the worst was there were something like 800 iterations, permutations of the Sentry application. Basically, the unique config of that Sentry image running on a container. I'm like, 800 permutations of our monolith. And there are a few other services, granted, that's most of them, but Sentry is not 800 permutations of complexity. It's just like, why is it so like that, right? And so it's gotten to be this point where complexity is literally the fundamental issue. And so if you look at an SDK, it's not actually that complex. You can reason about everything.

You can fit it in your head. Our infrastructure is very, very hard to reason about. And it's to the point where we've sat down and we're like, we're going to make a drastic investment to change this. Exactly what that investment is, we don't know. But it's got to be different. And I think part of this is where I'd go on the Kubernetes rant is, I don't know what's wrong with all of us, but we have this culture of adopting complexity in every single situation we can.

It's like, oh, this cool, new, fascinating thing. Yeah, that must be the right idea. I am the polar opposite of that. I'm like, how do I just stitch together the dumbest technology and make it work in the simplest way possible, right? This is why SQL was the only thing we used in the early days because it's only one thing to run. And it's just, i don't know you look at kubernetes and it's like there's a bunch of layers like on on one pod and and it's really hard to unwind it's really hard to figure out how to optimize even because like we should have seen um we use we use a big uh another monitoring vendor that's not us because we don't do everything right um i'll avoid shaming them i do it publicly

elsewhere but the bill is like through the roof all the time it's just we use it only i know who it is yeah you know who it is everybody knows who it is it's only metrics for us and metrics are very very important for running our infrastructure our platform right and because we'd have all these runaway things and it's so expensive and i'm not even necessarily blaming them half of it's just complexity for you know in the industry but we had to put a layer in front of that SaaS provider to try to limit cardinality and all these other things that explode the bill. And then that layer costs us more than the bill costs. And I'm like, come on, people. We got to figure this out. If the point is to save money, it's got to actually end up cheaper

at the end of the day. And I think it's just, again, it's so complex, it's hard to reason about. And I don't know, it's one of those things where I think Kubernetes has value as an example of a technology that's awfully complicated, but I think we use things wrong. I think the right abstractions were not created, but everybody adopted them anyways. It's kind of like the JavaScript story, where a lot of this reigns true over there in the technology stack. I don't know what it's going to look like, but something has to change in the technology stack. And so I don't know what it's going to look like, but something has to change in the industry. Yeah, yeah. Interesting.

Okay. I want to switch into open source. And first I want to talk about open source as it relates to, you know, Sentry itself is open source. You can run it yourself. But as you're saying, it's also getting more complex over time just because of the demands of your business. How do you sort of balance that

where like if someone wants to pick up and run Sentry themselves now, do they have to run? click house and Redis and you know key value and sequel or what's that look like? Yeah, it's honestly awful We actually have an investment to simplify it, but the problem is so what's it's hard. It's hard. It's hard It is hard. Yeah, like you are a different business now. Yeah. Yeah, and it's I mean, there's no there's no comparison Like it used to be easy. It's very much not easy. It uses a lot more resources, even for the exact same use case. And so what we decided was we need to find a way, like kind of the classic example is here is you need to go more towards a service oriented architecture, right? You need to be able to split concerns. And when you think about Sentry as a product, there's a lot more it does, but not everybody needs all that stuff. And that's fine. So maybe, answer is like, give error monitoring, keep it somewhat simple. It's going to require a little bit different of a tech stack than it used to for a variety of reasons. And that's okay, because you

can abstract a lot of that. But the fact is, it requires so many services. And a lot of them are for like functionality you might not want or need, right? And that's a problem. And so we sort of have that same problem for what it's worth is why our infrastructure is so complex and expensive. And it doesn't have to be that way. And so a really good example of this is Kafka is already expensive, but we partitioned every consumer across Kafka. And we have a lot of consumers for these different batch jobs. When you run,

you know, this self-hosted thing, maybe you only need one, maybe one is fine. Right. And you can simplify that whole thing. Or I think the better paradigm for what it's worth, and this is how century used to be pros and cons is you create a service abstraction layer, basically an interface. And so my analogy here is, um,

old school, uh, century, we had a key value, right? The key value was in Postgres. It wasn't abstracted, but I abstracted the key value. This is just for Jason blobs. And I made it just literally just a class in Python. It's like, there's just a class in python it's like there's a git there's a set there's a delete right and everything calls that it it deals with it for a while i just wrote it to the same row in the database or whatever right straight forward we switched to react which doesn't really exist anymore r-i-a-k which was a distributed key value store um pretty good at the key value part everything else it tried to do less less good um but at least for the time it it was one of the more operationally simple things to run. All I did was write a service adapter for that. You can plug it in. It's basically

DI at that point. And it just worked. And we did that for Cassandra. We did it for a bunch of things. Now, that to me is the right solution for that problem because you can scale it down, scale it up. You just got to swap tech. The counterpoint is, we had a Cassandra adapter, we didn't use Cassandra. Might have worked, community contributed, not well tested, you know? And so at some point we're like, okay, there's a time and a place for adapters. We don't need that many of them, right? But the interface is probably the important abstraction.

Now the challenge is like that, that's easy with a key value. It's like the simplest interface of all time. It's much harder for a lot of other things. And so SQL, because we use Django and it has different adapters, so we used to use it, but then all of a sudden you had Postgres-specific code to optimize performance paths. MSSQL, which I assume still exists, never worked correctly. SQLite was definitely not something you should run in prod,

but it kind of worked with all those. But that was such a problem at the point where people would use MySQL and it wouldn't scale that we're like, you know what, we're only supporting Postgres going forward. It's fine. Just run Postgres. It's not that hard. So I don't know what the solution is at our scale, but the high level is like, okay, let's disconnect the product enough that you could run an errors only version of it. And so you can get that up and running and that's fine. Like, I actually think if we, if we can get to that where it's like much less resource intensive, much less complex. So if something goes wrong, you can actually fix it. That's a good enough outcome. Yep. Yep. Do you talk to many people that run century themselves and like, do you know people

like that? Like, yeah, I guess like what's your interaction with those, those type of people? It's honestly less these days because our growth of actually literally for the last decade, And do you know people like that? Yeah, I guess, what's your interaction with those type of people? It's honestly less these days because our growth of, actually, literally for the last decade, self-hosting kind of plateaued. And I don't think that's just because it got more complex and stuff. Certainly in a period of time, that's part of the reason. But it's mostly just people will buy cloud services. People don't actually care. And Sentry's not expensive compared to, yeah.

Yeah, 100%. Yeah, yeah, yeah. And so it's like, because it kind of crawled, we just saw less and less big companies using it. And I optimize for the many. So I'm like, if I have 10 customers doing this, it's not important for me. It matters, of course,

but it's not where I'm going to prioritize. And so it used to be that all the big companies were probably self-hosted. Now, generally speaking, it's the big companies that either refuse to buy SaaS, but probably will in the future, or we can't sell to. So I don't know if they still use it, but Yandex was a good example of a company that did use self-hosted. Even before the war and everything, we were not going to sell to a company in Russia. It wouldn't make a lot of sense. It's complicated.

Latency would have been awful for them. All these little things, right? And so it was often a lot of those kinds of scenarios. China's another good example where we don't operate. And so for that use case, it matters a lot. Or if you're like a government agency that needs FedRAMP, we don't do FedRAMP. Great. You self-host it. That'd be awesome. But now it's like, you kind of got to be a bigger company to justify it, or you got to be a pretty capable independent that's like, whatever, I can spin up the VPS and do all the random shit. It's okay if it breaks. I like that stuff. Yeah, yeah, for sure. Okay. And on that open source note, let's talk about open source pledge, which I guess, is that directly affiliated with Sentry? Is that like your project? Like what's the affiliation there? What is it generally? All right. So the abstract version is I believe in brand association

as marketing. I believe in brand marketing. And so Sentry or me, but Sentry, we'll pretend it's Sentry pushing this. We're like, we can invest in a bunch of things that are good for the world or good for whatever that are not about error monitoring, which is very boring. And we will gain brand association off of it. There's usually multiple angles, but like from the business point of view, that's how we think about it. Pledge is one of those things. And so there's actually three kind of core things that live in that umbrella. So there's syntax, which is this podcast, this front end podcast we have, right? Has nothing to do with Sentry,

right? Whatever, overlapping audience, but that's it. We don't sell century, even though they do advertise century, that's their choice. We told them they didn't have to, to be clear. Oh, really? Okay. Yeah. I love century. I listen to century all the time. It's a great podcast. Yeah. Um, but that's because we believe we offer value. You'll figure out what we do. And then if our product offers value, you'll purchase it. Right. it right that that's the fundamental uh math that we put behind this and so open source stuff is similar so we did two things in open source one because century century's core product is not open source anymore it's um it's proprietary license we made our own recently but the idea is like you can self-host it you just can't be say amazon or somebody like an amazon and sell a cloud service

that uses our technology right um we'll see how it goes. But so far, that's worked really well for the business. To be clear, because people always get confused about this, Amazon was not the concern. There was just other more product cloud services that were going to try to monetize it, not big infrastructure cloud vendors. So we made that license. We invested in it. We put that out there. Whatever. That's for us. But we put it out there not just because we didn't just be like, we're changing our own license and that's it. We're like, hey, there's some other people using worse versions of source-available license. You should consider a better version like this. And so the TLDR is, after two years

of code being public on GitHub, it is Apache license in the Sentry project. So the majority of our source code is actually Apache license. To the point where i could so i could run two-year-old century and offer it up as a product yeah yeah wow okay but most importantly it's like free so you can self-host it it's the the cool thing is it's the same exact product that we commercialize and that as a user who yes i believe in open source and all this but as a user i just want to get shit done i don't want the worst version of the software because usually there's like one fucking feature that i actually need as a hobbyist. And I don't get it all of a

sudden. And then it's like 200 bucks a month to get it. And I'm like, I can't afford 200 bucks a month for my hobby project, you know? Um, anyway, so, so I don't want to go off on that, but like, that's a thing, uh, that minor, right. More core to us. But then we're like, okay, we're tired of this argument in the industry about, you know, it's either open source or it's not open source. Like a lot of users care about it's free and they can create value using the software. And so we built this thing called fair, fair, fair source, fair code. I think it's fair source. Um, it's complicated. We went through a bunch of naming iterations, but that was a bigger thing where we, we brought in people from externally and that was kind of

the gateway to pledge. And so while we're building this fair source concept, which is basically a middle ground between open source and source available, I would say, we had also started donating open source, kind of like as a marketing thing, but we just believed in it, like we should give back kind of thing. And we did it in a pretty significant way to the scale we were at. And so a year ago we gave half a million or something. And particularly, it's important because we gave it to random dependencies. We didn't just say like, hey, Node Foundation, here's 200 grand.

Like that's useful, sure. But I'm like, our JavaScript dependency tree is a little scary. So we started that and we were doing this fair source thing at the same time. We're like, hey, we could actually do this money giving thing also as a program and would justify it because like business ROI, it's marketing blah blah blah blah blah you know all these things uh we we do really believe in it for what's worth even our ceo who's a hired in ceo is like fully on board with this so he's like when we're like upping the budget this year he's like go bigger go bigger why not you know um and so we basically just turned that into a program and we

we spent a lot of money recruiting on it, resourcing it. So I'd say Pledge, we have built Pledge and we are funding, I'd say, getting it off the ground. Long term, both Pledge and Fair Source, we expect to abstract from our company. And so I don't know what that looks like. We'll figure it out. Because I think there's a mistake. And it's like premature optimization. There's a mistake in, say, going all the way to the end state day one.

So like foundation, community, blah, blah, blah, blah, blah. And so we're like, yeah, let's get it off the ground and then we'll separate it from the company. So how are you feeling about the state of open source right now? I know we've seen a lot of changes in the last 10, 15 years. Do you think it's in a good spot? Do you think it's in a rough spot? Is that why you started Pledge? How do you feel about it? I think it's complicated i think there's some things that look like they've always looked i think there's some things that are totally okay to happen um i do think there's a shift and i all i can look at is like the big

picture and say what's going on and it's interesting because if you talk to people everybody's got the skewed lens on it so like linux people think like the linux ecosystem is literally the only thing that matters in open source like javascript whatever blah that's not that's some other world you know um and so i think that ecosystem ignoring supply chain and sort of risks looks kind of the same as it has before like linux foundation is like immensely well funded you know uh so it's on them if they fuck it up. The supply chain thing, I don't think is an open-source problem, to be fair, in that context. That's not a problem

because it's open-source. Those are pretty well-maintained things. Now, there are a bunch of big problems. Curl is a go-to example where Curl's a pretty widely used project. Maybe it's not considered critical infrastructure or something. It's not the kernel, but a lot of people rely on it. If there were vulnerabilities or it's broken, that would be something. It's not the kernel. But a lot of people rely on it.

And if there were vulnerabilities or it was broken, that would be a problem for a lot of people. And so there are still things in there that it would be nice if people that spend basically more than their day job hours on projects could just support themselves doing that. So that's a truth that we believe in. I think the bigger issue is there's a lot of kind of fragmented software out there now that's open source and we all rely on it we're building projects there's a lot of like 20 line javascript dependencies i'm not saying it should be it just is that's just the reality we live in right that stuff poses like an existential risk because we don't want that first off like i would rather just have like a standard library that's maintained even if it's a third party standard library

however there are some people that are outsized you There's the saying of 10x engineers. I fully believe in that shit. And you see it. I would love for the people that are just so outsized in their presence and their contributions to be able to fund their work. That's the hope. And this thing where, well, the rich get richer, like Linux Foundation gets richer. Some of the big JavaScript projects don't. And the only way they succeed is if they're a corporate project, like Meta has done phenomenal work here, right? And so I do think there's this gap

where anything that's not a big corporate project and that's not part of the Linux Foundation is much worse off from an ability to be funded or sustain it. And the problem is a lot of this infrastructure you come to rely on, the only out they have right now is commercialization, right? Either they don't maintain the project

or they commercialize it. And we've seen a lot more of the commercialization in recent history, right? Where it's like Redis or HashiCorp or I don't know what's going on with Elastic. That one's a little fuzzy. And so I think that's fine, right? Because we need this software to exist,

but I don't like that it comes at the cost of availability of the software, right? I don't care. I actually could care less about the four freedoms. I don't actually care about that. And that's fine if people do it. But what I do need is Redis or whatever that looks like Redis to always be free. And I'd love if they had a way to sustain the funding on that to develop it and all these other things. And there's going to be a variety of approaches.

Sentry's approach is one approach. It works great if you can do a SaaS service. But a lot of things, there's no way to actually commercialize them in any tangible way. Yeah, yeah. We're seeing that in the JavaScript ecosystem where a lot of great frameworks out there, but basically they're only out is to get sort of sponsored or acquired by one of the big

hosts or Shopify in the case of Remix or something like that. And it also makes me wonder what's going to happen with the whole TanStack ecosystem. I don't know what their funding is right now, but hey, they got some critical stuff that a lot of people are using. I believe he's still independent right now. I don't know what their funding is right now, but hey, they've got some critical stuff that a lot of people are using. I believe he's still independent right now. I don't know how much is public about that, but there are people that are helping sponsor Tansac. Tansac is actually a really good example. It's independent. All the people I've ever interacted with in that group,

amazing. You need companies like that or groups, I guess, like that or groups i guess like that ideal world they're non-profits or companies support them in a very tangible way and so we did this recently we hired uh ryan carneado solid js wacky ass experiment has nothing to do with our business uh and i'm like i'm like and the deal is like 80 of the time just go work on solid i don't care century actually gets nothing out of of ryan solid, but he's a really, really sharp guy. I think what he does for the community is really valuable. And then like 20% of the time, work on solid, but maybe somehow we can align like community events or something where Sentry's present and we pretend there's ROI in there.

But you know, there's not really, if we're honest with ourselves, that's an extreme, I would say, right? I think a better model is if you just suck it up and you fund, I think, I think rather, there's a lot of ways we can contribute back to open source, we should contribute back, like with our high resource engineering teams, we should open source more software, we should just give away money, you don't need something in return, we should hire people and let them work on it sometimes, you know, like things like that. So I think we should do all of these things. They're not mutually exclusive. And I am still, even though I'm on vacation right now, like we're like right now exploring, like, are there other high value technologies where we can maybe do a

50-50 split? But the problem with Solid is we don't use it, right? And okay, if it was something we use, maybe you could help us make it better. That's actually useful, I think, from the open source side too, because it betters the project. And other half the time just go work on it like maintain it make it great you know and i would love to see more of that kind of stuff out there yep yep i would too it's hard because like there's such a free writer problem like you know you're being a great steward on on this stuff but like a lot of people just aren't and you know and then it's like so are we it's weird to say this but like are we um getting less sort of free and open JavaScript than we should? Which is kind of crazy to think because there's so many out there.

But like, you know, if the funding problem, the sort of incentive matching problem could be fixed, it seems like there should be even more JavaScript, free JavaScript out there, which is kind of just wild to say. But yeah. It's also interesting because here's the bigger issue I see in the community. I don't, I've not tried to figure out what's going on. I community. I've not tried to figure out what's going on. I'm probably not even smart enough to figure out what's going on. But one thing I've noticed about JavaScript versus, I grew up in Python and whatnot, the community seems much more fragmented. And I don't know if that's because it's larger, but it's like there's so many frameworks

and they're pretty dang similar, right? And part of me thinks maybe that's because a lot of it's semi-commercialized, right? Like, so React came out of meta. I would not say React is the same as Next.js. Next.js is sort of from a company. And to be fair, I think all credit to Vercel. They don't actually, they try to do right by Next.js and then they try to commercialize their business, sustain it, et cetera, et cetera. Right.

But there's obviously an incentive for them to focus on that. And more importantly, for other people to not focus on it because it is sort of commercialized, if you will, right? And so ChanSec looks pretty similar, slightly different ideas, but independent, right? Okay, so I'll just, independent looks pretty dang similar, right? Okay, Angular is different, but it's Google. Ember, I don't know if it's, I think it still exists, but it's not really growing in adoption, different. And part of me is like, better example, Node, why are there like seven web frameworks that do the same thing? They all look like Express. It's like, couldn't we just make Express faster? Is it fundamentally impossible? And those are the things that bother me. I recognize the competition is good, but sometimes I feel like there's a little bit too much of it in,

in a JavaScript community where it would be a little bit better. There was some, some force that convinced people to work together versus just straight up like, Oh, I want to do it my way instead, you know? Yep.

Yep. Well, like you see sort of like in the PHP community, the way that like people have gathered around Laravel and symphony and like, you know, it's, it's at least just two web frameworks and,

and even they have like, I can't remember what they're called, but it's, it's basically like pips, know it's at least just two web frameworks and and even they have like i can't remember what they're called but it's it's basically like pips but it's not even pip it's like some like shared standards across libraries well they have composer for the packages but but so i don't know if this is still true laravel was built on symphony and so there's probably a lot of interoperability it might be it might be much less symphony these days i'm not sure this is a long time ago um but yeah there's less fragmentation yeah i was talking to someone and they have like uh pip like the python improvement proposal not quite like that but they say pep pep yes python enhancement okay pep um it's sort of

like that but it's not actually implemented in the language but it's just like hey if we're gonna have like a cache interface and here's the standard cache interface and now blarevel and symphony and all these different web frameworks like hey we implement the cache interface and it's like sort of gathered at the php level to have like a little bit of standards on on some of those things yeah that i i feel like there's probably something to learn from that or some approach like that the problem is it's a human problem it's not a technology problem and i i don't know and so it's it's tricky we'll see yeah yeah for sure last thing i want to talk about like marketing developers you say you do a lot of marketing you've been sort of all over the

the c-suite uh within century now but like what what works and what doesn't work with when marketing developers what doesn't work is pretty easy so and i don't even think it's just developers but like yeah i was gonna say do you think we're like i think all developers like i'm What doesn't work is pretty easy. And I don't even think it's just developers. Yeah, I was going to say, I think all developers are like, I'm impervious to marketing. Do you think all professions think that? What's interesting is if you just ask yourself, am I actually genuinely providing value? Not the bullshit version that people think it is. And it's sales and marketing, right? It's a pretty easy litmus test.

Syntax is marketing for us, right? Or a podcast or something, but it's not because it's a podcast about how to set up century. Who gives a shit? Like that is not interesting. And that's, that's valuable, but it's such a minuscule amount of value. And so you can't, you can't artificially do that, right? It's valuable because we give something away that provides value and maybe we get a return on it. Right. But if you just do that value assessment, things go a long ways. The problem is most people don't want to focus on that or they outsource it or something in a bad way. Like a great example is I'm sure I get more cold calls than, than an average person, but I get so much that I had to change my phone number that like in my inbox, a lot of stuff

goes to spam. That's not spam anymore because I like have to block everybody and nobody's done anything to solve it. It's only gotten worse over the years. There is zero value in any of these, right? Like nobody offers me any value. It's like, Hey, uh, I heard you're an executive. You want to get on a phone call? What do I get out of it? Like wasting my time? And I'm like, it's like the simplest test. And I don't blame, like, in that case, like the salesperson or whatever. I would encourage them to find a different career path because that stuff is gone. But I just think if you just look at it and like, hey, what would somebody appreciate?

Like, genuinely appreciate. Like, I'll give you an example. We did this campaign, which there's no good reason to do it. We're like, this is our CEO's idea. He's like, we have 100,000 logos on our cloud service. By the way, the 65 versus 100 is 100,000 total, which includes free. Oh, gotcha. But he's like, let's just give away $100,000 in merch for this 100,000 miles.

And I'm like, okay, whatever. Sounds like a wacky idea, but sure, let's go for it. And we gave away a bunch of stuff. It wasn't that interesting. It was like a scavenger hunt. But I'm like, okay, whatever. Sounds like a wacky idea, but sure, let's go for it. And we gave away a bunch of stuff. It wasn't that interesting. It was like a scavenger hunt. But I'm like, let's do something cool as part of this. And I know some of the JavaScript people like Ken Dodds, and I know they like OneWheels. And I always thought OneWheels were these cool things.

I don't have one. I'm like, what if we gave away a OneWheel? And people are like, what the fuck? Like, t-shirts, mugs, blah, blah, blah. I'm like, no, but people won't buy this for themselves. And we can brand it. It's fun. And they'll be like this is so cool you know it's something that's just different and you can imagine if like if you just happen to get one you

like it's a good story it's a good memory it's like memorabilia almost and it's part of it and i messaged like three guys uh kent joel hooks uh joel i remember the other guys unboxing it but he's part of the Egghead crew. And I'm like, there are three that I knew liked them, that I knew would appreciate this thing, and that I kind of assumed they would say nice things, like that they might help market this campaign a little bit. And they did.

They were great. But they had no idea it was coming. I think, you know, I've not asked them directly. I think they were very appreciative, right? That is value add. Even if you would call that bullshit, like that's like a thing that if somebody walks away and they're like, I really liked that. Like I, they made, it made me happier. It made me feel better. I got something really good out of it. You know, that that's what you need to strive for. And so all you got to do is frankly, use your

brain and look for things you can do that now doesn't mean they'll always work. Right? So pledge is actually a really good example. We're paying for billboards and stuff in sf and that that's not that cheap um i don't know that we're going to get any roi from a brand awareness in that because also there's a there's a balance like we can't be like pledge brought to you by century the program would never succeed we're kind of hoping that people that care and matter like oh yeah century is a big part of this it matters okay um and that's why we're willing to fund it to some degree the other half is like we'll just say like you know we morally believe in this and it's okay if we willing to fund it to some degree. The other half is like, we'll just say like, you know, we morally believe in this and it's okay if we don't make money on it, right?

So it doesn't always work. Even syntax, half the people don't know that we own syntax. It's like, I'm like, okay, whatever. It'll sort itself out over time. And so I think it's an interesting dilemma. Yeah. Do you try to measure ROI on things like syntax

and pledge and billboards and things like that? So we don't. And I'll give you an anecdote. Early day century, we were talking to Microsoft a lot. And we were trying to position ourselves in case there was a situation where we needed to be acquired. Microsoft was a really good partner early stage, right? This was before they bought GitHub and stuff. And so I had the opportunity to talk with a lot of folks over there.

And one of my favorite conversations was with, I forget her last name, but it's a woman that at the time she was, she was one of the CVPs, a mini CEO of Microsoft who ran visual studio.net, et cetera. And what I knew about Microsoft at the time was everything had to funnel money into Azure.

That was a single driving KPI of Microsoft. Now it's co-pilot or something, right? They're really good about rallying on something they think drives the business or they know drives the business. That division did not have to. They never had that KPI per this CBP. They said their KPI was just drive developers, drive developers on Microsoft products. And I'm like, interesting. How do you

connect those numbers? Because I'm like, you must have to connect them to measure this, blah, blah, blah, blah. And I'm like, interesting. How do you connect those numbers? Because I'm like, you must have to connect them to measure this, blah, blah, blah, blah. And they're like, we don't. Once upon a time, we did kind of this study, and we were able to correlate revenue extraction with just two metrics that would grow independent but together. And that story has had such a profound impact on my decision making where I'm like, that was good enough for Microsoft. I don't give a shit to argue anything. I'm like, if we grow syntax as listener base, engagement, whatever, I'm like, somehow we will achieve ROI on that, right? Without it being artificial or fake or whatever, because people do figure it out eventually. They do value our products.

And I just think if you look for those things, there's so many of them. And you just got to be a little bit clever and stand out from the noise. And I think that's the big thing. Yeah. Are there any other, I would say, good programming industry podcasts that are, I guess, now owned by a company? I don't think there's actually that many big things, first off. And so if you look at the most successful tech podcast, it's a little wonky, right? Because it's like acquired FM, or is it FM? Yeah, yeah yeah yeah really big great podcast it's basically some venture guys and it's it's phenomenal but it's not really like that it's

not like programming um historically we never saw any i i've not evaluated this in eight years but we never saw any real roi from from most podcasts and i think mostly because they did not have great listenership and i think partially because the value test matters everywhere. Right. But Syntax had like, we knew forever that it was great ROI for our business that like they drove good audiences and they've grown really well over time because of that. We do know it's a little like, it's not quite in the exact circle of our target customer. Cause like things like front end is like front end is not just the programming. It's also the design and all this stuff. Right. But I don't think there's anything else that there's a lot of smaller ones, you know,

that, that I think have value. And I don't know what the, what the Delta is. Cause they've also been at it for, I don't know how long you've been running this, but they've been at it forever, you know?

Yeah. I've listened to them for a while. They're good. Yeah. They got great stuff. The only other one that I, I really enjoyed that is very, very large is darknet Diaries,

but it's not owned by a company. The guy's done very well. He's a good storyteller. But I don't think everything else that I've seen owned by a company is started by the company. And they fuck up the value prop where they don't understand that the goal is not to advertise their company, right?

And they just never get anywhere. Yeah, yeah, for sure. That makes sense. Interesting. What about now, even looping back to another thing, when we were talking about open source, there are open source databases,

there are open source frameworks. Are there any other open source SaaS applications like Sentry? What are some other? I guess Calendly is, or which one's the Cal? Cal.com. So Cal's a little different.

So Cal's like a typical open. There's a lot of open core. Cow is open core. Git lab's open core. There's not really like, so controversially right now, WordPress, I used to always associate pretty similar to us.

Not quite the same, but it was like, we're pressed.com was the hosting effectively. Everything that like, maybe not their infrastructure, but like the product was totally open and free and you could use it, you know? And so that was one I always associate that was kind of similar to us, but it's very uncommon.

There's a bunch of smaller ones, but I think it's not worth talking about small companies because it's not a real good test. I can't name a company that kind of looks like us that is truly free, that is at the scale or anything. I guess, is Supabase? I guess, is Superbase, I mean, Superbase is much smaller than Yala right now. I think they're kind of like that where you can run a lot of Superbase. Yeah, I guess there's a few. I forget.

I think they're also open core, but I actually don't, I've not looked very closely. Okay, okay. And usually that's a promise because like to monetize, you need a, there are some people,

I don't agree with them. I like some of the folks that believe in this, but people will believe that you can protect your open source through trademark enforcement we've had a trademark on century forever that shit is not that useful um and elastic is evidence that it's not useful um but so a lot of these the true open source kind of the way where it's free to like free software like we think about it um their approach is to use the trademark to keep it free and open to not be open core um now elastic was open core at the same time but um that's the

only another variant that exists and even those haven't gotten very far yet and so i'm curious i'm curious to see i think more people will try and our whole thing with fair source is like we want people to have more free software that is feature complete we don't want to have to sell what i describe as crippleware open core doesn't mean't mean it always is, but it's a judgment call if it is or not. It's like, you're trusting the people in charge to not open source the thing that makes you want to pay for it. And so it's just not really incentivized correctly. And that's why I don't like that model. But that's the majority. That's like probably every single company other than us that's achieved the level of revenue we have is open core.

Yeah, yeah, for sure. Well, David, I love this conversation. Yeah, I love learning about centuries architecture and then open source and all this sort of stuff. So I will let you get on with your sabbatical now. But if people want to find you on the Internet or elsewhere, where can they look? On Twitter or X, whatever you want to call it. I'm Z E E G Z. Uh,

don't ask old nickname for no good reason. Uh, that's about it. I don't, I don't know. I'm like none of the other social networks really exist for me yet, but I'm pretty easy to find. Just Google David Kramer.

You'll find me. I'm not the guy that was in mash. Nice. Nice. Well, thanks for coming on the show and yeah, enjoy your sabbatical. Awesome.

Thanks Alex. Yep.