Software Misadventures - Impact Driven Development | Matt Klein (Envoy, bitdrift)

Episode Date: June 18, 2024

From creating Envoy to co-founding bitdrift to reimagine mobile observability, Matt joins the show to chat about being told to simply “write some proxy in Python” in the early days of building Envoy, early influences from building “shrink wrap” software at Microsoft, the process of spinning bitdrift out of Lyft, and much more. Segments: (00:03:10) Being a plumber on LinkedIn (00:05:00) Early influences from building “shrink wrap” software at Microsoft (00:10:44) Getting diverse work experiences (00:16:36) Setting high standards for the team (00:20:42) Lessons from failure of the first startup (00:22:02) Building a successful open source project vs. running a startup (00:25:25) Why not start a company around Envoy? (00:29:54) Why not open source bitdrift? (00:36:01) Mitigating the risk of big companies building in-house solutions (00:38:16) Co-founding bitdrift to tackle mobile observability (00:40:37) Applying lessons from the first startup failure (00:44:14) Why mobile observability is so hard (00:50:06) Open source vs source available (00:53:33) The software licensing strugglebus (00:58:03) How bitdrift was spinned out of Lyft (01:03:36) Achieving work-life balance through leverage (01:06:13) The early days of Envoy (01:09:20) Impact driven development (01:13:43) The crazy decision to build Envoy in retrospect Show Notes: Matt’s blog posts on why mobile observability is a hard problem: https://mattklein123.dev/2024/04/24/no-one-talks-about-mobile-observability/ The new company Matt is building: https://bitdrift.io/ Stay in touch: 👋 Make Ronak’s day by leaving us a review and let us know who we should talk to next! [e-mail]   Music: Vlad Gluschenko — Forest License: Creative Commons Attribution 3.0 Unported: https://creativecommons.org/licenses/by/3.0/deed.en

The most difficult time was probably in the 2017 time range because the project hadn't blown up enough to make me relatively well known and then to have I would say more leverage over Lyft. So it's like you're kind of like in that intermediate phase where like the project isn't important enough like I'm not important enough you, but it's like I'm killing myself, like trying to make it like make it successful. So you're kind of doing that, like outside of working hours. And then at a certain point, you know, the project becomes successful enough that like the personal leverage increases, such that I can almost force like better work life balance, if that makes sense. Would I do it again? Yes. But it was a difficult time in my life.

What kept you going? I think what kept me going is I almost entirely, 99%, I'm impact driven. Like I do not care about writing code. I care about code being used. In 2017, it just started becoming clear that the thing was blowing up, you know, like I was like going to companies to give talks, people would raise a bug, I'd fix it like that day, I was invested, because I believed, I think rightly so, that the more involved, the more invested I was, the more likely it is that people would end up using the software. For someone who might be in a similar position where let's say they were building something at a company

and wanted to spend the software, but wanted to be on work terms and ideally have that company as a customer too, any lessons that you could share? Be prepared to spend a lot of money on lawyers. No, well, yes. And? Yes, but no. I think that communication is really the only thing, right? You're never going

to make something like this happen by going behind the leadership of your company's back. It's impossible. I mean, it's like, you can't, for example, go out and be like, well, you know, it's like, I can raise this funding around and like, we're all just going to quit, you know, and like F you. So unless you're prepared to really like leave and walk away from all the IP and face potential lawsuits, the only way of approaching it is to have really good, open and honest communication and start talking to people about whether this is possible and whether this is something that the company would be interested in. And I think that in the current environment, in some ways, it's better for this right now because companies are looking to cut

costs and if there's a way for them in some cases to shed headcount and still retain some of the existing services right and have like funding come in from the outside like that can actually be a great situation. own edge by sitting down with engineers, founders, and investors to chat about their path, lessons they've learned, and of course, the misadventures along the way. Cool. Matt, so according to your LinkedIn, you're the co-founder and the chief plumbing officer at Bithrift. And before, you were a plumber at Lyft. And before that, you were on the plumber Oversight Committee at Cloud Native Computing

Foundation. So at which point in your career did you realize that self-engineering is a lot like plumbing? I think it's been a while. But to be perfectly honest with you, when I made the switch was to avoid LinkedIn spam. Because I actually found that if your title is software engineer, you get like endless amounts of spam. But with my title as plumber, I do occasionally actually get a plumbing related spam on LinkedIn. But it is like a fraction of what I used to get. Um, so there's the, there's the LinkedIn spam component, but then there is also that I've spent, you know, my 20 plus year career, mostly working in the, uh, plumbing aspects of computing. So there's, there's obviously lots of overlap. Did you like try to throw them when they uh sent you like the spam message about like a real

plumbing job and you were like um no no no yeah i i i do do some amateur plumbing so you know it's like maybe a little bit within my within my skill set but i'm no expert so no okay okay okay so uh so you started your career at Microsoft and, uh, you mentioned that like, you know, the, the five years there play like a pretty important role, um, from the people that you met from like the engineering practices, um, like, you know, are there any habits or practices that like today that, you know, you still trace back to this time? Yeah. You know, it's a, it an interesting thing to think about, because it's been so long, I started at Microsoft in the early 2000s. And that that was, if you think back that

far, that was really, I would say the beginning of the main internet era, right? It's like we're, you know, coming out of a time in which most software was shrink wrap software. The way that people develop software was very different in the 1980s and 1990s. So the Microsoft that I joined in the early 2000s, you know, still for the most part was run, you know, like a shrink wrap software company, meaning, you know, they had like people that did QA and then they had separate people that, you know, did engineering. And then they had different people that did program or product management. And obviously looking at that today, you know, that seems crazy because people, you know, whether they're doing like operating system software or internet software,

you know, things tend to be a lot more agile at the same time though. You know, I, I think coming out of Microsoft during that time, and because when they shipped software back then, they shipped software. I mean, that was on CD-ROMs. You would do the software and put it out there, and updating the software didn't really happen. You know, I mean, like maybe you would put out a different CD-ROM or something along those lines. So I think in general, the rigor around finding bugs and fixing bugs and how important the quality assurance process was, was a lot more important for most aspects of software back then. And the Microsoft that I joined,

you know, that type of culture, like percolated throughout the entire organization, just in terms of the rigor that would be required, you know, to make, I would say, quality software. And, you know, I had exposure during those early years, and I was obviously straight out of college. And, you know, I had some opportunity to work on the Windows NT kernel at certain points. And just seeing, I would say, the quality of the engineering that came out of that and the rigor, I think that's the main thing that has kept with me throughout my career. Is just, to me, it's always been important to me in the craft of engineering that things should work. You know, it's like nothing makes me more unhappy than software or systems that don't work correctly.

And of course there's bugs, and of course there's, you know, different severity of bugs, but I'm really of the opinion that the systems that we've built, they should work and they should feel magical. And yes, it's true that times have changed.

We obviously live in a world now where you typically can ship fixes very fast, though of course related to my startup, not on mobile. There's still many environments in which you can't ship things fast. But the vast majority of work in a world where you know, you have a problem, you ship something in your hotfix channel, or you know, you have a problem in your internet service, and you can get a fix out within an hour. So I think, I think, you know, as an industry, we've definitely moved to the side more of ship faster, even if it's broken, we'll just fix it later. I think there's something to be said for that, but I still believe in the craft of quality software. And I think that does come from my time at Microsoft.

Interesting. Like when you got first exposure to like Agile, like later on in your career, were you just like, what the crap is this? Well, you know, so I worked at Microsoft for the first, let's say four or five years of my career. And I did actually start a company when I left Microsoft, which ultimately failed. And I was pretty young, and there's lots of learnings from that.

And then, you know, after that, I ended up, you know, basically going and working for Amazon. And when I started Amazon, I wasn't only working at Amazon. I was working very early on within AWS. I worked on EC2, and this was in 2010, after taking a detour through a couple of other jobs. And I'll be honest, it was shocking to me, actually, like coming into EC2 during that time, just in terms of how fast things were moving, how broken, like everything was constantly. And don't get me wrong. I mean,

as we all know, AWS and EC2 is, I mean, probably the most successful software business of all time. So, you know, they were doing most things right, right. But like coming from a Microsoft perspective, I just I just wasn't used to that culture of the ship as fast as possible. Doesn't matter if it's broken, fix it later, right? You know, so that was my first real exposure to, to like hyper growth, software development, you know, just in terms of shipping fast, even if it's broken, ride the wave. And of course, the the 2010s was an epic era of growth just in the entire internet space. So you know, if you look at my time at Amazon, and then I was a Twitter, and then I was at Lyft. You know, I was always involved at that

point in the ship it fast, make it work later culture. But I think I still approach things in a way that really stem from my time back at Microsoft, you know, which is just having a rigor to software, like having a rigor to the craft of software like where do you find that balance it's tough i mean there's no there's no like there's no great answer you know and if there was an answer i think people would have figured out some formula that they could follow and and they're just there just isn't one um so i don't even have a good answer for you there. Do you think all engineers should at least get to sides of the extreme in order to develop that balance throughout their career? I always tell people that I think particularly early on in one's career, I actually think there's great opportunity in working in different environments and on different things, like even for the first five or 10 years of like one's

career, meaning, I think it's great to work at a startup, I think it's great to work at a giant company, you know, I think it's great to work on different types of engineering. And the reason that I think that is, I think you're going to learn different things, you're going to learn different approaches to software, different keyed-ins of building systems, like different approaches to quality. And, you know, I wouldn't change all of the experiences that I've had at all because they've all been really formative to me. But, for example, you know, when I did Envoy, right, and that was in 2015, starting in 2015 at Lyft, we could have a whole conversation about whether Lyft should have allowed me to do Envoy in the first place. The answer is probably no, but they did.

And my approach to that was a very rigorous approach. It was my goal to build a highly functional and reliable piece of software. And there's no way that I, I believe that there's no way that I actually would have been able to do that if I had never started my career at Microsoft. I never would have learned that I would never would have learned that at Amazon, like maybe maybe later on in Amazon, right. But like the Amazon of 2010 that I joined was a like chaotic hyper growth startup environment where, you know, I, I never would have learned those skills. Um, you know, and it's like, maybe I would have learned them if I had worked in a

certain portion of Google or maybe other, other systems or other types of companies related to like low level systems or operating systems or things like that. But I really do believe that starting my career at Microsoft and learning some of the rigorous approaches to engineering did shape my views on this topic for sure. So a couple of questions on those lines. One aspect is, you said this about EC2 being one of the most successful business in software, plus lots of things were breaking, initially at least. In that environment, how do you know if you're actually making progress or rather doing most things right?

You know, that's a great question. And then we get right into the intersection of what does, quote,, quote, right mean, right? I mean, is that like, is it is it engineering, right? Is it business, right? And I'm a pretty pragmatic person. I mean, like, ultimately, we're getting paid to make money. So, so, so, so at the end of the day, we're, we're paid money to hopefully make more money than we're paid. I mean, that's, that's like the whole point. So if you look at, let's, let's take revenue. I mean, that's what I'm saying. It is probably the most fantastically successful business probably of all time in software. And maybe there's others like the Oracle databases.

I'm sure there's other examples or like Windows. But it's in the top five for sure. And so to me, if you look at the time that i was in amazon even though technically things were breaking all the time you look at the hockey stick revenue growth and you're obviously and you're obviously succeeding it's like all you're trying to do is keep things from completely falling over um and of course, I mean, I haven't worked at Amazon since 2012. And yes, of course, like Amazon and AWS has matured a lot. Now they do lots of foundational research

and changes take longer. And like as businesses mature, like of course that has happened. But, you know, when you ask like, how do you know if you're quote doing it right? I mean, in my opinion, it's always been, are you solving customer problems and are you helping the business be successful? And that's always been my guiding light, right? In terms of, like, I've never, ever in my career been attracted to shiny, I would call them academic type concerns.

It's just not very interesting to me. I've always been driven by a customer impact. If what I'm doing is not being used and having impact, I don't feel that I've done anything useful. So even though I really appreciate people that work in academics and do all of those things, that's just not for me. I mean, like I tend to be a more hands-on, let's fix problems and let's help people succeed. And in my experience, doing that often comes with favorable business outcomes.

So one aspect of what you mentioned was the rigor that you have today, you can trace it back to your time at Microsoft. When we have moved to now environment where shipping fixes is relatively much easier, it's like, let can trace it back to your time at Microsoft. When we have moved to now environment where shipping fixes is relatively much easier, it's like, let's put it out. If it breaks, we'll trade on it. How do you kind of teach this rigor to the team you work with? Because not everyone could have had the same experience as you had. Again, it's tough. I mean, I think at a certain point, the only thing that can be done is to lead by example. Like I still write code, I still do engineering. So I think the only way to do it is to write code and do engineering and like show people the way. And I'm not going to say that I'm always thinking with like 100% rigor coverage. I mean, the amount of times in the last 10 or 15 years that I can think of, oh, there's

some bug. Let's look at the code. Oh, oh, to do Matt Klein, one, two, three. All right. And, and then I like to tell myself, well, at least there was a to-do. Yes, at least you knew this would happen. At least I knew that this was going to happen and I wrote it down, but I didn't obviously fix it back then. So I'm not claiming to be a person that doesn't move quickly, right? And that's kind of what I'm saying is that this is just one of those topics that there is no right answer like there's not even there's not even a way to describe a formula that you can use it's completely subjective and every situation is just a gut check on like is it worth to do now or can we write that to do matt klein one two three and like we'll we'll do it later you know okay um so we want to get into a

bit drift um maybe a little bit before that is like so you mentioned this already that after microsoft you co-founded this like video delivery startup it didn't quite work out but you've learned a ton from that experience um so that was like 15 years ago like um yeah did you always know that like you want to give it another go in terms of starting a company? No, no. I'm not the serial entrepreneur type person. As I mentioned, I like solving problems for people. And so for me, I like being in an environment where, don't get me wrong, I like tough technical

problems. But if it's a technical problem that doesn't have any real-world impact, it's not interesting to me at all. So it's like I've always been drawn to solving problems for people. And those people could obviously be end users of some app, or it could be internal teams or something along those lines. But as long as I'm having impact, that's the thing that's obviously most important to me. So, you know, it's like when I, when I did the first

startup, when I was really young and super naive, like I didn't really know anything. I mean, I didn't know what it means to start a business. I didn't know what it means to have a co-founder that would work for me. So like in terms of learning a lot, you know, most of what I learned, at least back then at a really young age is what, what not to do, right. in terms of learning a lot, you know, most of what I learned, uh, at least back then at a really young age is what, what not to do, right. In terms of like, what's important to, you know, finding a co-founder or co-founders that are compatible with the, with, with the, with the skillset that I have, you know, the importance of like having very clear goals and, you know, very clear outcomes.

And I'm not saying that what we're doing now is, is right. I mean, don't get me wrong. Starting a company is really difficult. Um, and you know, but, but you muddle your way forward the best you can. And, you know, the other thing that I would say is that doing Envoy in many ways was like starting a company, just not like a company for money. Like I think actually doing open source well is almost identical to starting a company in terms of finding product market fit and like doing marketing and like, you know, hiring maintainers and all of those things. So, you know, I think I learned a lot about what it takes to be successful in the general sense of starting something from scratch through Envoy. But doing open source is different than doing business as well, right? I mean,

because with a business, you're ultimately trying to get people to pay. And with open source, you're trying to get people to use your software. There's a lot of overlap, but there's also a lot that's different. Right, right, right. To stay on brand with the podcast, was there any memorable fails? No, I mean, it's been so long that nothing's super, super memorable. I was young and I was not very intentional about starting the company with the right goals and with the right co-founder that had the right mutual skill sets and the right time and personal investment and goals. it's not like a memorable flashy soundbite fail, but I think it's a really important fail, which is just that when one starts a company or a new endeavor, I think, you know, being really honest about with the people that are starting the company and

just what everyone's goals are and like being aligned on that is super important, you know, because if you start such a big endeavor and people have different ideas about whatever it could be, right. It's like what the outcome goals are or, or, or like what the, what the area of focus is or, um, like how much people are going to work or, you know, there's like 20 different things. Um, like I think that that's the most important thing. You said this, uh, working on open source project and startup, there are

obviously similarities in terms of you want people to use the product. In the case of startup, as part of using the product, you get some money as well. What are the differences that you would think of? I think there's more similarities than differences, to be honest, in the sense that you build something, you put it out there and like i said you have to engage in marketing and outreach and you know you have like a lot of the same hr type concerns and all of that um and you might even have to raise money in order to like do do development for it so i think there's lots of overlaps I think the big difference is that how a product is adopted is different in a paid environment versus in like an open source environment in the sense that in an open

source environment, you're implicitly giving away something for free. So all of your marketing and all of those things, you never have to fight the, do I have budget for this? Do I have to do this? Or do I have to do that? People can always see the source code, right? So it's like, I think that everything is the same from like a marketing and a quality product perspective and all of those things until you actually start trying to get people to physically use and then pay for it. And that's where things really diverge. Because, you know, from a from a getting paid to do something perspective, it typically involves a lot more people, right? There's like procurement, there's budgets, there's like all this other stuff.

And then, of course, if your product is closed source, or there's proprietary components, you know, you might have to deal with people, you know, not being comfortable with that. So I think that I think that much of it is the same until you get to the actual, we'll call it the adoption phase. Whereas on the open source side, someone takes it, it's effectively given without warranty. It's like if they use it, it's great. Obviously, the better the software is, the more likely it is like if they use it, it's great. Obviously, the better the software is,

the more likely it is they are to use it and to keep using it. But that's kind of where it ends. And then when you're talking about payment, then you're starting to involve support and lawyers and contracts and more security and all this other stuff. So I think the bar is just higher

in a paid business scenario where when money exchanges hands, there's just yet another level effectively. So back in 2017, you wrote a post called Optimizing Impact, Why Not Start an envoy platform company, which I thought was a very eloquent way of telling VCs trying to throw chunks of cash at you to, you know, just please go away.

So I think it's safe to say you've given the subject a lot of thought, right? Like before actually deciding to start the company, what finally like put the trigger for you to build Bitrift? Well, I mean, so to come back to the Envoy thing, it's funny because there are still people to this day that will come up to me and say, wow, that's such a missed opportunity that you didn't start a company around Envoy.

And look, I mean, if we're being honest about it, could I have started a company? Sure. It's like, you know, could we have made some money? Sure. What's not clear to me actually is if I had done that, would Envoy be as popular as it is today, right? So it's like nothing is without trade-offs and they may say, yes, I'm not so sure. And look, I'll be honest, I'm not a poor person.

So it's not like I'm, it's not like I'm hurting for money for like not having done that. So for me in the Envoy decision, Envoy will always be my baby, even though I'm not as involved in the project as I used to be. And for me, it was always about seeing it to, I think, its ultimate potential. And I still believe today that if I started a company, it would not be what it is today. I mean, like Envoy is, even starting this company now, it is unlikely that I'll ever see a success like Envoy. I mean, it's like Envoy is used literally around the world. I mean, it's like it is used by almost everyone. So to have that kind of impact, it still gives me goosebumps. It blows me away. And there are many other companies that have been started around Envoy,

and they're doing okay, not fantastic. And it's like, could I have had one of those companies? Sure. But again, would it be what it is today? I'm not sure. So when it came to doing BitDrift, I think, again, so the other part of it is that I'm not the type of person that does the same thing forever. I just can't. My brain doesn't work that way. I get bored. I need new challenges. And I respect people like Linus that have worked on Linux for 35-whatever years or something like that.

But that's not me. I mean, I just can't do that. I'd be so bored. So for me, it's not that I don't love Envoy and I don't care about Envoy. It's more that it wasn't going to be the thing that I was going to do forever. So anyway, so, you know, we had worked on a bunch of stuff at Lyft. And towards the end of 2022, I mean, it just became clear that if we were going to make a serious effort of this, it had to be an independent company.

So we decided to make a go of it. And, you know, I'll spare you the gory details of spinning out of a company like Lyft. It wasn't super fun, but we made it happen. And Lyft is an investor in our company and we left on good terms. And I think that I'm passionate about what we're doing now. And kind of like when I did Envoy, I don't know if we'll succeed as a business,

but I think we're solving a problem that no one else is solving in the way that we are. And what I'm excited about is I think lots of companies out there, you know, they're started in like almost like a me too fashion. You know, it's like, oh, AI is hot. Let's like have another AI company or, oh, this is hot. Like we'll do something like this. And yeah, I mean, I thought it was the different me too. company. Or, oh, this is hot. We'll do something like this. Okay, that me? Yeah, I mean... I thought it was the different me too.

I was like, whoa, what's... Oh, no, no, no, no, no, no, no, no. Sorry, sorry. Like a me too, I'm going to do the same thing. Yeah, yeah, yeah, yeah. No, sorry. Not that type of me too. And so it's like with what we're doing,

even though we are operating in the observability space, which is, of course, a very, very hot space, we're like kind of out there, like we're doing something different. Right. And whether we succeed or not, I have no idea. But I love in my career doing things differently and showing people that it is possible to do something differently and like be successful at it. Because I think a lot of times in our industry, we have a lot of copycatting and a lot of incremental, very incremental to no change. And I just think sometimes you have to just step back and do something differently, you know? And it's like when I did Envoy, I mean, initially, not surprisingly, a lot of people said to us, it's like, isn't this done? You know, like there's NGINX or whatever, or there's HAProxy, like, isn't this done? You know, and the answer is no, it's like, we're not done. Like there's more that we can do.

And similarly in the observability space, I think we're not done. Like there's, there's more that we can do. And similarly, in the observability space, I think we're not done. Like, I think there's a lot more that we can do, but it takes thinking outside the box to actually go and do that. And that's the kind of thing that I love. It's like, I love just doing something different. So it's interesting. Envoy is definitely different when it comes to proxies. As you mentioned, there are many proxies that have existed before Envoy, but you wanted to do something different you did it in an open source fashion like you built an open source project around it and we'll get into why Lyft shouldn't have allowed you to do that and you and they did why not do the same with something like BitDrift too because

if I compare the two parts that you said of working on an open source project, which is a company, from a company perspective, what's different was lawyers, contracts, paperwork. So far, I haven't heard anyone, at least from an engineer, who can say, oh, that's the part I love the most. People say I love working on the product itself. So you've obviously chose to do BitRiff outside Lyft. Why was that? Well, I guess there's two parts to unpack there. First, why outside Lyft? And then why not open source?

So let's take the why not Lyft. Again, I'm just being very honest. I think towards the tail end of the 2010s and during, I would say, the height of the internet boom, like during the, during the middle of the COVID era before things started to really collapse. You know, it was very popular for these large companies to, to like say, ah, like we're going to do an internal startup. We're going to let you go and do some stuff or whatever. And, you know, the honest reality is that 99% of these are a way to retain people and, like, keep them from quitting. And it's like these projects are never really going to go anywhere. And if I'm really honest, you know, that's what we were doing at Lyft.

Not that we didn't build cool stuff, but it's never going to become a real business within Lyft. It's just never going to become a real business within Lyft. It's just never going to happen. It's like the only way to make it a real business is to spin out, become a real startup and like actually try to sell the thing. So that's, that's, that's like the first part. Why not? Why not Lyft? The second part of why not open source is, is more complicated, right? Because I think that we've seen in the last five years a big backlash actually around trying to build businesses on top of open source. And we've seen a lot of very notable retractions of open source licenses once companies realize that it's inhibiting their

business model. And at least in the area that we are doing, um, I'm a big believer and we are actually going to be making, um, much of our code, not the server code, but the client code source available. I'm not going to say open source because I don't want people jumping down my throat, but, um, we, we are going to make it source available. But at the same time, I think actually it's easier to become more permissive in your license than more restrictive in your license, right? And by that, I mean, we can start close. We can start with like a BSL type license. And if in the future we want to make it whatever, like AGPL or Apache 2 or something like that,

we can obviously do that. I think it's when people do the inverse. They start, like a GPL or Apache two or something like that. We can obviously do that. Um, I think it's when people do the inverse, they start like build a community and then they realize, well, well crap. It's like, we have to, we have to make money. Um, I think that, yeah, backlash. The other thing that I'll say that has changed is as you both probably know, right. Is that like the 2010s were an era in which making money was not required in business, like it just was not, I mean, there was so much money floating around, and so much hype, kind of like the AI situation right now. But But like in the 2010s, when it came to all the, you know, like the computer infrastructure world, It's just so many companies were getting massive amounts of funding and growing,

you know, without really actually making any money. And then now these companies have to make money. And that's where you see like a lot of the license changes. So I think, you know, we're building a business in 2024. And in 2024, we have to make money, which is great. I mean, it's like I would rather build a business in which we have to make money. But in order to make money, we have to make money, which is great. I mean, it's like, I would rather build a business in which we have to make money. But in order to make money, we have to make money. We have to have like have a product that we purchase or that people purchase. And giving it away is not a great way to do that. And especially where what we're doing at Biddrift, at least right now, a lot of the

initial interest that we've seen has been from very large companies because we're solving big company problems. And those companies would be the most likely companies that would just take it and would run it themselves and would never pay us anything. I'm not saying that you can't build an open source business. You can. But I think we view it as more strategic to attempt to build a proprietary business. And then if there are ways of opening things up later, we'll of course do that. Because again, look, I mean, I love open source. I think open source has been a critical part of our industry for decades now. But I'll be honest, and you probably already know this from listening to my other talks or whatever else, is that I'm not a free software zealot. I mean, it's like quality open source gets made by people making money to get paid to write the software, right?

I mean, it's like the highest quality open source software is not made by people making money to get paid to write the software, right? I mean, it's like, the highest quality open source software is not made by hobbyists, it's made by highly paid programmers, and someone has to pay them, right? So it's like, there's no free lunch here. And like, at the end of the day, all of this software costs money to create. And that money could come from, you know, companies sponsoring people to work on open source, or it could come from you know companies sponsoring people to work on open source or it could come from you know companies that are sponsoring open source um but at the end of the day it all it all takes money like nothing gets made for free any open source software built on nights and weekends rarely gets maintained rarely gets maintained no it it's not

it's it's not not only is it not maintained but in the current day and age i i cannot think of any major open source software that is used widely that is not maintained by highly paid programmers like i can't think of a simple i can't think of a single example so i'm going to go back to one thing you said about bed drift you mentioned that a lot of interest you're seeing in this problem space is from big players. Now, one aspect around a lot of big tech companies is that they typically build in-house solutions because in many cases, buying something from a vendor ends up being too expensive. That's where the conversation starts. Oh, we can build it more cheaply. Whether that's true or not is arguable in some cases, but that's where many big companies end up being. In that case, if you're seeing this interest from

big companies, don't you see this as a risk from a business standpoint where they would see how your product works? They might pay for it for some time, but they eventually would just go build it themselves if it ends up being expensive? I mean, it's certainly possible, but I'll also say that I think the end of the 2010s and now let's call it the austerity 2020s, right, is that companies are starting to be more realistic around how much money it would take to build some of these tools, right? And look, I'm going to be honest, like as a company, we are very early. I mean, we're like very early on our journey. We're working on our first pay contracts and all of those things. So I'll talk to you in a year and we can see how things are going. But just to make numbers up out of thin air, right? In the US, like especially for the higher tier tech companies,

I'm sure your average salary all in, including healthcare and benefits and all of those things is at least $250,000 a year, if not more, right? When you include all the benefits and all of those things. And when you start actually thinking about evaluating it that way, on like how many people would I have to pay for how long, right? Like in order to build this comparable functionality in the new world of not being able to hire unlimited people, something actually in many ways becomes an easier sell because you

can say, look, like I'm going to charge you even for larger contracts, let's call it between $100,000 and $250,000 a year, that's still going to be way cheaper than you could possibly pay anyone to build this thing. And then there's this company that's iterating on it and making it better and all of those things. And look, I have no idea whether we will succeed or fail. I'm just saying that I think that some of the economics are changing. I think some of the NIH culture is changing. I think that works actually in new companies' favor because now larger companies are going to say,

well, yeah, wow, I'd have to pay five people for two years to even come close to what you've accomplished. Well, that doesn't make any sense. It's like, let's just buy it. Coming back to what we were saying before, the way that I think that we sense. It's like, let's just buy it. And, um, you know, coming back to what we were saying before, the way that I think that we're going to approach it and, and obviously this is not announced yet and we're still working on the details, but with our client side libraries,

the code that runs like on the clients and within apps and like not our SAS, we are going to make that source available in a way that would allow effectively anyone who's not Datadog or AWS to actually come and take the code to build their own control plane. We don't think that's realistic because the control plane is very, very sophisticated, right? But that would knock down yet another objection for these large companies, which is to say, look, like if you like this, if you buy into this, here's the code. And if you really, truly want to write your own control plane, your own UI, all this other stuff, have at it. the last objection, because then at that point, they're not really locked in in the sense that

if they chose to do it themselves, they're even ahead, because they could take the client code, and they can build their own in-house control plane, and then, well, they can go off on their own. So I think that kind of hybrid model is a bit more interesting in the sense that it protects our business interests, but if you're the largest company and you really decide that you don't want to pay bit drift for their sass okay you can you can go write your own as long as you don't sell it to others um and i think that strikes a good balance um so to do a full circle back um about the first lesson that you or the lesson you learned from the first startup about being intentional right so this is super cool to hear like how intentional you are about

like the business model and all these things so the other part I guess is co-founders so how did you go about like how did you find them and what are the criteria I guess yeah I mean so I have I have two co-founders at BitDrift. They both work with me at Lyft. So, I mean, we have the benefit of having worked together for many, many, many years. I mean, one of them I've worked with at Lyft for about eight years. The other eight years at Lyft, and then I also work with back at Twitter. So, you know, probably 12 years of total experience. So it's like, you know, you're, you're going into it knowing

because we all have strengths and weaknesses. It's like, we all, we all have the things that we bring to the table. Um, and I think like going into it and being intentional about the breakdown of what we're good at and what we're not good at. And, um, you know, trying to, trying to make the best of it because again, um, as anyone will tell you, starting a company is very difficult. I mean, it's difficult mostly in the sense of you never know what you're doing. It's like you never know if it's right, right?

I mean, it's like you're just like constantly trying to do something and, you know, hope that it sticks. And you can build the most amazing product, but if it's not marketed correctly or the pricing is not right or the sales is not right, you know, you're still going to end up failing. So, like, it takes the total package in terms of, you know, having a product that has product market fit, having a market that's large enough

to actually support that product. And if I'm honest about what we're doing at Bittreff, because right now we are focusing on the mobile observability side of things, which I would argue is the most important observability, because again, the number of times in my career where some backend dashboard is like, oh, 100% success rate. Everything is great.

Right. You know, and then, and then like, and then, you know, there's some like malformed JSON and like every client's like crashing. Right. You know, I, so I, I like my, my, my point is, is that people are always proud of their dashboards. It's like 99.997 success rate, and then 40% of your sales are failing. It's like, no, you have a 60% success rate. It's really bad, right?

Anyway, my point is that I think as an industry, we've invested a lot in back-end observability and all of these things, and that's great, but yet we barely know what's going on like at the customer level. Like as we began this conversation in terms of like what gives me, like what keeps me going

is that I want to build systems that work for people. And, you know, if your app is like crashing or not working, it's not working for people. Like it doesn't matter how awesome your Kubernetes is and like or not working, it's not working for people. Like it doesn't matter how awesome your Kubernetes is and like how awesome your Envoy is and like your fancy CI system or all that crap. Like it's worthless. Like if, if, if it's not working for your customer. So anyway, that's a, that's a long way of saying that mobile and like edge observability to me is critical. However, people are not used to paying for it, right?

It's like if you look at the budgets, so much money goes into people's metric systems and logging systems and all of these things. And like typically a tiny budget actually goes into knowing what's happening with the customer. So our big challenge from a business perspective at Biddrift is actually, I think, not gonna be building an amazing product.

I'm very excited about what we're doing. It's going to be convincing people that it's worth paying for, you know, it's like such a way that we can have a, a good business. But like, these are the kinds of things that go into building a business, right? I mean, it's like you can have an amazing product and you've got to make sure that there's people out there to actually buy it. So you know it's it's complicated um in this case like when it comes to doing mobile observability for example um why is this not done already in a way like because we have been seeing

so many observability companies but why is this different? I wrote a series of blog posts recently that we can maybe put in the show notes that I think are interesting and cover this topic. I've been thinking about this a lot recently, and the reason that I think is that it is hard. It is very hard. And there are many reasons why it's hard, but I'll just like list a couple, which is

the main one is that you're dealing with these remote clients that have, you know, very limited resources. They can be shut on and off at any time, like the mobile operating systems, like, you know, background apps or like turn them off or whatever else they have sporadic networking. So they're like constantly going, you know, on and off the network. And if you look at a lot of the way that we do backend observability, like our existing modern systems, they're generally built around the fact that like these systems are mostly always on. Networking is reliable, right? Like, yes, maybe if you get overloaded, you'll drop some logs, drop some metrics, but it's like these things, they're mostly there, and you never really deal with backfill, meaning most of these systems, they'll ingest data from the last five or ten minutes, and then if it's older than that, it's just gone. In the world of mobile, it's like you might have a client go offline and come back with useful data two days later.

So it's like, what do you do with that? Do you drop it? Do you ingest it? Like, how do you actually think about all of those things? And then not to mention the fact that in the mobile, and when I say mobile, I'm talking about like point of sale. I'm talking like kiosks. I'm talking like desktop applications,

like anything that's sporadically connected. For those people, even paying for bandwidth actually might be problematic, right? It's like people are on metered cell phone plans, or maybe they're working in a restaurant where they have like a tiny, really slow internet connection. You can't just be streaming data constantly out to Datadog. I mean, it's like you have to actually think about the environment where you're running. So to answer your question of why not, I think because it's hard. Like I actually think it's because it's hard. And I think that we see a lot more of these companies that are

operating in the web space, I think because it's easier because those, the desktops tend to be persistently connected to the internet because like you're obviously getting the web page from somewhere. So I think because it's hard, honestly. In this case, because a lot of this is at the point, like you mentioned, mobile phones or kiosks are point of sale. I'm assuming a lot of work in this case is going into either Android or iOS client-side development, where instrumentation itself, how do you keep the buffer data for how long?

How do you preserve it if, let's say, the phone doesn't get restarted? Yep. Okay. Yeah, and that's where, like I said, is that we have a smart client. Our SDK is pretty intelligent. We've invested a lot in it.

And if we want to get into the nerdy side of things, the SDK is almost entirely written in Rust. And that's done for a bunch of different reasons. But one of them is that Rust, because it's C-like, is effectively cross-platform, right? It's like we can run the same code in iOS, in Android, we can run it in Node.js. I mean, it's like we can embed that code almost anywhere. And because the code is sophisticated, we don't want to write the code like four times. It's important to write it once and make sure that it works well. I was going to ask why it was, but thanks for sharing that.

But in this case, the client itself is so intelligent. The part that you mentioned where you might make its source available, that seems bold, if I can say that. And don't get me wrong. This is a discussion that I have with my co-founders frequently. And we have differing opinions on this. And this is why, if we do make it source available, you'll hear me being very careful. I'm not saying open source.

It's like we are not going to open source this code. We are going to make it source available in a way that I think will mitigate almost everyone's concerns. And like a lot of the other licenses that people are moving towards their server software, the intention is mostly to prevent one of our competitors from implementing their own control plane and telling people to use our SDK to talk to their control plane. So for the average end user company, it would actually be fairly permissive in the sense that take the code,

do what you want with it. Like as long as you don't use it to basically interact with some other paid SaaS, have at it. But you're right. There is a lot of intelligence there and we would have to open source a significant amount of intel or make source available a significant amount of intelligence and that's where it gets tricky but at the same time many of the potential customers that we talk to they demand to see the source code so it's like because because no one is going to

put some random blob in their app it's just like that that that over. It's like, it absolutely won't happen. So then you get into these games of like, oh, well, if your contract is large enough, we'll like send you the source code. I mean, but that has its own complexities, right? So it's like, it's this constant trade off of trying to figure out what's the best way of satisfying all of these concerns. Yeah, I won't name names,

but I've seen these conversations myself where we buy some software and we have been in exactly these conversations where it's like, whatever we are buying is not our core business. So obviously we are not going to sell this thing. But before we actually run this in our production environment,

we want to be able to see things. And that's where like we just, I'm of the opinion that it is risky, but it is actually in our favor to make the source code available because it will break down. Again, like when you're a young company, in my opinion, a lot of what you're doing is just smashing down objections, right? It's like, there's like security objections. There's like source availability objections. There's performance objections. There's like all these objections. So it's like, what you're doing is you're trying to just kind of like check off every possible objection ahead of time.

And lack of source availability is, is a pretty major objection. And even though, as you said, like we can solve it by if someone signs a contract, like here's the source code, fine. You know, it would be easier, of course, like if it was just on GitHub. But there are other concerns. I'm assuming there will be a lot of conversations with lawyers too, in terms of how you would construct something like this. Yeah, we are, we are talking to lawyers now, so you can check back with me in a little while. And it's not surprisingly because of what we're doing, it doesn't look like any of the existing licenses really work perfectly for us. Because if you look at the current crop of business source licenses,

there's BSL and SSPL and those kinds of things, they tend to be more geared towards server software. They're trying to prevent someone from taking a database and then having Amazon run that database as a service. It's a little different, actually, than what we're trying to do, which is that we're trying to prevent someone from taking our SDK and repointing it at their service. It's a little different actually than what we're trying to do, which is that we're trying to prevent someone from taking our SDK and repointing it at their service. So like the terms of the licenses don't actually line up, but we're still thinking about this currently. So one thing as a, maybe a practical tip to many listeners who are in a similar boat, let's say

they've either started a company or are looking to start a company, or they're trying to use something off the shelf and wanna see if they can do certain things with it. You mentioned a few of these licenses, which permit you to, let's say, take software off the shelf, pre-package it and maybe sell it.

In other cases, it doesn't. Are there sources where people can learn more about what these different licenses exist and how to differentiate from one to the other? That is a fantastic question. And I honestly wish there was a better, more centralized source that people felt comfortable going to. Um,

because a, I don't think there is, I, you know, I think you can Google, uh, I, I would not ask chat GPT this question because it will hallucinate you some crap. So do not do that. Do not ask chat GPT about software licensing. That's my pro tip. But, you know, when it comes, yeah, don't, do not do that. So, you know, like you can Google licenses and you'll find, you know, 40 different websites of different lawyers and people that are talking about this topic. But I'm going to be honest, it is really complicated. And the other part of it, which makes this even more difficult, is that there's a lot of FUD in this space, right? It's like there's people that are non-lawyers

that are spewing all kinds of things, you know, in internet forums around, like, what is real open source? What is this? What is that? And it's like these are complicated legal and business concerns. And it's, and it's like, not so simple. And that's why, like I said, you know, we haven't made any final decisions. So it's like, I'm not, I'm not going to claim on this podcast that we're going to do anything yet, because I'm not quite sure. But I will say that we're being very, very intentional about not lying to people. We're not going to call it open source

because it's not. It's going to be source available. And when we do this, we're going to try to the best of our ability to have a lay description of what this means for you as a company. It's like as the consumer. This is what our intention is

for who can and can't use this and what they can and can't use it for. Obviously, people are going to have to go talk to their own lawyers and they're going to have to go think about it. And yes, to your point, wouldn't it be great if there was like a centralized clearinghouse for licenses and like a better way of thinking about this? But there's not. And it's, it's like really complicated. One aspect of what you mentioned was like, you're, you're having different opinions about whether to do this or how you do this within a team right now. Now, back when you worked at, let's say, left or at Twitter with, with your colleagues, if you have disagreements, eventually it's like you, teams call it different things, but you have this clean escalation path it's like you go up to your common boss eventually okay i agree with abc today it's three of you or maybe more and if you disagree on things what what does that look like

yeah i so first i'll say that even when you start a business and you're the founders of the business, you still have a boss. This boss are your investors. So, I mean, so like if you really had a problem that you couldn larger companies, while it is true that yes, there is an escalation path, when you reach a certain seniority level at even these larger companies, you're generally expected to deal with the ambiguity of the situation and solve your own problems. We're all pretty senior people. So I, I think, um, if there's disagreement, we typically just talk it out and like, come, come to some agreement. Right. Um, so I, I, I just feel like, I don't know. I don't, I think, I think you're asking a great question. I just, to me, that's an earlier career or more junior type concern. And if I were in my 20s,

it's like that would be a bigger concern for me, but I'm not. Like I've been doing this for a while. And I feel like I'd be surprised if there's something that we couldn't like work out amongst ourselves. That's fair. That's fair.

You mentioned that you were doing some of this work at Lyft already and then decided to spin this out. You mentioned there are certain things which weren't as clean and we can leave out the details. But in general, when you're building something at a company and you want to spin that off, typically the IP is kept by the company where you're already working. What did that look like in this case? So we did a full spin out where we took the IP and the patents and then we raised external funding and then Lyft actually invested in our funding round. So obviously we had to negotiate a lot of details, but this was not like a let's leave and be on bad terms and we're going to start from scratch.

This was a fully, here's all the code. Lyft is still a customer, which is obviously fantastic for us. Because again, we're an odd startup. Most startups would die to have Lyft as a customer, right? So for us to be able to start out with them as a large customer, it means that this thing has been tested at scale. Now, it's odd, because I'll be honest in saying that you spin out of Lyft with Lyft as a customer, it doesn't mean that you have product market fit, right? Like you had Lyft. I mean, it's like, that's great. Like they like using the product and we continue

to evolve it and all of those things does not mean that your product market fit, like just because Lyft uses it does not mean that more people will use it and pay for it. So it is an odd situation to start a company. There are major pros and cons, right? I mean, it's like the, the pro obviously is that you have a big customer at scale, you know, that lets us vet the thing and like make sure that it actually works and get good feedback. You know, the con is that you spin out supporting a pretty large customer while you're trying to scrounge around for like initial sales and initial product market fit and all of those things. So, um, you know, I don't know that it's the way that

I would recommend that people start a company. Um, but I don't think starting a company is ever super clean. So it's like, you know, you just, you know, things, things happen as they happen. For someone who might be in a similar position where let's say they were building something at a company and wanted to do this, but wanted to be on good terms and ideally have that company as a customer too. Any lessons that you could share that they could follow? You know, be prepared to spend a lot of money on lawyers. No, well, yes. Yes, yes. And no,, but no. Yes, and. No, you know, I mean, it's like I think that communication is really the only thing, right?

I mean, it's like you're never going to make something like this happen by going behind the leadership of your company's back. It's impossible. I mean, it's like you can't, for example, go out and be like, well, I mean, it's like, you can't, for example, go out and be like, well, you know, it's like, I can raise this funding around and like, we're all just going to quit, you know, and like, F you. You know, it's like anything else in life. It's like, you have to have like really open and honest communication. So unless you're prepared to really like leave and walk away from all the ip and all the stuff and face potential lawsuits and litigation and like whatever else um the only way of approaching it

is to have really good open and honest communication and start talking to people about whether this is possible and whether this is something that the that the company would be interested in and um you know what i what i will that, again, I think that in the current environment, in some ways, it's better for this right now because companies are looking to cut costs. It's like they're looking to remove extraneous, non-business critical things, right? for them in some cases to shed headcount and still retain some of the existing services, right. And have like funding come in from the outside. Like that can actually be a great situation. I mean, it's like that, that like could be a way that companies, yeah. I mean, it, it, it could be like, it's not, it's not uncomplicated, but it could be a way of doing that. So I think like in all things, I would just say that communication is always key.

Like when it comes to business, it's like you have to be open and honest in my opinion. Interesting. So you've like walked through that journey of like exploring like, hey, can there be like a spinoff actually with leadership as well? It's not like the three of you kind of decided,

yo, let's, you know, we're going to do this thing thing so either it's a spin-off or with something else and then you kind of bring it up yeah i mean we we obviously had plenty of private conversations as to whether we whether we wanted to do this or not but you know ultimately we started having conversations with the the the old ceo of left at the time, not the new one, because that was during the transition phase. But we had conversations with the CEO pretty early on, right? Because it's like, there's no way that it's going to happen unless that person wants it to happen.

So those conversations have to happen. So bit of a pivot, I saw that you still made co-commits to envoy uh from time to time which i thought was quite cool um i assume you're not super involved with the project itself anymore um what was that transition like it's been a slow progression over time um i i think like we open sourced envoy in 2016 you know and then um it was a pretty rough few years for me of just, you know, like burning myself to the ground of making that project happen. And then over time, you know, the project has become a lot more mature, and we have, you know, a much bigger stable of maintainers. And,

you know, so like a big goal of mine in the 2018-19-20 timeframe was like like trying to make myself a less critical point of failure. And I don't write much code, if any, to Envoy anymore. I still am involved like from a management perspective, you know, it's like I'm behind the scenes, I'll whatever help if there's a dispute or some other thing. And like, you know, I'm still involved if anyone has a question. So, you know, I don't think it's actually dissimilar to any other situation. This is what I was coming back to before about saying that like doing open source to me is kind of like running a company. I mean, if you're running a company, you don't want to have a bus factor of one. It's like, you know, you like want to have a distributed set of people that aren't burning

themselves out. So it was a goal for of people that aren't burning themselves out. So it was a goal for me to make myself less important over time. And, you know, I've done that. And if I were to completely walk away, I think the project would be fine. I don't want to because it's still important to me. It's just being realistic about my time, you know, in the sense that I'm doing the startup, I'm pretty busy. And there's just like, there's only only so many hours in the day. So one aspect that you just mentioned about burning yourself down to the ground, like in

three years making this project happen. There have been times in my life where in my job, I would work crazy hours just because I want something to happen. And there have been times right after that where I'm like, hmm, I wonder, was that the right trade-off to make? What's been the case for you? Like, would you do it again? I would. And I'm one of those um, do I think that everyone needs to work 80 hours a week for life to be successful? Absolutely not. Um, I, I, I believe in a healthy work life balance.

I also believe that to do big things sometimes takes big sacrifices and that is just the fact of life. And I think people that believe that they can do big things without at times pushing themselves, they are completely delusional. And Envoy would not have happened in the way that it had happened if I had not basically burned myself to the ground. And sure, there's lessons learned, and maybe some of the stuff could have been better and like whatever else, you know, things are always better in hindsight. But would I do it again? Absolutely. Was it possible to do without doing that? I don't think so. Because when you're starting something like that, you know, and you're like,

effectively doing it almost outside of the normal working environment, it's like, yeah, okay, like lift support to me open sourcing it. But no one, including myself, knew what was about to happen, you know, in terms of the time commitment, or the like the pressures of like as things blow up and and then now you have this divergence of what's important meaning like important to the project versus important to lift and the most difficult time and again i'm just being really honest is that the most difficult time was probably in the 2017 time range because the project hadn't blown up enough to to make me relatively well known you know and then to have i would say more leverage over to lift um so it's like you're kind of like in that intermediate phase where like the project isn't important enough like i'm not

important enough you know but it's like i'm killing myself like trying to make it like make it successful so you're kind of doing that like outside of working hours or just like trying to do it within working hours and then at a certain point you know the project becomes successful enough that like the personal leverage increases such that I can almost force like better, better, better work life balance, if that makes sense. So like, it was that beginning phase that was really, really tough. And again, would I do it again? Yes. But it was it was a difficult time in my life. What changed, by the way, after you became well known, and this is at Lyft and outside I think you know I think what changed again if I'm being honest is just that I had a during that era of time it was still useful for

Lyft from a recruiting standpoint to like have me be there right but at the same time i had more leverage in terms of being like you know what this is mostly what i'm doing it's like i'm mostly just doing the open source stuff i'm not really gonna do as much for for lyft like that's the kind of leverage that i'm talking about so instead of doing two jobs now i'm now i'm like down to doing one job so my so so my life like instead of working like 70 to 80 hours a week i'm like working 40 hours a week right um and and i think i think that's what changed and yeah um you know and then obviously you know like things always ebbed and flowed in terms of my involvement with like lift technical stuff and with and with envoy but i think it was that

like 2017 ramp that was it was it was really difficult i mean i i yeah that was a very tough time of my life what kept you going during that like the initial like the phase that's really hard before you got the leverage i think what kept me going like as, as I said, is I'm almost entirely, 99%, I'm impact driven. Like, I do not care about writing code. I care about code being used. And I think in 2017, it just started becoming clear that, like, the thing was blowing up, you know? I mean, it's like, and, again, you can never know the alternative history of things without reliving that history. So like, I don't know what would have happened if I had done X,

Y, or Z, but I believe that I was so involved. I mean, it's like, I was going to conferences. I was like selling, like I was like going to companies to give talks. Like I, people would like raise a bug. I'd fix it like that day. You know, I mean, I just, I was invested because I believed, I think rightly so that the more involved, the more invested I was, the more likely it is that people would end up, would end up using the software. Um, so for me it was, it was, that's what kept me going. It was being impact driven and then you

know it's like during that time frame it's like google started using it you know and then like the rest is history and it's um but like it wouldn't have happened without just like a lot of pushing and that pushing was tough on me um Thank you for recognizing that, by the way, or acknowledging that. Oh, sure. I don't know. When the topic of work-life balance comes up, I have, I don't, personally, I don't believe in work-life balance. I think you should work on something that you're passionate about and that you don't

worry about work-life balance. Yeah. I mean, I think that, look, it's like I have family, I have kids. I mean, it's like there are competing concerns in the world. And that's an interesting topic because I have thought about this before, which is and would envoy have happened after my son was born i'm actually not sure um just just be just because of like the amount of time and just like physical and mental energy that actually went into that and in those first couple of years that would it have happened in the same way if i was trying to balance like more family life? I'm honestly not sure. And, you know, when it comes to work-life balance, again, it's like, I also believe in working on things that I'm passionate about.

But, you know, we are human and we burn out, right? You know, so it's like there's an energy budget and we have to make sure that we're in it. Sometimes it's a sprint, but often it's a marathon. And I actually think that sometimes people do the sprinting thing and then they're out. Right. When in fact, like it's the marathon that we're actually running. By the way, after Envoy picked up and you saw Google using it, people knew you as someone who built Envoy. Did any surprising opportunities strike at the time? Like any LinkedIn spam that you wanted to pay attention to?

I mean, you know, as you already alluded to, I mean, during the early days, there was a lot of investors like trying to, you know, get me to start a company. And, you know, that was one path and it probably would have been lucrative. But it wasn't the decision that I, that I made at the time. But no, you know, there's nothing like super notable. I mean, it was just, it was a very, very busy time. And it's not like now isn't busy, but starting a company with a team is different than like when i started envoy yes there was people around but i was doing like 90 of the work myself obviously quickly it you know became more of a team effort that's fantastic um but like

in in the beginning you know it was mostly me and that's different than like doing a company with a team so last question on this um open source stuff so uh in a previous interview you mentioned right like one of the most important things that made envoy super successful was like the open source community or like how open the community was that y'all built and this like ethos of basically never saying no and you described this earlier as well, like getting the bugs fixed on the same day. Like this is very cool, but sounds like super difficult. And I'm kind of curious,

like why was this like so important to you, this ethos, like even on day one, like before you had all this traffic? So it's actually funny you ask that because I think this actually also comes back to my time at Microsoft and Microsoft obviously, you know, had come up building a platform that people wrote lots of software for and all this other stuff. And Microsoft's ethos, at least at the time that I was there, was always like extensibility,

extensibility, extensibility, extensibility, right? It's like the way to build a platform is you, is you build all this hooks and then people start writing, you know, it's different. Like people start writing software for all of those things. And then, you know, you have this platform and then everyone uses your platform and you make tons of money. Um, and I, I saw that being effective. And then, you know, I think when it came to Envoy, obviously there's a practical reason for all of the extensibility. It's like, well, you want to do all these things and you want to keep the code clean and, like, you know,

have a core and then have all the functionality. But then you start realizing that people come in with their concerns and, you know, you want them to have their concerns satisfied and there's two ways of doing that like the first way is you you know somehow build their concerns directly into the core code or you make it extensible and you know you you allow them to solve their own problems so I think I think in in the beginning I just believe it would make the project more successful if, if we actually, you know,

made it so that people could generally solve their own problems. So, yeah. Well, before we wind this down to a close, you mentioned earlier that Lyft was pretty insane to let you work on Envoy. And they probably shouldn't have done that. Why is that? I mean, more that when I joined Lyft, the company was less than 300 people, and there was maybe 75 engineers. It was a pretty small company.

And if you look at what i was working on initially which is mostly just you know during that year uh like microsoft let's try micro microservices are going crazy and all the stuff it's like everyone's got these awful microservice architectures and they're all broken and you know so i i was mostly there just to help them with their microservice architecture rollout. And from previous experience at Twitter and back at Amazon, it was pretty clear that having this type of proxy infrastructure would be useful. And then obviously we have some experience in NGINX and kind of like knew that we could do something different or better.

But it was a small company. I mean, it's like in a sane environment, people would have been like, no, just whatever. Use NGINX, like use HAProxy, like go and go and go and deal. Make it work. Make it work. I actually had some suggestions to write it in Python, you know, so it's like not not joking at all. And, you know, so write some proxy in Python or something like that.

And I think in like a sane environment, you know, that would have been the way to do it. Now, like why did I do it? Well, because I knew, you know, I knew the head of engineering who's actually my co-founder at Bit I knew the head of engineering was actually my, my co-founder at Bittrift, um, was the head of engineering at Lyft and he had actually been my boss back at Twitter. Um, so it's like, you know, so there's a, there's a long, long history there. So it's like there was

shared understanding of capabilities, like based on what had been done before that certainly made that more possible but at the same time you know we're coming into a tiny company and being like we're gonna like do this thing this seems crazy right um yeah yeah it seems crazy yeah but look you know i mean i i think if you look over the history of software, a lot of big things are done by some pretty crazy situations. You know, it's like sometimes you just go and build something and like some cool stuff happens. Well, I would say Envoy is very cool. As you said earlier, it's used almost everywhere.

So thank you for pushing for Envoy, building it, making it available to everyone. And Matt, thank you so much for joining the show today. We'll link everything we discussed in the show notes, especially Yeah, great recent blog post you mentioned about observability as well as bed drift. Is there anything else you would like to add before we wind this down? I don't think so. It was a great conversation. Thank you so much for having me. Awesome. Thank you so much. Hey, thank you so much for listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremisadventures.com. You can also write to us at hello at softwaremisadventures.com.

We would love to hear from you. Until next time, take care.