Software Misadventures - Spoons (Daniel Spoonhower) - On building Lightstep, being customer focused, developing systems at Google scale and much more - #12

Episode Date: July 9, 2021

Spoons is the Co-founder and Chief Architect of Lightstep. He joins the show to talk about building systems at Google scale and various aspects that make Google a weird place than other companies. We talked about Spoons's journey of leaving Google and deciding to join Lightstep as a co-founder. We dig into the challenges during the early days of Lightstep and discuss the importance of speaking to customers to build the right product. We talk about what it's like to start a family and run a startup and how one can be intentional about building a company’s culture. As always, we go through some of the misadventures and one of them involves a cable being cut under the English channel.

Yeah, looking back, the thing where I don't think you have that flexibility is thinking about who your customer or your audience is. So you can't disagree about that, or you can't kind of waver on that from week to week. It's okay if you don't know exactly what they want, but you should have a real customer, not a proxy for a customer, like an actual customer in front of you and like like i said like they might and like you were saying like they might lie to you right or you might not have the information but like you should be able to go to them every week or whatever whatever your your your cycle is and like get more information from them and hopefully get closer to what they actually want welcome to the software misadventures podcast, where we sit down with software and DevOps experts to hear their stories from the trenches about how software breaks in production. We are your hosts, Rannoch, Austin, and Guang.

We've seen firsthand how stressful it is when something breaks in production, but it's the best opportunity to learn about a system more deeply. When most of us started in this field, we didn't really know what to expect and wish there were more resources on how veteran engineers overcame the daunting task of debugging complex systems. In these conversations, we discuss the principles and practical tips to build resilient software, as well as advice to grow as technical leaders. Hey everyone, this is Ronak here. Our guest for this episode is Daniel Spoonhauer, better known as Spoons. Spoons is the co-founder and chief architect of Lightstep. Prior to

Lightstep, he was a staff software engineer at Google. Guang and I had a lot of fun speaking with Spoons. We learned about building systems at Google scale and various aspects that make Google a weird place than other companies. We talked about his journey of leaving Google and deciding to join LightStep as a co-founder. We dig into the challenges during the early days of LightStep and discuss the importance of speaking to customers to build the right product. We talk about what it's like to start a family and run a startup, how Spoons defines personal and professional success, and how one can be intentional about building

a company's culture. As always, we go through some of the misadventures, and one of them involves a cable being cut under the English channel. Please enjoy this delightful conversation with Spoons. Hey, Spoons. Super excited to have you on the show. Welcome. Yeah, thanks for having me. It's great to be here. So, I think when we were corresponding over email, and I was telling Guang that,

hey, we're going to have Spoons on the podcast. The first question Guang asked me was, so wait, you said Spoons? I was like, yeah, yeah. Guang asked me, what's his real name? I was like, his name is Daniel Spoonhower, but he goes by Spoons. So, we thought we would ask you, so wait you said spoons i was like yeah yeah like gong asked me what's his real name i was like oh his name is daniel spoon howard but he goes by spoon so we thought we would ask you how did that happen like how did people start calling you spoons yeah i mean everyone in my family to some extent i mean if your last name is spoon howard like it's gonna happen at some point um so we all use it now and then um and it was like a funny joke for my friends in high school to like, call my house and ask for spoons. No, but there's really like the thing, the two things that happened to my first job out of college, I worked for a guy named Dan. In fact, I had worked with him earlier as a TA on a on a course um that that we that he taught at cornell um and i think there was two tas named dan and then the and the instructor also too right so he was like well i'm

dan so you two go figure out your own nicknames so then i worked for him also so like everyone yeah everyone at the startup that i knew or they you know that that knew i was like knew him as dan and then i was the other guy. And then my first boss at Google was also named Dan. So there was like, and in grad school, I think in my group, my advisor had four students, three of us were named Dan. Like there was just a lot.

So eventually there just became, there were more people that called me spoons than Dan. There still are a few, like besides my siblings and my parents, like my friends from like my first week of undergraduate, like they still call me Dan sometimes, but no one, no one else does. Even my five-year-old, like if I don't respond to dad quickly enough, she starts yelling at spoons. Is it natural for you to respond to both at this point? Yeah, although I'm probably faster in most contexts to respond to spoons because there's like such a

small number of people that know me as Dan. If you call me Dan and you're not one of those, then you probably don't know me at all. That makes sense. Well, since you touched on academia, you got a PhD from Carnegie Mellon, if I'm correct. That's right. And you also touched on being a TA at one point. So back when you graduated, did you ever consider academia as a career choice? Or were you always attracted towards like going towards the industry side? Yeah, so I guess I did this twice, right? So when I graduated from undergrad, I had applied to grad school and gotten in and then decided to defer because my mentor was

basically starting the engineering group at the startup. And I thought I would go work. It seemed like a, you know, an opportunity that doesn't come all the time. So I'm going to go do that. I deferred at CMU that one year rolled around and I it's like, well, I've only been here a year. I kind of want to stick around for another year to really build something, but they didn't allow me to do deferrals. And so I had to be like, you know, I'll only been here a year. I kind of want to stick around for another year to really build something. But they didn't allow me to do deferrals. And so I had to be like, you know, I'll reapply in another year. I'm still excited about grad school, but I just want to kind of see this project through. And so another year went by and I reapplied, but I only reapplied to CMU, which totally freaked my boss out.

He's like, don't do that. Like, you know, I didn't want to steal you from academia, but like all of your eggs, it's like one basket. But it was fine. It all worked out. So I was excited to go back and, and to do that work. Um, and I graduated from CMU in 2009 and the choice there actually is like a little bit more like just context dependent. Like 2009 was kind of a hard time to get a job generally. Um, and I had just spent a bunch of time working in a pretty isolated way, not only just like working on trying to get my dissertation done. But my wife was actually working at CERN at the time. So I did most of job offers in academia,

but I was just really, where they were and how they were situated, I was just worried that I was going to get stuck by myself. And the thing that I kind of got from that is I really wanted to go work with a team of some sort again, and that's kind of how I got back into industry at that time. I see. So you joined Google, I think, in 2009, which some people might feel like still early days. I think YouTube was acquired around that time. No, yeah, not that early. No, not that early. Yeah. Like, what was life like at Google

as an engineer at the time? Like, what did you work on? Yeah, so I worked in the New York office, which is a little bit strange, because it was a satellite office, although I think there were 1000 people that worked in it. And so it had a lot of its own inertia. And I, my PhD is in programming languages. And the guy that hired me saw that and was like, Oh, yeah, we've got one of those, you should definitely work on this configuration language. And so like, the first thing I did was work on this crazy configuration language that was used to schedule jobs on google data centers um and uh but i mean it was interesting as like a infrastructure point of view and even as a programming language research point of view

because it turned out we had every every program that had been written in this configuration language was all in one repo, all 200,000 of whatever there were. I don't know. A lot of them were machine generated, as it turns out. But if whenever we wanted to make a change to the implementation of the interpreter, we just run it over all of the programs

and see if it changed anywheres in ways that we didn't think were good. But it was kind of weird from a language design point of view or like a language implementation point of view. You don't usually get that opportunity. But I think overall, like Google was still a place that you could experiment in lots of ways.

I think, I don't know, for me, I guess it's a little strange. I never have really worked at big companies or old companies, I guess. I have done internships at places like Microsoft and Intel, but I don't know. My experience since then a little bit has been like, oh, that was weird because that's Google. I don't know. Maybe that's what I should start with. Google is a weird place. I think that weird means many different things for different people who work at google like depending upon the team they're working on um so i do like how did you get involved with

uh just like distributed tracing side of the world uh i'm considering now at lightstep yeah so i sat next to ben sigelman who's one of my co-founders at lightstep that's kind of how and we had the same boss one of these dance um so that was sort of how i got involved um i didn't work too directly on on tracing itself because by that time actually when i joined ben had also already moved on to another project and the tracing project was actually in the kind of maintenance mode for the most part um that was being used for a bunch of other things um but yeah i think we the, the thing at the time, I mean, generally as we, in the infrastructure team, we're really trying to figure out

how to make Google infrastructure easier to use. A lot of it had been built for these very big and, um, profitable teams like ads. Um, but there were a lot of other teams. It turns out most of the teams that weren't super happy with it. And so, um, like the, a lot of other teams it turns out most of the teams that weren't super happy with it and so um like the a lot of the work on the configuration side that i did um the work that ben was working on at the time um which was called monarch google's um the the sort of second generation of monitoring infrastructure a lot of the goal of that was actually to make it much easier for people to use and make it more of um almost like a self-serve product that is a little bit more like the developer experience you might expect on the outside i see makes sense uh so since you and ben both both met at google

and then both of you in addition to there is one more man who co-founded light step uh like how did that all happen to be like how did this idea was how was this idea formed uh did all of you just got together one day and said hey we should do this we should start a company around this thing yeah so the way the story went is that sigelman left google because he google was his first job out of college and he didn't want to be there forever and he had some idea that he wanted to start his own company but he didn't think he should just leave immediately so he went and he got't want to be there forever. And he had some idea that he wanted to start his own company, but he didn't think he should just leave immediately. So he went and he got another job for another startup just to kind of, you know,

learn what it's like to work in the real world. And then after that, then he did, he founded a company actually, maybe, I don't know if I told you this some other time, but like that was actually an anonymous social networking app. Oh,

I didn't know that. It was like secret if you remember secret but nice so it wasn't just about complaining about your co-workers it was like the whole idea oh yeah it's like you yeah there's one like called blind right now or something like that right now i think yeah yeah yeah so this one was called matter and that was totally anonymous like it didn't even know who your contacts were it would just you could write about what you were feeling and it would show it to the right people. And they could support, this is back like before,

back in the days when you can only like things on Facebook. And so they spent a bunch of time figuring out like how to like laugh or hug or support, like different verbs made sense for different contexts. So they did some cool NLP stuff. And he hired, um, the other Ben,

Ben Cronin as, um, one of the engineers on that so they were doing that i he showed me it at various times like he showed me mox for it stuff i was like oh this is cool it's cool i mean he's like oh do you want to come work on this i was like not really i don't really do social networks like i like don't i didn't have like a facebook account i still don't have a facebook account like um i don't know i'm old or something i just don't have a Facebook account. Like, I don't know, I'm old or something. I just don't like using technology in that part of my life or whatever. And so they went off and did that.

And then the kind of end of the story is that Sigelman's first kid was born and I went to take him over some food. And I was like, oh, how's your startup going? Like, first kid, startup life, woohoo. He's like, oh, not that great. I think we're going to do something else. And I was like, oh, what was like oh what he's like oh i think we're gonna do some sort of like infrastructure monitoring some sort of enterprise software kind of thing i was like oh that sounds like fun like we should talk yeah um so that's kind of like how we got together and

then and he was like oh do you mind if we bring if i bring my co-founder because basically he and the two ben's had really enjoyed working together but agreed that like that matter was never going to make it um it turns out like they had incredible incredibly high retention but almost no network growth and so it's hard to run on social network when you're not when you have zero percent growth is that a commentary on how people are not very nice and there's only explosive growth when you're allowed to say bad things about it might be it might be yeah no no they did they did all these interviews with folks to try and understand how they were using it and you know people would say like oh it got me through this hard time you know it really allowed me to connect with people you know

it was really it was really important and then they'd ask well well how would you describe it to your friends people say i'd never tell my friends about this it's way too personal that might play a factor in the growth i suppose yeah that is interesting because once i guess it gets more commonplace it's not as special right and then you want to hold on to that and that's very much anti-growth interesting yeah so the thing i think about the beginning of lightstep and maybe the back to like the google component of it or whatever of whatever is like google is a weird place like i said like google operates at a scale that almost no one else

does and so i feel like i've seen a lot of other um folks who tried to take some idea from google and and make a company out of it or even inside of google we were trying to figure out how to take our internal tools and make them into products on it. Or even inside of Google, we were trying to figure out how to take our internal tools and make them into products on the outside that people wanted. And it's often, I think, a crazy idea. But what we tried to use our experience is like, Google can be kind of looking glass into the future in a lot of ways. Google was building microservices of some sort in 2004 or 2005,

even if they weren't called that. And so all the pain that those engineers have been feeling, you know, since then, like, that's actually real data that we can use. And what we were starting to see when we kind of talked to our friends, and as we started to kind of think about the market is that people were feeling a lot of those same kinds of pain points. And that's where we thought, like, okay, well, we know what's coming next. And we're going to have to adapt the solutions and think about the technology in a different way. But that experience in terms of what tracing can provide and where to go from there, I think, was really what made us excited at that time. It is really interesting you say that because I think I heard that quite a lot.

And I think the difficult part is timing. Because even if you look at Hadoop, MapReduce, and Databricks now, but it's like so many years, right, since the paper was first published at Google. And how did you guys, did you guys, was that sort of part of the, you know, thinking process when you guys like, hey, is this too early? Like, is this the right time? Yeah. I mean, I think we picked tracing a little bit because that's kind of was the new thing. And we actually, um, and we went into our series,

a fundraising, we were totally open. We said like, we don't know if this is going to like explode next year or not. But, um, so just like FYI, we might not spend this money for a little while and you guys had to be okay with that. Um, and that was actually like why we were excited in the end. Like the partner we found at, at red point like i think totally got where we were and saw where we were going but agreed kind of on the on the trajectory and the approach that we wanted to take and we wanted to be ready you know when other folks were ready but we didn't we didn't know exactly when that was going to be interesting that's really cool to

have that optionality i was recalling a conversation uh the spoons you and i had at, I think, SideGlass in San Francisco in 2016. You were generous enough with your time and you were telling me about distributed tracing and how cool it is and the kind of problems it can solve. At that time, I remember thinking, this is really cool. But to be honest, it took me four years working as an SRE at LinkedIn to truly appreciate what you were talking about. Yeah, 2016 probably was a little early for a lot of people, I think. It actually is kind of interesting in that we agonized over the name, over LightStep. And we really, we didn't want to call it tracing in the beginning, because we didn't think that anyone would care about it. And we also didn't, you know, want want to box ourselves in but there was a point um kind of in maybe 2018 um early 2019 when tracing really was

a thing that people would ask for like we would go and be talking to people and they would say do you have tracing is that is that is that what you do yes yes that's what we do yeah um but that i think that has changed again a little bit lately and that i think people understand that tracing by itself is not the only thing right so we people talk a lot more about observability now and um and maybe a little more of a holistic approach to things yeah uh i would imagine like back in 2016 2017 uh the tracing or even observability as a term were new for many people like these kind of problems were relevant at like you mentioned google scale because well it was already going through those pains of having to operate hundreds of microservices and how to kind of look through

these things and debug and operate them now i don't know if it's fair to use this word, but it's more commoditized in a way, where you see tracing as, if I just go on Google and search tracing, I see a lot of options that a lot of APM vendors kind of provide tracing in addition to what they already do. So like initially, when you were talking to many of the potential customers or just looking at the market, what were some of the initial challenges that you saw? Yeah, I mean, I think initially we could go to folks who already had some background or context on this. I think we got some advice from an investor at some point early on, which is like, don't go to try to find people and convince them that they have this problem, like find the people whose hair is already on fire. And I think that's certainly what

we did for the beginning. I think, you know, it helped that actually that Twitter had open sourced their tracing implementation, because that was a thing that other people had used or people who had come from Twitter, and gone on to other organizations were used to the kinds of problems that it was solving. So we could sort of start the conversation from there without having to start from, from what tracing is. And we could, we could jump immediately to what Lightstep could provide. Um, but yeah, I mean, I think that it's always, I mean, even, I think even today, to some extent, tracing is still kind of an expert tool in a lot of ways. And I think if you're talking to an expert, it can be easy,

but there's a lot of developers, I think, that actually need tools to understand what's happening in their production systems that haven't necessarily used tracing before or don't have that state. So I think it was certainly a challenge back then, but I think it still is. Makes sense.

Talking about your decision to join LightStep, can you walk us through what was that process like? Like you're at Google, you, I'm assuming, were enjoying your job. And then when asks you to join Lightstep at the time, which was still new and early, like you were still figuring out what to do exactly. Yeah. What did that process look like yeah i mean i i mean the thing about google is like it's a great like the scale that's there offers a lot of opportunities to do kind of crazy things um at the same time there's a lot of risk for google when you're doing new stuff and so there's a lot of really smart people at google that are there to help you not make mistakes. And like with the best intentions, right?

It's not like they're mean or anything like that. But they really like there's that's their job to help you not make mistakes. And I was just feeling a little bit like I wanted to make some mistakes, like not in a bad way. But I just wanted to like try some things without knowing if they were going to work or not and like learn from them. So for me, it was a little bit just like taking a different approach to stuff in a way. Um, and I think the, the bigger part of it was just, um, like I said, there was, um, like, like basically the bigger part was maybe trying to get a little bit closer to customers. So I spent a lot of time working on internal infrastructure at Google and working with the big teams and trying to find the small teams that I could get some attention from.

I ended up moving into Google Cloud because that was actually a little bit easier to see who your customers were and figure out what they needed. But I think in the end, I just wanted to, you know, maybe I should have become a PM or something like that. But I just wanted to like talk with customers more and see where their pain points were and then try to help them like that. But I just wanted to like talk with customers more and see where their pain points were and then try to help them through that. And I guess Lightstep seemed like a place to do that. I don't know.

I was excited to work with Ben again. That's like an opportunity I didn't want to miss. I meet this new guy, Ben that I didn't, that I didn't know. And I guess, I don't know. I like,

I like building things and not, but not just software things. Like I like building organizations too. So for me, LightStep was sort of, I had been at the startup before grad school, right, where I was like the fifth employee or something like that. And I had seen a lot of stuff grow as an early 20-something. And now I kind of felt like I had an opportunity to do some of that myself as like a leader in that organization. I think it's kind of rare to hear engineers talking about, oh, I want to get closer

to the customer. Like, was there big events where, you know, maybe something didn't ship because there was a big mismatch in the sort of the product spec versus what customers actually wanted that sort of shaped that sort of desire to move closer to the customer side? No, not that. I i mean it's a good question i think it was more just like um i just wanted the ground truth right i just want to know that this is like the right way to do it and i don't want to argue about it just want to be like they're paying us for it it's got to be the right way because right this is okay when people say google is weird here's

what i mean by weird it's like as much as it's like um i don't know a capitalist monster on the outside or something it's a weird like socialist thing on the inside because it's like there's just like like ads it's just like has infinite money right and so it's like how do you like understand like the market of developer tools internally at Google when there's not like a true currency to measure things, right? So anyway, that's a very ancient mirrors take on why we want to wait. So what happens when customers lie to you? Like, I remember when I was trying to work on a startup idea, I think the most helpful book I read was like the mom test.

And then, you know, it just kind of exposes, you know, all the different ways that, you know, your customers can be telling you things to make you feel better about your shitty product. But it's actually shit and nobody's going to use it. Like how how did you deal with that sort of things? Like trying to gauge the true ground truth? Yeah, I mean mean i think like renewal conversations are maybe the main one like light stuff actually has a little bit of a weird path as a developer tool like we never um a lot of developers will start in a way where you you know you you somebody swipes a credit card and pays ten dollars a month or something like that like

we never did that like our first contracts were like we go to vp of engineering and like you know they've got a tracing initiative and we can help them do that. And I think, you know, the first sale a lot is like, especially early on, it's about, I don't know, our reputation or something like that. Right. But that renewal, like one year in is like, that's up to me. That's the ground truth, right? Because they look across the organization and they see who's using it, how it's been

used. And they've got to go, especially, I don't know, as budgets change or things like that, certain customers get close to IPO that's happened. Their finance departments are starting to ask about like these non-trivial line items that aren't $10 a month or anything like that. Right. And, and I think when those conversations are difficult, like that to me, it was like the signal where we had things that we had to fix. Got it, makes sense. I think I got very emotional

because all the bad members coming back from working on my startup idea. Cool. So maybe changing gears a little bit or focusing on the topic of building LightStep. So your LightStep co-founders were also engineers as you mentioned but you became the cto for the startup which everybody knows means that you're the better engineer you know they're co-founders i don't know if that's true i think i think everybody

would agree with that okay okay okay but but so so titles aside um so so obviously there's a ton of new like non-technical skills that you guys needed to master, like sales, community building, marketing, you know, all those sorts of things. How do you guys decide on splitting the responsibilities? Like, was it kind of obvious because now that, you know, they've worked on this idea, the social network for quite a bit so that they already have a lot of those set up and then you can just kind of come in and then hey like this is my background or was there a lot of um sort of change ups yeah it turns out that like the go-to-market plan for a social network and uh application performance monitoring are pretty different um it was different for all the different kinds of things i mean i think we all have i mean as much as we're three engineers, like we are three pretty different approaches to things and three different kind of levels of experience.

So I think in a lot of ways, Ben Sigelman, BHS, had the vision for things. And so he drove a lot of early product decisions and he did a lot of the early sales stuff um i guess i sort of i spent a long time you know hanging out with sre at google and and working pretty closely with them and i also had the most actual management experience coming in and so i think that's partly why i i got sort of a set of theTO things, which is not to say that like the other, the Ben's didn't get involved in building and even in, in, in the engineering management, but that sort of made the most sense.

And Ben Cronin also had worked on a lot of software and was like, I mean, he's much more pragmatic sort of planner than I am in a lot of ways. And so I think that's partly how we divided things. But there was other weird parts where it was just like, I think paying the bills, I think stressed them out more than it stressed me out. I just paid the bills for a long time. And like, then suddenly,

like I also was like building financial models at some point and I hired our first finance person, but only because I think they got stressed out by paying the bills years earlier or something like that interesting so for the uh sort of necessary evil like tasks that you guys had to do did you guys have like a wheel to draw or how did you go i think we just went by like who was gonna there's like we were pretty i think we were pretty transparent to each other about what we didn't like doing or what stressed us out. And so there were times when it was just like, I can't do this, Ben, but you just got one of you has got to do it again. And then there are times where it was just like, you know, they're both busy with something else. And hey, we've got to do some compliance work. Okay, I volunteer, I'll do the compliance work, right. Or, you know, maybe it was because I'm connected with the folks on the engineering team that are going to carry out a bunch of that work. And so that makes sense. I don't know. I,

I mean, like I said, we, I feel really great about them as like a founding team, in that I think there are ways that we are totally aligned about what we want to accomplish. And there are ways that we think totally differently about how to get there and that's like that's great that's like what you want otherwise it's like if you all want to do the same stuff then you'd have all these arguments about like who's in charge of what or you'd have to use do wheel fortune or whatever but um but i think for us we just fortune you know in our case yes yes exactly wheel misfortune important important game day uh uh testing but um yeah anyway so like i said i think like having folks that had different just ways of working almost made a lot of that stuff easier even if

it meant that we also had some conflicts at times about like when we had to decide like a common way of doing things um if we didn't have to decide then maybe it was a lot easier so it's interesting right like in this case you're building an engineering product and you you mentioned your desire to be closer to the customers and build for what they actually wanted to use and you could also use a lot of your experience to as as input into the tool that you were building when it came to some of these conflicts per se if if there were those were in the early days where you're still trying to hash out exactly what the right product is and the kind of small decisions also matter a lot like how did you go about navigating those or how did that look like in

general like how can one go about being productive in the in those conversations yeah i think the thing you know we we try to talk it out and but i think we just time box stuff and i think this is a place where i would probably just argue with people forever um or until i don't know until i got really tired but i think, I don't know. I think this is the thing where like, I would just argue about it forever. BHS would just be immediately like frustrated that we weren't coding, uh, and, and not even want to talk at all. And then I think Cronin had some totally reasonable thing to say, which was just like, Hey, let's time box this.

Let's spend 30 minutes. And if we can't get to a conclusion by the end of 30 minutes then um then and like we still don't agree then it's great you go spend a week building that thing and at the end of the week we'll know more and we can come back to this but like there's no need for us to actually agree on this thing right now because it would just be more productive we don't have enough information to actually make the decision if that's the case right i don't know i feel like that's a thing i did a bunch i hadn't there was an early engineer at some point who was like and who was a pretty senior engineer um and i you know fully capable of like building great stuff and even figuring out what to work on but he came to me at some point and said he's like i don't i

don't know what i'm supposed to be working on. Like, what is the most important thing? And I was like, I don't know either. The error bars are like on important are huge right now. Right. We just, we don't, we don't know. So like go pick something that you think is maybe the most important and go work on it for a week or two. And then, you know, worst case, the error bars got smaller and, um, and you'll know that there's some other thing that could be more important. We should go spend some time on that. But I think, I guess this is maybe the hard part, I think, about early startups

is you just don't have a lot of information, right? Yeah, but I also feel like it's, I feel like it's just really nice that when there is no clear answer or there's no right answer, it's just different perspectives. Being able to just spend some time on something for a week and say okay hey let's now we have either more information or

more data about what to do next that's that sounds like a very productive way of taking conversations or discussions forward but how much liberty does a startup actually have in doing these when you're early, yes, but when you're growing through a growth phase, I can imagine shipping quickly, making decisions promptly is a cost function sometimes one has to optimize for. So how do you balance these two things out? Yeah, I think the thing, yeah, looking back, the thing where I don't think you have that flexibility is is thinking about who your customer or your audiences. So like, you can't disagree about that. Or you can't kind of waver on that from week to week. exactly what they want, but you should have like a real customer, not like a proxy for a customer, but like an actual customer in front of you. And like, like I said, like they might, and like you

were saying, like they might lie to you, right? Or you might not have the information, but like you should be able to go to them every week or whatever, whatever your, your, your cycle is, and like get more information from them and hopefully get closer to what they actually want. Going back a little bit on responsibilities. So I imagine an important part of being the CTO is to like evangelize the product to developers, since you know, it is a pretty technical product. To me, the role of developer advocate is super interesting, because you can't really have huge impact, like, you know, I think aboutsey hightower for kubernetes um but you do need to develop like a different skill set than um that usually i think engine like engineers don't

have uh you know things like how do you write in a way that actually engages your audience how do you give really good talk um you know i was kind of curious like what was your journey like kind of um you know getting good at these uh sort of things and if you have practical tips for people that are maybe considering i don't know if i have practical tips yeah i mean when people ask me what i learned in grad school what i'd usually tell them is that i learned how to write and i learned how to speak um so go to grad school is the answer i guess yeah let's say it's not it's not a very practical way of doing those things though but um i mean go ahead no i was going to dig further like uh i've heard something similar from other folks who who also went to grad school that uh if one thing that i learned in grad school is how to write uh i don't

know so much about the speaking part that i've heard before is uh but like what about grad school helped with that yeah and i should be clear like, what about grad school helped with that? Yeah, and I should be clear, like, I think, I mean, talking with folks in other programs, I think it's pretty different. But CMU, like, basically, I don't know, they, they asked a bunch of, you know, whatever, 10 or 15 years ago, they asked a bunch of their, like grad students, like what their former grad students, like what was in their way? Or like, what was what was their what were their obstacles and are looking at like, where were people getting jobs and things like that and identified speaking as like a really important thing. So there is this like speaking skills requirement that was part of the PhD program. And you had to like people

failed it on the first time frequently, I think maybe at some point, like the majority of time, people would fail on their first time through it. Um, so there was, it was like, it had to be a general CS audience and you had to like prepare and actually explain things that people outside your immediate group could understand. Um,

so I think that, I mean, so seeing me as a little bit weird in some way, and I think not my, my, I actually had two advisors and they both cared a lot about writing um i think one of them actually kept like a stack of this book that he really liked on that was like a not a style guide but like a bunch of style recommendations and i think

i was having some he had like redlined some draft of a paper i was writing and he was and i was starting to argue with him about his red lines and he just kind of reached back to a bookshelf behind him I'm not even sure he looked and just like pulled a book off of it and like handed it to me and he was just like why don't you just read this and then we can talk about it and I was like oh okay um did he but I mean I guess it did it work well that the book didn't work but like many draft many drafts years later something it did work um so i i mean i think i guess there's two parts that there's like learning how to do it but i think also like learning to learning and appreciating the importance of that clarity in writing and speaking i think is something that I'm really happy that they hammered into me or whatever at the time.

I mean, I like doing those things too. So maybe like how my journey a little bit was not like such a hard journey. Like I, I like talking about what we've built. I like talking about what I think can make things, you know, better for SRE or for developers or things that I've learned or things that I've failed at right in the past. So like getting involved with the sort of developer advocacy was like an easy choice for me. And I guess, I mean, the hard part is more like, I like Twitter is really hard for me. It's just like, it's's so short and so like in the moment i'm always agonizing over like you know character 75 or something like that and like i just i don't think in these like short bursts or something like that and that's actually i don't know like i feel like a big part of where a lot

of developer advocacy happens these days so i don't know maybe I should have maybe my advisor could have helped me with social media more how does this dude look? 128 characters Twitter is certainly hard I think it's just I think it's a skill set being good at Twitter too depending upon the role

I think people use it as a portfolio as well I suppose skill set uh being good at twitter too uh depending upon the role i think uh people use it as a portfolio as well i suppose especially in like you mentioned dev advocacy roles yeah yeah i mean there's like different things you can use it for but certainly i think some people drive awareness with that with that medium yeah uh so so thinking about like uh some of the discrepancies we kind of alluded to between Google and LightStep. At Google, you mentioned the scale is like nothing. Well, I am not aware of anything that operates at that Google scale for the most part from an infrastructure standpoint.

So a lot of the decisions are technical decisions that one is making. The cost functions one is trying to optimize for are like, hey, is this going to scale for Google? We need to meet these reliability guidelines. And I have one of the ex Google employees showed me a video on YouTube about, I think broccoli man, where they just want to serve a static file. So thinking from that experience to, uh,

to a startup at light step where one is you're still building the product uh the scale is obviously not as big as google and the trade-offs that you think about are very different as you mentioned in some cases like there are some mistakes you kind of want to do i mean not intentionally but you know try things and see yeah yeah so how how did that look like in practice and how did you kind of go about coming from a lot of that experience you and ben both uh from google where you you knew how to build software in a google way and then you have to switch how you think about things in general so how did that look like overall yeah i mean one thing that was really hard is like at google you could go to some random team who who you know, built a piece of infrastructure you could use.

And you could just ask them, like, is this a good idea for me? Is this like fit my use case? And like they would tell you no, right? Because they don't have any incentive for you to use it. But if you like, I think one of the things I missed about that was like, okay, well, we've got to find a database to put time series in or something like that. And it's like, if you go on the web, it's like, well, a bunch of these things say that they're good for that but like are they really like right it's like i can't tell if that's just the marketing speaking to me or not right um but i mean speaking of time series database this is a thing that we've chosen many solutions

to i think we put our time series in my sequel originally which is an insane choice from a lot of ways but it's like really easy to install my SQL. So like, it'll probably be fine. Um, and then, yeah, you just have to plan out like when that's going to break and, and, um, yeah, I think we've done that four or five times now. Um, like moving, I mean, we moved to some, um, I think we put it in Cassandra for a while. Um, and we actually, then we moved it to Spanner, which is like not a cheap solution from a dollar's point of view,

but Spanner scales like crazy. And so like the main thing at that point was just like, look, I don't want to be in a place where like we had, I don't know, I'm not, we're not Cassandra experts or anything like that. And like our, one of our, not our customer facing environment, but one of our, not our, our customer facing

environment, but one of the other ones had gotten into a bad state and it took like eight hours of downtime to like fix it. And I was like, like, this is a bad place. I didn't want to be here. Right. And so it's like Spanner, the, you need scale, like, here's the button you click to get more scale. And that put the control back in our hands. Right. Then it was like, okay, we don't have a forced migration of this in the future. It's always going to be a question of dollars versus engineering time. And at some point that question came up, right? And it was like, are these dollars worth it?

Is there something else that we can do? Spanner is also not like great for time series for the kinds of stuff that we were putting in, but it worked fine from like a latency point of view. So, you know, go for it. No, like speaking of that like do did a lot of the social capital or like network from google helped uh in those regards like uh in terms of like hey you know we're thinking about doing this like would that

be a good idea or are they very much used to working at google scale so they can't really help you with a lot of that trade-off yeah i mean at least at the time like everything in google was like their own thing right and so like none of the people inside knew anything about what was really or not almost no one inside knew anything about outside so um you could kind of try to explain it to them in terms of oh you know it's like dremel or something and they're like oh yeah that's probably a good idea but like then you already basically built the entire mapping in your in your head right Right. First one.

Or first principle. Oh, reasoning. Okay. Yeah. Yeah. Yeah, exactly.

So, um, no, I mean, we said some networks of people on the outside, they could help with some stuff, but, um,

yeah, I, I think, I mean, it was more like when it did come to the point where, where the money started to matter more or when reliability started to matter more i think then actually the first principles did help because then we could sort of go back and look at like yeah how should we be building this stuff um and just the experience i

mean the mistakes that people had made in building these things for google or i don't know there's like a yeah this is like weird things about the way google wants to build these systems that like don't need to be carried out in the real world um i think there's some i think people usually attribute this to um i think to jeff dean but like i think the the the quote anyway is something like you shouldn't try to build a system that that spans more than three orders of magnitude for any for any measure that you care about, right? Because if you try to handle, you know, 1000 requests per second, and a million requests per second, either it's going to be like, way too expensive to do the million requests per second,

or you're going to strip it down to like, no features that the people that 1000 requests per second actually want or need, it's going to be way too hard for those people to use and so like just try to think about that scale that like there's just limits to how far out you can go and things and i guess that informed a lot of our decisions because we were coming from google that was doing billions of requests per second but you know no one else that we talked to was above 10 million or something like that. Right. And so like, it just didn't make sense to kind of use the same, the same sorts of solutions. I see.

I see. I see. So other than the technical stuff in terms of like a culture, like I think Google is pretty opinionated in terms of, you know, what the sort of value and culture that they want to build and preserve. When you guys were building out a light step, were there any specific values that

you really liked and that you really intentionally carried over? Yeah, I mean, I think one of those was this customer drive. I think that's a thing that wasn't true everywhere. I mean, there are people at Google that do care about customers, but there are lots of people that are just excited to build cool stuff. And that is not something that we have time for at a startup. I think, what else did we think about? I guess we used to talk a lot about beginner's mind. Like I think part of maybe knowing your customers, like that they might not be as much of an expert in distributed tracing as you are.

And I think we were pretty careful with our early hires to make sure that we weren't just hiring people that had built distributed tracing systems. They were also hiring people that had used them, right. Um, or it just worked on other kinds of software so that we actually kind of got people because we were relying on ourselves as, you know, as, as beta testers, um, that we got different perspectives on, on like how the product would be used. Um, yeah, I mean, I think, you know, we, we were never like, um, I don't know, we're, we're all kind of old, right. Ben and Ben and I, like, we didn't ever have like, uh, a culture of like, we're just going to like eat every meal together. And like, I don't know, like,

you know, Sigelman had a kid at this time, like my first kid was born, like, I don't know, whatever, a month before our series A. So like we had places to go to. So I think part of it is like we did have kind of this work culture of like that we are like excited to work with these people and like that the people are really important and the team is really important but we also like have a space to go do other stuff and like um that like that professional success is not like our only measure of personal success and i think that that that's been true for for for a lot of people. I think, um, there are a lot of other founders that we would kind of talk to that. I mean,

not that all founders are jerks, but they just seemed like so focused on their thing. It just didn't seem like that much fun to hang out with them. Yeah. Yeah. I think that's a lot more healthy than startups that are very like focused on like family, which I feel like there's so many things that just don't really,

you can't really make that analogy that it just seems very maybe i'm just too cynical but i mean i think it actually helped us in a lot of ways because i think companies that form like these social groups where people just only hang out with each other like they get to a point where that doesn't scale right and so if the culture is based on this mutual trust derived from you know sharing meals and just like going to movies and whatever all the time together, that you reach, I don't know, whatever, 30 people, and then suddenly, it's like, you can't do that with everyone anymore. Or you're like, you have to hire people that have other obligations in their life or something. And then it's like, what is the basis

of trust anymore, your culture, whereas like, for us, like, we never relied on that, right, we had to kind of build it through other ways. So a lot of the i feel like actually we we got through a lot of those early scaling things um a lot easier than maybe than i don't know maybe other companies had i think the the follow-up i was originally going to have is sort of like how do you be intentional about creating these cultures and values but i really liked how you said um you know in terms of getting people that care more about the customer just like hiring for people that actually have used tracing are there any other examples where like you know this is sort of like the more i guess practical things um what do we think about um

yeah i think i know we try to be pretty intentional around the interviewing process i think um like i don't know i'm not sure i think some things we did like all of our interviews are all pair interviews this is always two interviewers um which i think that just i mean interviewees generally really like that because they get to meet more of the team um it's a lot it's more hours in interviewing overall for for for our team but it actually means that you get more information i think sometimes when there's only one person in the room you're thinking about the question you're going to ask and you're not getting all the signals so um so i think that is something that worked out really well and um i guess it's not specific to a value but it's yeah it's like a

good cop bad cop situation or is it like a shadow or uh like they actually both like would interact with the candidate we usually we'd have one in the lead and just from like a logistics point of view and so like um you know they're in charge of kind of driving the conversation and maybe the other one would take more notes or something like that but but they can ask questions too or whatever or like if they didn't understand something they would bring it up um i mean that itself can like reveal all kinds of interesting like interpersonal dynamics about like how do you manage that kind of stuff i mean you you also mentioned ben had a kid when uh i think around the time when light step started and yeah you you also had to get anything one month in or after series a,

you also had a second kid recently. So, um, yeah, we have Ben and I have like a long history of like, uh, inconveniently time children. His second kid is actually born like right in the middle of our series B

negotiations. Like I think he was actually in the hospital, like negotiating a term sheet. Oh, um, and then, yeah, like we were, uh middle of an acquisition process when my second kid was born, which was a little bit stressful. But, you know, it's hard to, you don it like to start a company, have two babies, and like be a parent, as well as be also play that role of a co founder, like you mentioned,

like there are timings you can't control on either sides. Yeah, I mean, that's the that's the lessons you're not in control. So yeah, I mean, I don't know know i think part of this was like i mean why i like working with ben and ben because like to them they're they also have things that are important in their personal life and that's a thing that you know allowed us to you know recruit a bunch of great people as well that have things outside outside of their professional life um i don't know it's tiring sometimes i'm i'm really good at coming back to sleep when someone wakes me up in the middle of the night um yeah i don't know i think i for me in a way it's like um

man i'm blanking on the guy's name now. There's a, I should really remember this because it would be a better story. But there's a musician who was like a software engineer. And he like quit his job to be a musician rather than having this like steady job as a software engineer. Like around the time when he was starting a family. Because like he wanted to like have, he wasn't proud of his job as a software engineer. And he wanted to have something to like be be proud of for his for his kids in some way and my thing is like not necessarily anywhere as extreme as that but i think um i did think a little bit about like yeah i want to do work that i'm proud of and um that doesn't have to be at a big company it could

be at a small company too. And I am proud of a lot of, I mean, I'm proud of the work I did at Google, but I'm really proud of a lot of the stuff we did at light step. And plus, I don't know, like no companies are stable.

I know that's the other thing. Like I remember recruiting for this other startup and like, we were trying to get people to come from Xerox park and they're like, can't leave Xerox. It's a stable company. Can't go to your little startup. Seems really, really whatever. And like then park shut down. Right. right so it's like there's like reorgs happen

in big companies too right so um i mean there's different kinds of stability for different kinds of people but i guess maybe that just meant that well i don't know as a co-founder i can argue for like nice health care and a good parental leave plan and things like that, which, you know, I mean, I did it for everyone else, but it turns out I can benefit from that too. So we, we, we covered a lot. And one of the things that we don't want to miss in the conversation is some of the misadventures like we usually like to do in the podcast. There are so many more questions, which I want to ask to the response you have, but I'm going to probably another time. So I'm going to jump on to the response you have but i'm gonna probably another

time uh so i'm gonna jump on to some of the misadventures you would like to discuss uh so you were at google i'm assuming you saw systems break uh we mentioned you also were sitting next to sres you you spoke with them so are there any stories from google or either at lightstep that you could share with us where things didn't go as expected? Yeah, at Google, I like benefited a little bit from building infrastructure tools. I mean, benefit depends on how you think of it. But like, I didn't have to carry a pager for a while I did like witness a lot of things. I don't feel like they're always my stories, though, because I wasn't I didn't really have kind of a stake, I guess, the way that other people did. Although I was on all these mailing lists for like production incidents in some way.

Right. And you'd get these notifications if there was like some big network event, just in case you got paged, you would know what was going on. I just remember getting this one in the middle of the night when I got paged. It was like, oh, yeah, there's like some like cable has been cut across the English channel and we've dispatched the ships. And it's like, oh, OK, I's like some cable has been cut across the English channel. And we've dispatched the ships. And it's like, oh, okay, I'm going back to bed. They've got it.

There's nothing I can do. That's the first one. Yeah, it's not mine. I mean, I wasn't involved. I wasn't on the ship. I didn't dispatch the ship. I didn't fix the cable. I don't even know how it works.

Did you ask about contact information? Can we get a hold of the people that were they didn't cc me on the post-mortem so i don't even know what happened afterwards you know like wait so but they literally i guess have to send out the ship to do the repair in the middle of the ocean i guess so i don't know i don't know this is like the weird another weird part of google now um i don't i feel like my my my stories from lights up are probably more mine to tell or something um there's one i shared with some other uh i think for some like halloween spooky thing because we was this um we were sitting in this co-working space it was there's probably like maybe seven or eight employees um we had some early customers and it just like at one point in the day like the product totally stopped

working for me and like i looked around and like it stopped working for everyone that at light step right and we're like what's going on like did anyone deploy no one no one deployed like what's going it's wait is anyone hearing from customers like what's going on and like the weirdest part was like it was just us like this the app wouldn't load but customers were like hey everything all right everything's great what's up how's it going we're like oh my god what's going on like when we roll back we didn't even can't do anything and we like tracked it down it was like there was some bit about how this is like the lesson here is like don't make your own like experience different

than your users but there was something that was like special that was set for light light step employees that were like operating system like that we got a slightly different like a not exactly debug but like kind of a debug version of it and it turns out that like some table in the database that was like pre-loaded into some part of the app state got too big and went across that threshold like some part failed where it just like wouldn't load anymore. And, um, you know, we all like finally stopped panicking and, and whatever compressed some data or, or changed some, you know, max payload size or something like that quickly and everything was fine. Um, so that was one, I don't know, I guess the other lesson there is something about like, yeah, make sure that you're, you're, you're getting the real experience or something like that.

The other one, which was a couple years ago, which did affect customers, is that we had properly renewed LightStep.com. But there was some other feature that our register provided that was some security thing that we paid the extra 20 bucks for or something like that that we didn't renew and when we didn't renew it they decided that they should just like lock the whole thing down and so they were like redirecting lightstep.com to like some registrar web page or something like that and the worst part about it is like do you know about like soa records it's like the the meta dns information kind of um like they had also like polluted that in some way and the ttl on that was 24 hours and so like it wasn't even just like we had to push a new like record set or like i don't know retype our credit card number in or something like that

like we actually like couldn't get it to like flush these these soa records anyway the lesson there is like i don't know just click auto always click auto renew also have you know we also improved our monitoring around some of our dns at the time that's interesting uh did you use light step uh like uh what i'm trying to ask is like did you dock for light step uh like do you have a dock wording instance which you used to debug the app itself yeah so we do we do have that it didn't help in um in that case because we don't we don't really help with dns at all um but yeah we run a separate instance we call meta because it's like the meta monitoring. And then that receives data from LightStep. And that's what we use to, you know, build dashboards

and get alerts and look at traces, do analysis, all those things that happens there. So I was just bugging someone actually, because I feel like we do a lot of like product demos in all hands, but we don't do a lot of like real life kinds of things. So I was just bugging someone actually, cause I feel like we do a lot of like product demos in all hands, but we don't do a lot of like real life kinds of things. So I was just bugging an SRE to see if, if they would do like a short all hands presentation for light step to kind of walk through. And, and you know,

I think a lot of the engineers kind of had seen what happened, but like for the rest of the team to see how light step is like really used and like, I don't know, try to convey a little bit of emotion in the moment when like your load balancers are like freaking out and you don't understand why that is very important uh yes uh have has there been any uh like any tips to do dog food right because i know in certain cases i saw this one of our internal teams where you sometimes cross that boundary of one system is observing the other one and you end up in cyclical dependencies sometimes.

Has that ever been the case for you in your setup or do you always avoid that? Yeah, I mean, so maybe one, I mean, there's a couple different parts of that one is just like, I think, kind of back to the customer question, like, you know, are you really your customer? I think for LightStep, we because we did sell to larger organizations, like we didn't necessarily, I mean, we dogfooded in a way, but we also used other tools. And I guess I think a lot of the value of LightStep is as you start to break a system up into pieces when you have multiple on-call rotations responsible for different pieces that are independently being deployed and lots of changes happening quickly. And when LightStep was like six engineers, we didn't need tracing to tell us. I mean, it was useful sometimes, but it wasn't like this integral piece of like being on call. And I think, you know, as we've grown, it has become more integral in terms of like actually meeting the needs of our organization. I mean, there are other parts, like you might be thinking like, okay, great.

They're using the meta instance to monitor the customer facing instance, but who's monitoring the meta instance? Exactly. Yeah. There's sort of two paths you can go on that. You can make like the meta meta, you can kind of just like keep building them up. But, um, the risk there is like those things get progressively less and less used. And so there's like more and more risk that like, they're not really serving the need they have. So, um, we've done this at differently at different times, but today actually the, the two instances actually point to each other. So we've done this differently at different times. But today,

actually, the two instances actually point to each other. So the customer facing one also is used to monitor the meta environment. And that we felt like was the right trade off in terms of making sure that it's really used and also just limiting the like, operational burden of yet another instance that needs to be deployed or things like that we just have to be a little careful not to make like big changes to both of them at the same time yeah the other very significant downside of having too many chained up is that you know thinking about all the times that engineers waste trying to find like the best inception like memes trying to describe that's really wasted time yeah, yeah, it's true.

Thankfully, we've avoided that. Yeah, so one of the things you mentioned around just not having professional success as the only measure of success and also having family and something to go back to, and this is something that I think about a lot. And, well i i don't have kids yet but uh and i don't do a very good job of balancing my own personal time at work

versus my time at home uh and i was reading this book uh by i'm not sure that i said i have good like a good balance yeah yeah um yeah i actually think like work-life balance is a term I have mixed feelings about that term itself. Because like, I think the lines are blurred. In fact, like, I don't know if I would like to work on something that wouldn't seep through my life. I want to do work that I'm excited about. But I was reading this book by Will Larson, staff engineer. And one of the things he talks about is like as you progress through your career, you have more responsibilities on your personal life front. And there are the things that you are doing

as part of career growth are more important. There are more things to do. So you have less time and more important things to work on. So you have to be more efficient and like balance priorities really well. I was curious, have you found certain practices balance priorities really well. I was curious, have you found

certain practices that work really well for you in identifying what to work on? And obviously, time is limited as a resource for all of us. So how do you think about that where the things that you have to do are more important now and then you have less time. Yeah, man, it's like the hardest question yet. Um, I don't know if I've got a good answer. I think, um, I mean, I guess maybe part of the answer is having a team that you, that, that, that'll push you in the right ways. Right. I think this is kind of going back to, to like, you know, a lot, I guess a lot of, I feel at least my own success, but I think, um, a lot of light steps success back to like a founding team that we have this

like level of trust and like, they'll tell me if I'm not working on the right thing and not going to tell me that I can't do the thing, but they're still going to tell me that they don't think it's the right thing to be working on and I don't know that's probably part of it I guess the other part we have this

one of our values is be a multiplier and one of the things I like about it is that you can interpret it in lots of different ways. So some people kind of think of it as like, yeah, I'm going to sit down next to you and we're going to kind of pair program through something. And that's how I'm, I'm kind of multiplying. But I think at a startup that's changing a lot, that's growing, that there's an opportunity for everybody, not just founders that, um, when they contribute in some way like you know maybe it only helps one or two other people today but you know in a year maybe it'll help five people and that i mean your

company's growing hopefully your customer base is growing like there's a real kind of multiplicative effort or effect of what you could be doing and so i guess maybe one way I think about it is just a little bit like, like, how is this going to affect people, you know, a quarter or a year from now. And I think that helps a little bit. Because I think it's, I mean, I think the real tension in a startup, I mean, you all know, right, is like, there's just like a lot of things on fire all the time. And it's just really tempting to put out a lot of those fires. But I don't know, some of them will go away. And some of them will be not as bad as the other fire that's on the horizon. Say no, that's the other.

Actually, you want to know, I got this advice from one of my Dan mentors at some point. But this is like a super finance geeky way of thinking about this. So do you know what net present value is or internal rate of return? Okay, so quick finance tutorial. It's like the opposite of compound interest. Compound interest is like how you calculate the value of money you have today in the future. It's like you could invest some money today. You want to compare two things you could do.

So you want to know what they'll be worth in the future. And his argument is that startup folks totally underestimate the discount rate here, which is like the interest rate equivalent. And they basically will put work off and they'll say like, I can't do that this week. Too busy, I'll do it next week. But they're just like, the like, basically, now I'm getting this backwards. Okay, here, let me start again. Like, basically, like, I was saying, like, Oh, I'll do that in the future. And he was like, if I'm not going to do it, like today or tomorrow, or maybe next week, like, I'm just gonna say no, there's no, like, I'll do that in six

months or something like that. Because like my time time and like i could be doing other stuff that will have like so much more value in six months i don't know maybe this doesn't make sense to self-worth people anyway it's like some financey thing but the short of it is maybe the takeaway is more like um that like a lot of people in startups like tend to um undervalue like the impact they could have in the longer term and And so his kind of take is like, if this is not something that you would make time for right now, just say no. And don't like plan to do it in the future because you don't,

I don't know, you don't have enough information to know what's going to be valuable. Other valuable stuff will come along. Who knows? Just do it or don't do it. I was thinking almost along the lines of like, if you like sort of cast it forward, it's like a little bit cheaper to do it now rather than like,

if you wait like two weeks, it might be even more difficult, but I't think that yeah never mind i was thinking something else his example was something like oh hey do you want to meet and talk about this thing and he was like if we can do it like this week and yes and i was saying like oh i just kicked those meetings out like six months he's like don't do that you're just like you're creating work for yourself in the future and you um, um, you're thinking that your time, your time in the future is somehow you're going to have more of it then. And like, that's just not true. You're not going to have more time in the future. So like, just do it or don't just right now.

Nice. Nice. And before, before we wrap up, uh, my, my favorite catch all question, is there any question that we should have asked, but didn't ask? I don't know uh some question like would i do it all again or something like that i know we're like at this big moment right like we're we're being acquired and there's like another chapter of light step that we're all still excited about but it is kind of a moment when i've been looking back on things a lot um but i'm excited i think it's been's been good, even if it's been stressful at times.

I've certainly enjoyed working with all the people that I've worked with, and I think we're building cool stuff. Yeah, we completely dropped the ball on that. But yeah, no, big congrats on the acquisition. Sorry, sorry. We talked about this a little bit at the beginning. Yeah, we completely forgot. A little bit underhanded, maybe.

Sorry. Very professional podcast that we run here. Definitely recommend it to everybody. Sorry. Yeah, yeah. Tell us more about the acquisition. And yeah, what's the sort of um in the in the cards yeah i mean like i said it's kind of the next chapter for us i think um even if you go back

to like our series a pitch deck like we talk about tracing as a means but tracing is not like the value right it's like it's it's the way that you answer questions, right? It's the implementation. And the value that LightStep provided then and largely still today is really for folks that are on call or for developers that are trying to understand performance. But thinking back to some of those pain points that we saw at Google, like the data that was used by the tracing systems at Google was actually used to inform all kinds of decisions around finance and quota enforcement and um chargeback capacity planning like there's all kinds of other use cases for this data and and like i said going back to our even to our series a pitch deck like those were always part of our vision for for where lightstep was heading and um so in a lot of ways like the service now is is a way for us to accelerate that vision because ServiceNow actually already serves a lot of those other users that are in adjacent parts of the organization.

So there's a lot of work to be done still to make that happen, but we've got that audience in front we get a, we get a chance to like help those folks as well. So we're excited about that, I think. And I think, yeah, I think there's just the other side of it is just that like the folks at ServiceNow have been really excited about, you know, letting us keep the parts that are working well, right. In terms of how we work and things like that. So that's also like, kind of, I guess, as a founder important that like the organization that I helped build and

then the culture that I helped build, um, is like still something that we're, we're adding to what's there, but I'm also like the, the, the, what I've learned about the culture at ServiceNow is also very exciting that they have similar kind of values around, you know, doing, taking pride in your work, but, but also, you know, being able to, to distance yourself from that work in some way. And so, um, yeah, so that's exciting. Um, and honestly, you know, as much as I love doing all these other non-engineering kinds of things at times, you know, maybe I'll take a break and, you know, someone else can, um, do all those other important jobs that maybe I didn't,

you know, we didn't assign based on wheel of misfortune or anything, but, or maybe not like my professional strong suits, let's say so. As you, as you go through the next phase of light step, what role do you see for yourself?

Would you continue with what you're doing right now or do you see that changing in any way or form yeah so i mean my first job is the kind of like just like help light step find this new home and that i think that's just kind of as a as a founder what i'm what i'm aiming for and there's a lot of different pieces of that um i guess you know i'm excited to talk with service now customers and see where their pain points are and and how how we can potentially help with that um um and i'm excited to to try to understand i mean they have some pretty um uh i mean in terms of usage, like pretty, pretty amazing software, kind of back to this measure, like it, that serves some real needs.

And so understanding technically like what's there and how can we integrate with that better or something I'm excited about doing. But yeah, so I think I have a lot to learn, but I think I'm going to probably keep doing some of the parts around, around talking and writing. And hopefully, like I said, maybe I can free up some of my time to do a little more building than I've done lately. I can still build things, but most of my commits these days are just to Terraform files.

I don't know if it really counts. In my opinion, it does. Okay, cool. So the question we didn't ask, would do it again yeah for sure i mean i i i like like i said i like building things and light step was uh and is like a really fun thing to build and um i mean i mean i guess i would do it with the right people again that's really important, but for sure, yeah. Nice, nice. Well, in the actual closing,

after the mess up there, what was the tool that you recently discovered and really liked? Oh, man, this is the hardest question I think that you put in here. I'm like, it's just like as a... We got some wild answers, man. So yeah.

I was just like, what have I used lately? You know what I really appreciate? Like command line tools just to do boring stuff. Like, you know, I just had to like find a list of all our static IP addresses. And like, there's just like a tool that Google Cloud has that does that. I don't know. It's like the most unglamorous kind of thing ever, I feel like. But I don't know.

I mean, for a technologist or whatever, I'm kind of skeptical of a lot of technology. So I don't know about new tools. It sounds scary to me. I just want old ones. Make them work better. Good answer, good answer. And is there anything else

that you would like to share with our listeners? No, this has been fun. Thanks for having me on. Sweet. Thanks so much, Soons. Really enjoyed talking with you. Yeah.

Thank you so much for making the time. Hey, thank you so much for listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremisadventures.com. You can also write to us at hello at softwaremisadventures.com.

We would love to hear from you. Until next time, take care.