Software Misadventures - Lessons from the early days building Kafka and Confluent | Jay Kreps

Episode Date: June 4, 2024

From writing the first lines of Kafka over a Christmas break as a LinkedIn engineer to running a public company as the CEO of Confluent, Jay joins the show to chat about how he and his co-founders convinced investors to take a chance on their vision, what many engineers get wrong about communication, and why engineers can make great CEOs - even when coding is not in the job description. And much more. Segments: (00:01:16) The Shaved Head Bet (00:04:07) Fundraising (00:12:16) The Role of Technical Background in VCs (00:15:48) The power of believing in the possibility of important changes (00:18:29) The Journey to starting Confluent (00:27:11) Kafka's Controversial Beginnings (00:34:30) Effective Communication in Engineering (00:44:20) The Early Days of Kafka (00:48:31) The Power of Storytelling (00:57:19) Early days of Confluent (01:03:06) Do Engineers Make Good CEOs? (01:07:59) A Typical Day in the Life of a CEO (01:12:24) The Evolution of Data Streaming Show Notes: - “The log” blog post that solidified Jay and his co-founders' conviction to found Confluent: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying - Jay on twitter: https://x.com/jaykreps Stay in touch: 👋 Make Ronak’s day by leaving us a review and let us know who we should talk to next! [e-mail] Music: Vlad Gluschenko — Forest License: Creative Commons Attribution 3.0 Unported: https://creativecommons.org/licenses/by/3.0/deed.en

You know, engineers say there's a problem when there's a problem. And then, you know, at the director level, people say, oh, we have a challenge. And senior director level, people say, oh, we have an opportunity. And then at the C level, people say we have a big problem. So it goes full circle, right? And I don't think you have to be mealy-mouthed. I think you say there's a problem. But I think being aware of what is the impact i'm trying to have and getting good at that that's a skill

where you develop over time but only if you try and then you know that's the only way you can have impact on trying to get a large group of people to do something differently is by developing that if you don't do that you, you're just kind of shouting into the wind. Welcome to the Software Misadventures podcast. We are your hosts, Ronak and Gwan. As engineers, we are interested in not just the technologies, but the people and the stories behind them. So on this show, we try to scratch our own edge by sitting down with engineers, founders, and investors to chat about their path, lessons they have learned, and of course, the misadventures along the way.

Jay, super excited to have you with us today. Welcome to the show. Really excited to be here. Thanks for doing this. So we thought we would start with asking you about a video I saw on Twitter, and this person, Eric Wishria, I hope I'm pronouncing his name right. There's a video of you shaving this person's head, you and a couple other people. And Eric is on the board of Confluent, if I know right. Can you tell us what's going on? Yeah, happy to do that. Eric was our first investor. So just as we were starting the company, we went and talked to him. We actually didn't have a name for the company yet. So I think we pitched it to him

as Nuco. And nonetheless, he invested. And so, yeah, I think I'm not sure what the milestone was. It was some revenue target or ARR target or something, I think, that we beat. And so he had offered to shave his head if we actually hit our hit. I think it was if we hit our plan for the year or something. And so, yeah, we made a video of it, which think it was if we hit our plan for the year or something. And so yeah, we made a video of it, which is it was originally meant to be just in inside the company. And yeah, you didn't look bad. Although you know, if you've got the shaved head, you've got to you have to eventually tan the head. Otherwise, you look kind of pale, you look like you maybe just came out of the hospital versus you did intentionally. So in my case, you know, it's just from balding. It's not necessarily from breaking any plans

or anything like that. Hashtag relatable. Yeah. So it's pretty cool, actually, that he went through with it and actually published a video on Twitter in this case. I'm hoping Eric's family was okay with the specific choice. I heard it was not that popular, and that was why the shaved head look didn't stick around.

What was the reverse bet, by the way? What if he won the bet? Was there something that you would I think it was one sided. You know, it was the people doing the shaving was me and the head of marketing at the time, both of whom were, you know, pretty bald. So we had nothing to lose in that respect. Nice. So you mentioned that initially when you pitched to Eric, you named this company Nuco. So curious, how did that evolve to Confluent? Yeah, well, we were really picky about the name that we wanted to give.

And we didn't want to put down something that wasn't good. And so you've got to put down something that wasn't good. And so, you know, you got to put something on the slide deck. We just came up with Nuko, although apparently there's a bit of a tradition of that, you know, companies that don't have a name yet are often called themselves Nuko. But we, I guess we independently invented that name for the interim. I mean, my tip to people would be if you're pitching a company, you know, you should probably just come up with a name. You can change it later versus having not decided. But the specific sequence of events was kind of unusual because we talked to the people at LinkedIn as we were leaving.

And they said, oh, we would be interested in investing, but you would need to go get kind of VC that would price the round. And so we went out and pitched a bunch of VCs much sooner than we'd actually expected to. We'd expected to leave LinkedIn and really kind of spend some time and kind of get everything baked. So we were effectively doing everything a couple months ahead of schedule at the time that we were fundraising because we wanted to make sure the LinkedIn folks didn't lose interest, kind of strike while the iron is hot. And so the result was a bit of a rushed initial fundraising process. I think it's pretty cool that you guys have a playful, if I may say that, relationship with this VC, I guess, in particular. How has that relationship, was it always like this from the beginning or how has that evolved over the past years? Yeah, I think

we've been lucky to have really you know, really good early investors. And I think that was really helpful. Anybody who's involved or on the board has a stake in the company, you know, they're going to be part of the early team. And, you know, that can be a, you know, that can be a great addition or it can be a bit of a pain,

really depending on who the person is. You know, it's. I think some investors add significant value, and then I think some probably subtract some value, right? So I think the trick is trying to figure it out when you're raising money, because at least for us, we had very little experience with venture capital, didn't really know the ecosystem of people. So we were coming into it a little bit blind and trying to make sense very quickly of who all these people were. Again, because of this rushed process to try and keep LinkedIn interested as we were leaving. So in this case, you went from being an engineer at LinkedIn to starting the company.

And obviously, Kafka existed at that time. It was open source and you wanted to build the company around it. Fundraising is not a skill that engineers typically have. And especially when you were doing it, like you said, a little bit rushed in the process. Did you have any guidance at the time in terms of how to go about this and have these conversations?

No, absolutely not. There was some information on the internet, not nearly as much. Now there's so much information on starting companies that you can go listen to like 10,000 hours of podcasts and blog posts and books and so on. Yeah, there wasn't a ton out there, to be honest. There was some at that time. And so yeah, I'd read some stuff online. That was probably the extent of know, so yeah, we definitely did not know what we were doing. I don't think that's necessarily a bad thing. I mean, I actually think the skill for early stage investors is kind of being able to see the diamonds a little bit in the rough because the reality is the level of polish may not be a perfect predictor of the quality of the team or the idea, you know,

in that very early stage. Later on, should be. And so, yeah, you know, I think actually the fundraising process was pretty successful. We had a number of people who were interested, you know, despite us having no idea what we were doing. And it also being just three very technical people who had not really even worked in an enterprise business, you know, let alone founded a company. Interesting. How did you guys make that happen? Who wrote all the emails to reach out?

Or did they just one person or two VCs know that you guys are doing this and then more just kind of reach out to you guys? Yeah, we sourced a bunch. So, you know, we kind of went out to our connections and tried to figure out who knows these people. Basically, we didn't know any of the VCs. So we just looked at companies we thought were good and then tried to figure out who invested in those companies and then just met with a bunch of people. Right. So they tried to get introductions to them. Neha, who was one of my co-founders, you know, I think it was her cousin was the CEO of a company

who was extremely helpful in introducing us to people. There was a few other paths that we had, but we just collected a bunch of intros by whatever path we had to the people that we thought might be good. But of course, it was pretty hard because we didn't really know who to talk to. We were just trying to figure out like,

oh, who covers this kind of space and might be interested i heard there's like a quote not a quote uh someone told me that oh for vcs right like their product is money and money is like the most commoditized product there is so how do you differentiate is a big question was it clear to you during that process, like some VCs were just so much better than others or did it all feel kind of like middle of the road? Yeah, for us, you know, I think board members can definitely be very helpful to a company.

So I don't think it's commoditized. You know, I think that can be very valuable, but it could also not be valuable, kind of as I was saying. So the question is, how do you tell? And yeah, the answer is, you know, mostly you talk to other people who've worked with the person, probably the same as any hiring decision. And you talk to the person and try and assess it. So yeah, so I think probably for us, I think a lot of the decisions were like, hey, how do you weight the brand of the firm, the kind of seniority and experience of the individual and like what they've done, plus your conversations with them, like how much they seem to understand the company and what you're trying to do, plus the references that you do.

You know, how do you kind of weight all those different factors? I don't know if we came up with the perfect system but it worked out well so um you know but yeah i think we definitely thought a lot more about it in the first round i think we actually took like a week to pick which is probably about six and a half days longer than we really needed that's that's nice right usually it's the other way around but once you've gotten enough traction right then it's like you're picking the vcs i know this is a long time ago but do you remember any question that you asked everybody and then you were kind of judging uh like their answers to that question yeah yeah i don't i don't think it was so much that i mean the

conversations are on both sides right they're going to ask you questions and you would ask them questions i actually go i put a lot of weight in the questions the person would ask. So I actually like people who could understand the business we were describing, why it could be valuable and understood some of the problems that we would probably face. The more that somebody could pick it apart and have like kind of a very critical take on what could go right or wrong, the more I thought they really got what we were doing, it would be helpful. And even though in some sense, I could come across as negative, at least for me, that kind of shows, oh, okay, they really get it. I've always felt that way. I actually feel that way.

You know, if we interview candidates to work at Confluent, sometimes people don't really have any questions at the end, which is fine. But when people do have questions and they're actually quite insightful and you're like, yeah, that is our biggest problem, I always feel like that's actually a really good sign that they kind of have gotten right to the heart of the matter. And to be able to do that, you actually have to have a pretty good understanding of the space and the competitive dynamics and the type of business and what really matters. And, you know, it actually shows a lot. It's like you actually care, right? Yeah, yeah, yeah. Well, both care, but also can put all the parts together.

So I thought that was important. The other thing for us, you know, we felt that there was a bigger opportunity around data streaming. And so the, you know, the kind of cookie cutter thing was like, hey, this is the next like message queue. Like it'll be like TIBCO. And we felt like, no, there's more here. Do real time stream processing of the data. You could like build really a whole platform around this in the way databases have been. So, you know, we wanted to try and find people who were signed up for the bigger thing rather than the smaller thing. And in theory, VCs always want the bigger thing. But then in practice, the cookie cutter thing is much easier to see.

Right. And so, you know, I think that was another factor for us as well as how much did it seem like people kind of got the bigger opportunity in vision i see did that naturally mean that you had a preference for vcs that had a more technical background or did you also see that people that didn't have that but were able to ask really good questions yeah yeah it's interesting eric who we who we went with from benchmark you know i would say he's technical but not hyper technical you know his background is not necessarily like software engineering i would describe him as a very smart generalist and so yeah i don't know that it was so much oh somebody who knows a lot about the building of software so much as somebody who was willing to dive in and understand this space you know at that time there was not going to be anybody who would like study stream processing who was also a venture capitalist.

I was like seven computer science papers and, you know, whatever. It was just not a it was not a thing that you would expect somebody to already know about. So so but yeah, that kind of willingness to really dive into an idea. I definitely think that that I think that's important. I think generally if you have somebody invested in the company, you want them to understand think that's important. I think generally, if you have somebody invested in the company, you want them to understand what it's about. And the reason for that is, that's what will drive conviction that it can be successful. If you don't really understand something, then you may be excited when it's hot, but you will not be excited when it's not,

because you're going off of other people's judgment and you know so i tried to maintain that in later fundraising just the people who are willing to dive in and kind of get into it more not necessarily the most technical people but sometimes yeah you know i think that's actually really valuable in as kind of a test of how they approach these things and i think it was useful for us. Nice, nice. We were talking to Josh Wills and one thing that he said that stood out to me

was that it was like, oh, engineers don't make good angel investors because they're very pessimistic. It's hard to see, right? Like so many years down the road that what's the billion dollar thing that you're trying to build. Did you find that to be true in your experience um yeah i actually i probably had the opposite problem

as an engineer i was always like i don't know how to describe it i'm not like a super optimistic person i i think i i heard myself described as kind of grimly determined, but which is not quite the same. But the, you know, at least for me, I always feel like it's very clear, the bigger thing that you could do, right. And especially in technology, you can easily prove to yourself that it could be done. There's nothing fundamental. I mean, maybe there's some fundamental problem that can't be done. But but once you see that it can be done, like once you prove to yourself that the core problems are solvable, then I feel like there's, you know, the only thing standing between you and that outcome is kind of the execution. You know, like, are you going to go do it? Right.

And so, yeah, I do think like engineering operations, some areas of finance, you do tend to have more, call it pessimistic realists. You know, to some extent, these are people who their work is in direct contact with reality. So they see all the ways things go wrong. But some people, I think, over-index on that versus seeing the thing that you would describe for an angel investment, which is kind of, is there any fundamental reason why this couldn't go right? And I think for a lot of big technology projects, that's the thing that actually gets a team through it is you see that end state. You're like, we can definitely make this happen, right?

We might fail, but there's nothing fundamental that will prevent it. It might be hard, but we can definitely do it. And I always felt that about these big systems engineering projects and any kind of big change or transformation in a company you're trying to do internally with technology, externally in terms of creating a startup or a new product. And I think it is a little bit of a test of personality, you know, whether people believe that you can kind of accomplish some larger change. But I think it's also a question of experience. What I've seen with people is people who've had a lot of failures tend to be kind of beaten down around it. But somebody who in their career, even if they're able to accomplish some smaller change, which led to them being doing

some larger change, maybe in an organization, which led to something bigger, you know, they tend to actually feel no, you can do these big things. There's nothing preventing you. Worst case, it doesn't work. But like, there's nothing, there's no fundamental problem. And so if there's something very valuable that should happen, then, you know, you should go try. And so, you know, I always think for people in organizations, you want to find a way to try and build that experience for them, give them something that they can go do, that's kind of within the scope of accomplishment that they may be able to achieve. And as they've done a few of those, they tend to get more ambitious in terms of what they want to go fix. And I think that's a really powerful thing to build.

And to some extent, a company is very much like that, right? So that was definitely our take on Confluent. From first principles, we were like, hey, this kind of streaming model for data, it makes a lot more sense than batch and batch processing. To get from A to B is really hard. There's a whole ecosystem around data warehousing. It's very batch oriented, right? You move the data in at the end of the day, you have a bunch of jobs that turn over, you know, you have to recreate all of that. And so when you sit down and you try and think of like, oh, how would you do it? It seems like, oh, this is really a lot of work. But then all you have to know is that it would be much better if it worked that way.

And there's no fundamental technical problem that limits it from happening. Once you know that, then you would say, well, okay, there's kind of an inevitability that somebody is going to do it. And so if we try and do this, what would be the steps? Can we come up with a first step that we could do that would be valuable, right? That would actually have some use cases that people would use. And then if we had that, could we come up with a second step that would be a little bit further along that would also be valuable? And, you know, as you start down that thought process and you're like, oh yeah, this could absolutely be done. It's a long journey, right? It's not going to be like a one-year project, you know, but it's definitely doable. So yeah,

I think that, you know, I think that ability to do that, I think is a very powerful thing, especially for senior people in an organization. You want them to have that, you know, that kind of belief in the possibility of important changes, because most of the big valuable things in companies are that kind of change. Going along that, like, do you, can you remember like the specific milestones in your career that led you, like, right, like the first few steps that finally led to you, you know, quitting LinkedIn and then starting the company? Yeah, yeah. I had, when I came to LinkedIn, I actually came to try and work more with data than on data systems.

So my background had been in machine learning and I was really interested in trying to find a place where I could do that in my job. And so, and I thought these social networks would be a really good place for that. And so I came to LinkedIn for that reason. So the first project I had at LinkedIn was this kind of news matching thing

that we were trying to build. And it was the most ridiculous, bad, you know, ridiculously bad approach. Is that the speed thing initially? It was like an unranked, yeah, it was worse than that. It was like an unranked keyword search based on your company name for news.

So like everybody working on it was like, this is the dumbest idea. Like it's going to produce terrible results. A lot of company names are just words, right? So you can't, you can't do this. But anyhow, so we did that, you know, I had some other efforts that were in the, you know, the use of data. But at that time, it was very hard to do that at LinkedIn for, you know, a couple reasons. I think one, the infrastructure was very immature, like LinkedIn was basically running on a very old centralized technology stack. So, you know, there was this social graph system that would get called like a zillion times to try and figure out the routes between people, You know, it had the whole social graph on one computer. And so, and there was replicas of that computer

that could also be queried, but the number of edges in a graph, you know, grows quadratically with the number of people. And so there was like this continual engineering effort to try and shrink that data set. So it was like, okay, first let's write it in C++. And then it was like, well, let's try and compress it. And the urgency of this was very high.

So, you know, it was like, if this project is not completed, LinkedIn will like go off the internet. And so I was on this larger team. That wasn't my project, but, you know, it was a kind of peer team. And, you know, there was just a lot of that where we wanted to do these very sophisticated things with data, but the underlying infrastructure get or, you know, hook up to systems or come up with any way you could do anything. And then you would end up with these solutions in the end that were just this like dumb keyword search or something. Because that was the only thing that was possible. So over time, a lot of my efforts then switched to kind of infrastructure. And I liked it for two reasons.

One, there was, you know, there was no product managers. And so you had more degrees of freedom in how you would solve the problem. And I was so frustrated from that keyword search thing, where I was like, this is not the right way to build a news relevant system, but nobody would listen. I was so frustrated about that. That was like, oh, this is better. In the infrastructure world, the customers are internal, you know, come up with the right approach. You can reason with engineers on that front and not the product managers. Yeah, yeah.

And so, you know, and at that time, the technology stack at LinkedIn was extremely frustrating. So, you know, so I was riding the train back and forth to work from San Francisco. And, you know, to do something on the train, I had a programming project, which was to build a database. And I was interested in some of the distributed systems. There'd been a paper that came out of Amazon on DynamoDB, and it was like a key value store. And so the first, you know, the first project was, oh, could I implement that? And if I did that, I thought, oh, maybe that can be a better replacement,

because we're trying to do everything on top of these Oracle databases that were like really, really difficult to scale. And so I thought, hey, if you did that, you could have like really fast access to data, you could do a lot of better front end features. And, you know, at the time, everybody thought I was kind of nuts, because, you know, it wasn't, it wasn't a thing that you would do to like go build the database. But this is a very simple database. I thought, oh, there's no reason you couldn't do this. So I want to pause you there for a second.

Yeah. Exactly this part. Like today if someone goes and says, hey, I want to go write my database, at any company or even friends, it's like friends don't let friends write databases, for example. Yeah, it's a terrible idea. It's a terrible idea. And the other thing is I think if you place this back in time, this was like somewhere between 2007, 2009, somewhere around that, I think. Yeah, that's right. The open source technologies weren't as mature as they are today. Yeah, there was nothing.

There was nothing. Very few options to work with. Where did you get that conviction from to say, I'm going to go write this database? Oh, well, I'd read this DynamoDB paper and then the underlying storage engines, there's these things like BerkeleyDB, you know, it was like an embeddable. Now everybody would use like RocksDB, but it was the equivalent at the time. And that was already there. And so I was like, you know, look, these systems are not that complicated. You know, a lot of the data access patterns for live systems are pretty simple.

And so if we could just scale that one thing, that would be pretty good. And, you know, so, yeah, it's definitely true that you should not write your own database. And I later learned more about why. Because if you do that, then you also have to run your own database. It turns out to be the hard part. But, you know, directionally it was right. Like LinkedIn needed more scalable data infrastructure. Like, you know, buy, build, borrow, whatever. Like there needed to be some solution.

Like the running indefinitely off Oracle was not viable. And so, yeah, it was a bit of an internal battle to release the thing. But it was used internally at LinkedIn for a long time. It was open sourced as a project. And, you know, for a while there was this, you know, a whole storm of

key value stores and it was quite popular, but I think it was eventually displaced by the ones that had some company that was, you know, investing in it continuously. And, um, um, I had kind of moved on to other, yeah, that's right. Yeah. It was called Baltimore. Uh, so we were talking, it also, it also had a, uh, a's right. Yeah, it was called Baltimore. So we're talking. It also had a weird name. That was the other trend. I think a lot of projects at the time at LinkedIn for some reason had references to Harry Potter. Like I think Baltimore. Yeah, yeah.

That was my naming scheme. Oh, I see. So Kafka, I think, diverges from that. Yeah, we ran out of good Harry Potter names. You know, the name has to, like, sound cool in a ridiculous way. And so, you know, some of the names were good, like Voldemort. And Azkaban, too, right? Yeah, there was a Nimbus, which I think is like the broom or something.

I don't know. There was a few of those. But then after a while, we kind of ran out of cool Harry Potter names. And so then we switched to writers. So there was Kafka, there was Camus, there was a few others. But, you know, these things turned over over time. So I'm sure Kafka is probably the only one that exists now. Wait a minute. On naming it, did you read these authors at the time?

I didn't know of these authors. I had learned about Franz Kafka through the Project Kafka. So curious, like, how did that happen? you read uh these authors at the time and knew of like i i didn't know of these authors i'd learned about franz kafka through the project kafka uh so curious like how did yeah yeah i had uh yeah i had i had read kafka you know i i'd i'd done like a literature minor almost i i think i did it was like maybe one class short of it so i'd run i'd read uh some some kafka i thought it was interesting. And it's also sort of synonymous with complexity. So the idea was to unite a lot of the different data flows inside LinkedIn. And it was indeed a very kind of Kafka-esque setup. We had all these different scripts and little hacks to get data out of one system and into another.

So this was meant to kind of unify those. And so it was Kafkaesque. The problem was Kafkaesque. Oh, I see. So when you're talking to Felix, who actually introduced us over the email, he mentioned one legend where he was basically talking about Ubenness which is the successor of waldemort and he was basically saying that legend has it that jay crips wrote waldemort on a cattle train so that part is yeah that's true that's true there's a second one that we heard about and maybe you can help us confirm if that's true or not that kafka you started writing kafka over a

christmas break is that true as well yeah yeah so Yeah. So I, I, um, you know, both these projects were a little controversial internally and, you know, obviously it wasn't just me. Like I wrote some of the initial code, a bunch of people did the later Baltimore work. Um, the, you know, for Kafka, it was tricky because of course the, each one of these data pipelines was owned by a different team. And so, plan was to replace all their projects with one thing. Not in an aggressive way, but just because for a company like that that was growing very quickly, they were all breaking all the time. And then I owned some of the downstream infrastructure at the time. So I'd gotten the Hadoop setup going and the things you can build in that kind of

data lake area, they only work if the data is reliable and both reliable in terms of shows up and doesn't get lost, but also reliable in terms of what are the fields, the structure of the data, the schemas, if that's dependable know, that stuff, it was just utter chaos. And so everything downstream would just break every night in some different way. And, you know, my theory, so and the traditional theory on how you do that is you just hire more and more kind of ETL engineers to scrape data out of different systems. Raise exceptions. So at one point in the budgeting process,

you know, there was a column. So if you wanted to fund a project, it would be like, okay, how many engineers do you need? You know, how many PMs do you need? How many designers do you need? Okay, fine. Then they added like these

data warehouse ETL people on the end. I was like, you know, Jesus is now like a tax on everything we do. We're going to have some number of full-time humans just to scrape data. Like, this does not make sense. And so there's got to be some more, like, infrastructure solution that takes some of the pain out of it. Because the data lake was like one destination. But there was, you know, a search system and a whole set of analytics applications and other real-time systems.

And just each one was its own little team that was kind of building integrations. We had kind of one pipeline per data type or up to five pipelines for a single data type, depending. So that was the idea was to unify it. But it was controversial, I think, in part because nobody wanted to work on data pipelines. It just seemed like, you know, not a very cool area to work on. So, you know, I think for some of the engineering management and so on, it was kind of like not it, right, was the attitude. And then, you know, I also cut across all these other technologies that had full-time teams. So it wasn't political. It was just like, you know, it also cut across all these other technologies that had full-time teams. So it wasn't political. It was just like, you know, I just thought practically we'd be better off if we could unify some of it.

So, yeah, anyhow, it was somewhat controversial. And people also thought that you couldn't build something that would work for all the different use cases. So it would, you know, kind of have strong guarantees and be scalable, but also be real time. There was some reason that we had a batch pipeline and we had log aggregation and we had change data capture and we had three other things. So the coding on the vacation was the way through that. I was like, well, if I show you we can do it, then you can't say we can't do it. Right? So let me prove it worked. Yeah, so we wrote a very early version of it

that was kind of a bunch of individual single node instances, but very efficient. And that actually ended up getting released. It was part of the news feed early on, but it was, you know, it was quite buggy. You know, I think we, and then we, you know, got better after that as other people started working on it. Like you're saying this, it's like a huge undertaking and you mentioned manager. Oh yeah, was DJ Patil your manager at this point?

No, he was on the product side. So yeah, it was a different group of, there was a whole set of different people involved in different ways. I just think nobody was very excited about data pipelines. Understandably so. How did you get buy-ins from the manager, from management? Ah, the prototype.

Prototype solves all problems, right? If you have a prototype, everybody's like, well, prototype solves all problems, right? Like if you have a prototype, everybody's like, well, it could almost exist, so we should finish it. So, and then I think, you know, it got rolled out for some of the, you know, user activity, kind of real-time logging stuff. And then from there, I think it spread over time

to some other use cases. But yeah, it was, you know, it was a challenge. It was, Anything that cuts across the natural organizational structure in a company is always hard. From a management point of view, that's what you're always trying to do is actually design the org structure to make the problem you want to be solved easy. But whenever you do that, you make some other problem hard. And so it's a little bit like the sharding or partitioning of data in a distributed system. Some queries will be fast because they're all within one partition. Some queries are going to be slow because they cut across.

So it's a little bit the same with the org design stuff. So when you were working on this prototype on the cow train. Yeah, this was a Christmas vacation, sorry. Yeah, this was a Christmas vacation, plus I think at one point I flew out to see my grandma. No, I guess it was a wedding that was in Pittsburgh, and so I was on the plane and I was staying with my grandmother.

My cousin was getting married or something, so did some work on it there. My cousin was getting married or something. So I did some more time there. Eventually got to something that, you know, mostly worked. So I imagine like a lot of people would try to start something, right? When they are feeling inspired by like a paper or something else. But then they inevitably gave up, right? Like after they hit the second or the third roadblock. Like for you, there must have been some kind of reward function right where as you were going through it you're like oh man I'm

like learning so much or what was that reward function for you yeah you know I think it was a combination of things like I I like computer programming so you know it's just fun to work on like there was interesting problems to try and design the system and then you know once you're kind of working on it you're in some rhythm you, it's interesting to work on. So I enjoyed doing it as kind of a spare time project. And that was my way of kind of learning and keeping the skills sharp at various, like at the time I was doing Kafka, I think I had more of a, I can't remember if I was a engineering manager or architect at that point, but, you know, basically I wasn't writing as much code. So this is a way of like not becoming totally a pointy haired um you know paper pusher yeah yeah

so um you know so i enjoyed it from that point of view and then you know i think i'm just extremely stubborn so then once you know once i was convinced that this solved a big problem, you know, I was just going to keep pushing on it till, you know, till I hard failed, which, you know, hadn't happened yet. So I just kept pushing. You know, I think in retrospect, at that point in my career, I think I was a little, you know, I think I was a little controversial internally because I was always pushing on these things in different ways. You know, I think internally people thought I was a little bit of a little bit of a cowboy. But, you know, it was well intentioned. So hopefully there's hopefully there's no hard feelings. Looking back, like, would you change the way you approach some of this? You know, I got better at communication over time. You know, I think probably like a lot of engineers early on, you know, I would just try and like say things that were true.

Most people the wrong way. I agree with you. No, I think that's good. Like you want to say things that are true. That's not wrong. But that was pretty much the extent. So I would just be like you know wrong and then i realized you know i think managing people is a helpful exercise

because you're trying to get people to do stuff and you realize like okay you know what do i need to do i need to really understand the world from their point of view. And then when I'm speaking, it's not sufficient to just say things that are true. That's necessary, but it's not enough. I need to also say things that are going to change what they think. And so if I want to change what they think, I have to really understand how they think. And then explain it in a way that makes sense in their world. It is likely to produce that outcome. And you see this a lot where you actually,

it's common among engineers, their communication is actively provoking the opposite of what they want to happen. And so the solution to that is not to become totally mealy-mouthed where you just don't say anything, but the solution is to think a little bit from other people's point of view, what's going to change their mind and what's going to change how they think. I think that's a very important skill.

Communication, I think, is one of the most powerful things, broadly. If you want to get people to adopt some new technology, if you want to change how an engineering team works, if you want to start a company, you know, ultimately this is some idea that you want to propagate. So you got to think a lot about that. And so I think that was very,

it was a valuable exercise for me. You know, it was definitely a maturity on my part, early in my career. And it was a good evolution. Now, I don't think what that means is just to become, like I said, mealy mouth. My joke internally is that engineers say there's a problem when there's a problem. And then at the director level, people say, oh, we have challenge. And senior director level, people say, oh, we have an opportunity. And then at the C level, people say we have a big problem. So it goes full circle, right?

And, you know, I don't think you have to be mealy mouth. I think you say there's a problem, but the, but I think being aware of what, you know, what is the impact I'm trying to have and getting good at that. That's a skill where you that you develop over time and but only if you try. And then, you know, that's the only way you can have impact on trying to get a large group of people to do something differently is by developing that. If you don't do that, you know, you're just kind of shouting into the wind. Yeah, I totally agree with that and this is something i noticed a lot of senior engineers too where uh takes a little more intentionality takes a little more patience than needed or than people think is necessary and you see when some people when they go into well so-called conflict where two teams are basically screaming into the wind uh someone

comes in and can provide that clarity and still achieve the desired result where The stuff that is wrong gets fixed, but there's a way to put it in words and it's not a common skill set though I agree. It's one of the most important ones for engineers Yeah, yeah, yeah, it's you know, I think that kind of really building empathy is an important skill I think for You know management leadership any of those things? Which is you know ultimately just comes down to trying to figure out where where people are coming from and uh yeah one aspect of this is like uh when you ask people who are really good communicators um one thing that people often say is like it's practice You do more of it and then you get better at it. And initially, no one is perfect. So have there been resources apart from trying it a whole lot that have helped you in improving this? Yeah, I think that's accurate. I mean, I actually think the problem

with communication is people don't like to work on it. And so, you know, different people think of different things as work and people don't like to do things that they don't think are important work. And so you find this a lot in technology. So if you want, you know, if you want a product or an idea to be adopted, you have to really think about how to communicate that idea. And, you know, it turns out for these kind of enterprise products that has a word, it's called product marketing. And, and yet a lot of engineers who actually care a great deal about building things that people use, they actually are quite allergic to product marketing. They think it's, you know, awful, right? There should be no, you know, and yet it's actually quite hard to convey things. And, know if you think i always think about it like look you know people are very busy you know so

they don't have a lot of time for new thoughts and ideas so if you want to get something through you have to really boil it down you can think of it almost like a kind of compression so it's very you know compression is very cpu intensive if you want to compress an idea you have to really think about how to convey it what is it that people understand that's like that thing? How can I tell you about it in a way that's brief and compelling, but also accurate, like actually provokes the right understanding on your side? And if I can do that, that's actually quite valuable. And many of the biggest and most successful companies have been built on an idea that's ultimately relatively short. It's like actually easy to convey, like a picture of the world like that.

And so if you can find it, you know, it may not be easy. I think it's a really valuable thing. I think it's a valuable thing if you work inside a company and you're trying to accomplish something. If you can boil down why that matters, that idea can spread a lot more effectively. And yet, for most people, they would not be willing to spend two hours thinking about that problem, even though they would agree that it's very important. So I think some of it just comes down to what do we think is work? Like, what are we willing to spend time on? That's super fascinating. Like, okay, make it practical. Like, for micro engineers micro engineers like what do you think

are like the most underrated or like the work that adds a lot of value but people like don't perceive it as like real work for for engineers i typically think that communication side is is lacking you know i i typically think senior engineers can have a lot of impact just by, uh, explaining what they've done well and, or what they want to do or what they want other people to do in a way that's, uh, actually very compelling. Um, you know, the same with speaking, et cetera. I, a lot of people develop this, but you know, the willingness to do that, I think is a big magnifier of impact. Um, you know, it was interesting when I was at LinkedIn, I think, is a big magnifier of impact. It was interesting.

When I was at LinkedIn, I was a mentor to a number of people. So you get paired up with somebody. And one of the things I realized is, oh, everybody, I actually really want mentoring. I actually just want to get promoted. Oh, yeah. How do I get promoted? That's the worst words ever spoken. Yeah, I have one question that I want out of my mentor. How do I get promoted? How do I get promoted? Yeah, I have one question about, you know, did I want out of my mentor? How do I get promoted?

Yeah, I was like, I actually don't think it's that hard, right? If you just consistently do good work and then you explain to people what you've done in a way that they can understand and why it's valuable, you know, and you try and increase the scope of that value creation over time, eventually they're going to promote you. You know, maybe it's six months later than it should have been, but like, it doesn't really matter as long as that line is going up. And it's interesting, like people don't think about that underlying value nearly as much as they think about the, you know, the path to promotion. And so I was like, well, you know, I thought the formula was kind of simple. It was like, do something valuable, you know, communicate effectively around it. That's kind of all you need to do.

You know, let's spend the time on the valuable thing. And yet most people want to spend a lot of time on the mechanics and promotion, which, you know, it seems like is kind of an inevitability. If you have somebody in an organization who's just consistently producing a ton of results, whether the process is perfect or not, typically they end up getting promoted over time just because what they're doing is bigger and bigger. What you're saying is like,

it's kind of a byproduct of what you do, not the goal in itself. Yeah, it's like a lagging, it's a lagging measurement of value, not a leading, and I think many people see it the other way in engineering. They're like, oh, I should get promoted so I can do bigger things. That's right.

But your earlier point is kind of like the mechanism is the more interesting work, right, versus the actual value creation. Yeah, well, it makes sense. I think what people actually want is just the output data. So many blog posts around how to get to this level, so and so. Do X, Y, and Z. Anyway, so talking about Kafka itself, sir, you had this initial version. By the way, what did this prototype look like at the time?

Did you have all the... Yeah, the early prototype was more like a single server Kafka that you had to have a load balancer in front of. And then by the first multi-use case version inside LinkedIn, we filled it out. And at that time, June and Neha joined the effort. So there was a full team of three working on it. And at that point, it was kind of a quasi-distributed system, at least, that would kind of scale out and handle a bunch of use cases. So depending on when you count that first version,

it was actually, by that point, it was actually quite similar in terms of the overall style, what it enabled. Not nearly as sophisticated or good as it is now, but it was, you know, the basic idea was there, that kind of core abstraction was the same. I see. And in this case, did you always intend it for it to be open source or did that just happen by chance? Yeah, yeah. It was built to be open source from the beginning. The, you know, that was kind of our strategy for those early infrastructure projects. It's really hard to get people to work on internal infrastructure because it's some

nameless layer inside the company that people only know about when it breaks. And so this turned the things into more products. And so my theory was, oh, we can hire better people and they'll do a better job. Because if you think about something like a product, then you're kind of like, well, is my product good, right? Like, do people like it? You know, does it have a good interface? Is there documentation? So like internal infrastructure, nobody will write any documentation, but if they open source it, it's like, oh yeah, they'll document it. So that was the theory. And you know, it was mostly successful.

Not all the things that we did got a ton of adoption. You know, Kafka was early on extremely unsuccessful as an open source project until we kind of went and did some talks around it and tried to get some adoption in the rest of tech. But yeah, it was intended to be open source early on because we thought, oh, this is, you know, this is, you know, I guess our feeling at the time was, hey, everything is kind of moving to a world of these, you know, elastic distributed systems. So all the things that we had that were single server, you know, will kind of move to distributed, you know, but a lot of the work at that time was these databases, right? So it was different types of key value stores, data lake, whatever, some kind of storage system of some sort, storage inquiry. And, you know, at least the feeling for me was, you know, hey, a more original and interesting problem

is the kind of movement or flow of data. So not like data storage, but, you know, kind of data, data in motion, not data at rest. And, you know, so the idea was really, you know, trying to explore that and do that right. And having built one of, you know, whatever, 50 key value stores that all became open source at the same time, I thought, oh, you know, it's better to work in some area where nobody else is doing it because, you know, it'll be more likely to, you know, kind of build something that people use. And I think that was true. Unfortunately, it was a little too true.

You know, being an area that nobody was interested in at the time, we initially didn't get any interest or adoption. But then, you know, I think as people realized, you know, how this could help with their kind of internal data flow problems, it became much more popular. Can you speak a little bit more about that in terms of how you got more adoption, both internally versus externally, with the talks and things like you mentioned? Yeah. So the challenge for us was our ambition was something that was a lot more than just a simple message queue, but it was really hard to figure out how to explain it to people. Right. So it was kind of back to that product marketing thing. You know, I think we

didn't really, we were like, well, what is it at the end of the day? Is it like a log? Is it something else? So the, the initial effort was we kind of went around and did some talks in Silicon Valley. I think I did a talk at like Airbnb and a few other places. And those went well. People were excited about it. As we were thinking, we were kind of contemplating, oh, maybe there could be some kind of company around this. And I wrote a blog post called The Log. It was this really long thing. And the idea was, well, if we could get people excited about the bigger picture vision, maybe there is some viability to this project.

And if not, maybe not. So I spent a lot of time on it you know i probably spent like three full days just writing this long blog post that was like every thought in this area collected in one place and and it was really popular so then we were like oh okay like people are excited about this we just have to explain it to them even if we don't have a pithy one sentence explanation yet, at least we have a 20 page explanation, you know, that people can get interested in. And, you know, kind of following that, I think we decided, oh, you know, if we really want to take the project all the way, you know, it probably makes sense to go start a company. Like at a certain point,

LinkedIn only needs an internal infrastructure layer to be so good. You know, like maybe there's six people working on this, you know, maybe it goes to 10 people, but there's not, there's not a world where LinkedIn needs to build this the way all the other companies need it to be. Cause it's just not there. It's not their problem. And so if we kind of want to finish the job, you know, we're, we're probably not going to be able to do that, you know, working in the, you know, in the side corner of a, you know, professional network software thing. We got to go, you know, turn it into something that directly goes out, figures out the problems that people have using this and adopting it and can start to build out some of the layers around it that kind of complete the product. And so that was the origin,

and that was when we kind of decided to go start Compliment. So how did you convince others at this point? Or did you and Neha, for example, all of you co-found the company, did you all convince at the same time, or did it require some... I think it was... Yeah, I think we were all kind of thinking you know i definitely been thinking about it i think neha actually originally proposed the idea across the you know the the three of us um but but i think it was kind of in the air you know we were hearing from a lot of companies that they were interested in this you know i think at one point um i forget what it was but it was you but it was some media or TV company was trying to adopt the open source.

And they were like, okay, look, we just need to pay you to do these things. And we were like, well, you can't. That's not really available. You can do these things in open source, but we're not available to help you. But we were like, well like well okay this is interesting like at least you know it does look like this is applicable outside of tech companies and so that was a big question for us um you know was how broadly applicable is this this is something where it's like yeah okay all the parts of the tech company that has to happen in real time

but does that make sense elsewhere like would a bank work that way um and it turns out yeah like these big you know other companies outside of tech are actually bigger more sprawling more business units more systems that have been built over more time you know more legacy yeah yeah so how it plugs together, in a sense, is even more valuable. And so, so yeah, in fact, the need there was probably, you know, even better. And so, you know, as we saw that we were like, oh, this is, you know, there's this fundamental transition from thinking of data as, you know, kind of almost static tables, right, where the fundamental thing in a database is, how do I lock the data in place, run a query, and then kind of unlock it? Or, you know, some mechanism to pretend that things aren't changing. And then, you know, that's most exaggerated in a data warehouse where you're literally just like copying in a snapshot or something. And, you know, that's a very different model from the kind of flow, you know, flow of data across the company. And so, okay, there's a, you know, there's a very fundamental change that's going to happen because ultimately, you know, companies are out there in reality doing business in some way, you know, and that's a very continuous activity.

You know, so like a LinkedIn, it's available all around the world. There's things happening all the time. It doesn't make sense that at 12 PM Pacific, we copy a bunch of data into some data lake and run our most sophisticated processing. There's nothing magical about 12. That was just a convenient time to schedule it. The natural way of doing things would be some kind of more continuous stream processing. And so it seems like, okay, inevitably that change has to happen because the problem in the world doesn't match the underlying infrastructure layer. And there's some

valuable initial step that's applicable to a bunch of companies today, right, that we have. So if you put those two things together, you kind of have a valuable first step with a really valuable next 10 steps. You know, that could be a business in some form. I don't think we knew exactly what form, but, you know, at least it seemed like a good direction to start walking. And I remember reading that block was the first time i learned about kafka and uh it was extremely valuable and that was like i was just starting out in engineering and uh just reading and a bunch of things just went way over my head at the time it's like why stuff is this really a problem and like as time went on i learned the

value of it uh but when you're starting this company and let's say you're pitching to investors, as you mentioned, what did the evolution of that pitch look like at that point? Or what was that pithy one line or two line statement that you could share? Yeah, yeah, that was definitely the evolution of my understanding of how venture capital works.

So the initial pitch was, you know, these are the trends in data and infrastructure. It was like, okay, like, look, there's more and more complicated, specialized systems and applications, right? So there's a ton of sprawl. And then the volume of data and what we consider data is growing. And then the needs from customers, like they want better experiences or the digital part of the business is growing. So effectively needs from customers like they want better experiences or the digital part of the business is growing so effectively there's like more data you have to use it in more sophisticated ways across the company so all the parts of the company need to going to need to kind of connect together and there's going to be a need for infrastructure for that and then sort of at

the end we were like oh yeah you know and we're a team that has done this at very large scale. And there's open source, which is like, you know, very broadly adopted that people are building like big, important production systems about. And so then what I learned was people mostly like tuned out the whole first part of theoretically why this would definitely happen and kind of ignored, you know, the first, you know, whatever it was, 25 minutes of a 30 minute pitch. And then at the very end, they were like, oh, people are using this and it's recognizable companies doing really important things. So then I was like, oh, huh. Take the side about the traction. Put that first.

Then explain the theory. Then they're very interested in theory. But the first thing is actually just like, yeah, there's a thing happening in the world and I can prove it. And that requires heavy involvement and investment from companies. And that means there's a good chance they'll buy a product in that space. And indeed, it's actually a smart way of thinking. It's very easy to come up with clever ideas. It's very hard to produce actual results.

And you see that, you know, I think it's true inside of a company as well. You know, the kind of actual results speak a lot louder than theory. And once you have the results, then you're very interested in the theory. And once you have the results, then you're very interested in the theory. But, you know, as long as it's purely theoretical, you know, we can all remain skeptical or have different ideas. Once you've kind of shown it, then it's a little different. And, you know, one of the results that's the hardest to get is always, you know, something that people actually want. That, you know, it's very hard to take time out of your day to go adopt some new software platform and build around it. That's a big undertaking.

And so if you're choosing to do that, there must be something that's very valuable to you that you would take that risk, you know, invest that time. And so that's actually a very good signal. And Kafka in this case as an idea was already validated in a way with like open source adoption. And like, again, you, when you were building Kafka originally, you were trying to solve this problem of like, instead of data at rest, putting data into motion. Business is not something that many engineers think about very, at least organically, it doesn't come to many of us naturally. So what did that look like in terms of developing a business around it? Like, how did you go about figuring out this is how we can charge people for it,

this is how we can pitch, this is how much we should raise? Yeah, yeah, it was an interesting thing, as we were starting, you know, at that time, there were some open source companies that weren't very successful or very well thought of. You know, Cloudera was a company, but it, you know, it's kind of grown to size, but it, you know, had a number of challenges and kind of skeptics, and, you know, whichoudera was a company, but it, you know, it's kind of grown to size, but it, you know, had a number of challenges and kind of skeptics.

And, you know, which actually increased over time. But, you know, at the time we were pitching it, that was the state of things. So that was kind of the only model you could point to. There are some, you know, much older things like Red Hat, but it didn't, it wasn't really a replicable model. And so, you know, yeah, that was the question is, hey, you hey, is there a product here that can be successful? And we felt like, yeah, there is. The dilemma for us early on was whether we wanted to start with a software product or a cloud offering. And indeed, it was a very big dilemma because at that time, if you looked at where money was spent it's probably 90 on premise and maybe 10 in the cloud right and if you looked at the size of like the data

streaming market it was basically like zero dollars approximately rounds to zero so we were like well you know on one hand like you know a cloud offering is a really great way to deliver this. But on the other hand, it's not really clear that you can take a really early market that's going to grow and just build in 10%. First of all, somebody else will do the other 90%. Second of all, it's just not enough to get going and start a company. It was a difficult time in that respect because I felt, at least for us, we had to do both early on. And that's very challenging to try and have, you know, two models or two products. It ended up being very valuable because a lot of the companies now are now in the same situation where they're actually spread across their own data

centers and cloud instances and so on. So that ability to kind of span all of that ended up being a really big deal for customers that have that problem. But for the initial part of the company, it was quite challenging to be able to build both products in a way that was good. But yeah, the initial challenge for Confluent was, yeah, go figure out how to sell it. Because we were actually kind of starting a little bit ahead. We had at least an open source offering that was pretty successful. Building a very light product around that was not super hard. So yeah, the first challenge for me was like, okay, go figure out how to build an enterprise go to market, which I knew nothing about. And I just had the unluckiness for that task to fall on me because, you know, the other two co-founders

had proper technical jobs and, you know, I was the CEO. So I was like, well, okay, you can do it. And, you know, and indeed I was quite clueless. Like even as we were pitching the company, you know, I think some investor asked me like, oh, what, you know, what's your strategy with services? And, you know, what they meant was consulting, but I actually had no idea. I was like, I thought they meant web services or something. And I was like, yeah, we'll definitely build some web services, but I don't think that's the first thing

we're going to start with. And the guy looked at me like I was crazy, but I wasn't crazy. I was just completely ignorant. I'd never worked in an enterprise company. I had no idea that there was a whole consulting services strategy that would go along with adoption. And, you know, so that was just the state of things. But, you know, the good news was we had people who were using the open

source that we could go talk to. And so, you know, the nice thing about enterprise products is you can actually just ask the customer, like, what would you buy? And, you know, you got to be thoughtful about, well, you know, everybody tells you nice things, but would they really do it? And so, yeah, just, you know, at that point it was just kind of a blunder through it. And so we, you know, we hired some early sales reps, you know, I hired somebody who had, was just kind of a generalist and had been in different roles in enterprise companies and could help build that out and run that part of the organization. And so, yeah, but the, yeah, the initial, you know, customers, I just kind of went and found. And then as we had sales reps that helped to actually get deals closed, because it turned out I was much better at the sales pitch than the, you know, getting the final money out of them at the end, which turns out to be a skill in its own

right. So the, yeah, so, you know, I was actually very close to the kind of early sales and marketing effort for that reason. And that's the best way to learn something. You know, I think in a large company, the go to market ends up quite complex over time in terms of different segments and strategies and strategies and teams and how it all fits together but in an early startup i mean it's just very simple it's you know it's like if you want to learn how um you know transportation works you want to look at a bicycle not a you know combustion engine automobile right because it's just one is so much simpler than the other you're just like oh yeah you turn the pedal and the wheel goes right and it's kind of like that it's like okay like what what you know what do we need to do you know we need to find a way to tell people about the

thing we have and why it's good and then we need to have a way to allow them to give us money to get the good thing and then help them use the good thing. And so it's not, you know, it's not a rocket science problem. You just have to, you know, figure out how you're going to accomplish that with a relatively small group of people in an early company and then how it will kind of grow and evolve over time. So, yeah, you know, I've been relatively close to that side of the business since. But, you know, of course, we brought in really amazing people who did all the hard stuff once we were past those very early stages. As you were picking up these like new skills in a very pressurized and like rabid manner, was there any sort of lessons that you've learned from engineering that you were able to apply to these new things that you were learning?

Yeah, absolutely. You know, it's interesting. I think engineering is probably the least useful skill in a lot of ways for becoming a CEO. But there's some parts of it that are really good. So fact, it's not that helpful if the CEO is trying to write a lot of code because they're usually not around to do all the work through to production and support it at best. They kind of fling something from some night coding session and hope somebody fixes it and makes it work, which nobody really wants to happen. So the core skill set is actually not a useful thing. You know, what do CEOs do?

They do a lot of communication. They do, you know, you might incidentally do some of that as an engineer, but of course you do some of that in every role in the company. So, but there are some things which are good. You know, one is engineers have to learn a lot of stuff. So you're kind of just on this treadmill of learning new stuff. And usually engineers have high confidence in their ability

to learn something. And so if there's something new, they're just, you know, they'll just read about it. And, you know, there's a belief that you can do that. I think that's really important for a CEO, because actually, you know, it's one of a couple roles that sits across, the CFO is this as well, where you sit across every function, you have to kind of understand how they work. And you probably won't understand it as well as somebody who's run that function for their whole career. But you need to understand it well enough to know, you know, if that's going well, you know, you've hired the right person to do it, you know, to have an opinion on what's needed there to kind of help guide it and integrate it into the rest of the company.

So you need to understand it pretty well. And so if you're not, you know, if you're not going to spend a lot of your time kind of learning and thinking about things outside of what you already know, it's not going to go well. And then I think as a company gets bigger, you know, I do think that ultimately an organization is a kind of system. And so that kind of system thinking and design, you know, kind of modularity, how, you know, how does this module, is this module overly coupled to that module? You know, do we have a good API? You know, what's our observability story for knowing if this thing is working? You know, what, like, what's my feedback loop to like run this part of the business and know it's on track versus no, it's not on track.

I think that style of thinking is, you know, extremely important in software engineering. And actually very much what you do for a larger company is definitely, you know, those same kind of things, which is okay. You know, you peer inside a module and try and get it in good shape and working. And then ideally you want to have, you know, what are the outputs of that, that I can reason about. I know kind of what the input is, like what's the investment and what's coming back. If that's good and it seems like there's somebody we trust running it, then I can just kind of let it run. That's the observability story. And if not, then maybe I need to get in there and do some profiling

and see what's going on. So, yeah, I think that's good. And then I think this is true of engineers generally, and maybe even particularly true of these distributed systems people. It's actually a relatively gritty profession where you spend most of your time on things that are not working. And I think that's true of management roles as well. You spend most of your time fixing problems and trying to get stuff to work.

And then once it works it's like okay right on to the next thing and so uh you know there's there's some analogy there as well and so you know if you look at people who run tech companies there's actually a good percentage of engineers even though the the actual core skill set is not it's not that useful and and i think maybe it's for those reasons. That's super fascinating. On the learning part, right, I remember the memes about front-end engineers having to learn a new framework every two years. But now I'm like, oh, maybe that's not a bad thing. I mean, you just keep your skill set of how to learn new things in check. So that's really powerful. Thanks. So it's not every day we speak with the CEO of a public company.

And you took Confluent Public, started with like Kafka as a project to a startup to now a public company at this point. So great achievements all around. What does a typical day in your life look like? So I'll pose two questions and you can take it in either direction. So like that part. The other part is, as you were describing all of this, where having analogies for distributed systems to basically having well-functioning orgs, have you developed certain practices to stay productive and kind of get ahead of the problem a little bit and still keep some of your time free to think? Because I'm sure there are a million things wanting your attention. Yeah. You know, the job for a CEO is, there's different ways to do it, right? So,

you know, I've seen kind of two models. One is, you know, people effectively, you know, kind of manage at a relatively high level and just kind of hold accountability across different organizations and then put, you know, a lot of effort into being the public face of the company externally and the face of the company internally, the integration with different stakeholders, which tends to be the employees, the customers, and the investors. So one model is you kind of spread yourself across and that's the full-time part. The other model, I think think is a little bit more t-shaped which is okay you try and take that horizontal bit and actually get

that down to about 40% of the job so that you know in steady state your day would be about half free and then the other 50% is just very topical so at any given point there's some big problems or some big opportunity, right? So it can be something forward looking, what's the next thing we're going to do? Or what's the horrific tragedy that is unfolding that we need to solve? And then you dedicate about half the time to that. Usually it's like one or two things that can slot into that and you have to just be very disciplined. And so I definitely follow the second model, which is I ideally want to push down the steady state commitments to a reasonable amount. And then the remainder of time is relatively kind of flex. And then I'll have a

focus area at any given time that I'm kind of more deeply involved in. And that could be an area where I'm trying to hire a leader for the team. It could be an area that I don't think is on track, or it could be an area where there's some big opportunity or unlock. They're like, oh, if we can get this right, that would be great. And then I'll try and dive in actually at a relatively low level and just really understand what's going on there as best I can. And, you know, try and help get it into good shape. These like bigger, these blocks, do they just kind of come up or do you have like a system to kind of track them? So that once you do have free time, you're like, sorry, flexible time, you start digging into one.

Yeah, yeah. The, you know, I always feel like time management is just a process. You know, I feel like you're out in the jungle with like a machete and the overgrowth that just grows and it just consumes everything, the vines and the weeds and whatever.

And then you kind of hack it back, but the jungle just keeps growing. And so, you know, so I think it's a little bit like that where effectively the day-to-day activity is just kind of eat up more and more jungle just keeps growing and so um you know so i think it's a little bit like that where effectively the day-to-day activity is just kind of eat up more and more because like things creep and there's always another one-on-one and another check-in and another something that gets proposed and then you just kind of hack hack it back and you're like okay you know uh got that done and

then it grows back again so i i don't know if that's a methodology but yeah it's you know, got that done and then it grows back again. So I don't know if that's a methodology, but yeah, it's, you know, it's just calendar management. Like where, you know, look at the calendar, where, where am I actually spending time? Does that actually match my priorities? And is all my time accounted for? If all my time is accounted for, then there's effectively no flex time for whatever the most important thing is, which seems wrong. It seems like there should be some time for the most important thing. So then the machete comes. Speaking of time, we are reaching towards the end of our conversation while we have a bunch of questions to ask, but we'll save them for another time. And thank you so much for taking the time from your busy schedule. But before we let you go, one last question.

In terms of putting data in motion, I think you have been successful at that with the open source project, with Confluent as a company, where Kafka has become this de facto choice. People don't ask, what should I use if I want to move data around? It's like, oh, use Kafka. Like, that's just the option. What is something that you're looking forward to next? Yeah, I think it's still early in the option. What is something that you're looking forward to next? Yeah, I think it's still early in the journey. You know, in what we wanted to achieve, we always felt like Kafka was kind of like step one, right? So you have to have some stream that people agree on that everybody will, you know, write the data out to and read the data from. But that's not the end of the stack,

right? So if you think about what are the needs for that, you wanna be able to do really rich, easy real-time processing of data, you know, that kind of stream processing and the reprocessing. And so the evolution of the ecosystem around that is at least as important in the product as that kind of core stream itself.

To some extent, you want that core bit to just be solved. Like it's like a utility. It's like the, you know, do the wires in the product as that kind of core stream itself, to some extent, you want that core bit to just be solved. Like it's like a utility. It's like the, you know, do the wires in the house conduct the electricity? Like if they don't, that's the big problem. But if they do, it's not something you spend all day thinking about. Right. And so a lot of what the electricity goes to, that's kind of the exciting thing. So, yeah. You know, so our goal is continue to build out that, you know, platform around data

streaming and fill out those other pieces. You know, so we've been putting a lot of work into our offering of Flink. You know, this is a stream processing system. You know, typically for stream processing, it's been pretty ungainly to run these systems yourself. You know, it's relatively hard. The opportunity in the cloud is really take that away, like make something that's just very elastic where you kind of write the code, you write the query, it goes and runs. I think there's really an opportunity to take stream processing and do it in a way that's kind of a generalization of the batch world. So I said, okay, a batch system, you kind of freeze the data in place and query at some point in time.

The opportunity in a stream processing system is you kind of process at a point in time, but as new data comes, the answer just keeps evolving. And that's a very powerful generalization that matches a lot of the activity and applications in a company and makes it vastly easier to build around streaming data. So that kind of growth of the platform that makes it really easy to connect it to all the systems, really easy to process the data, really easy to kind of govern it in the large. And this is something a lot of the companies that built the most around

this have, uh, kind of built in house, but, but our, um, you know, our goal and a lot of the work we're doing now is really, uh, you know, bring that out as part of the product. So you can just have something that now is really, you know, bring that out as part of the product. So you can just have something that's kind of, you know, very cost effective, very easy to scale, just kind of solves this problem for you. So every company can have like awesome infrastructure that everything plugs into around streaming and make that like a really default way of working, you know, a really obvious way that systems would interact and the applications would be built. And that's as important as the kind of database world in companies.

So that's, you know, that's the journey that's ahead. And, you know, to do that, of course, is well beyond Kafka. It's really how all these parts come together into a unified product. We look forward to what comes next. And Jay, thank you so much for joining us today. We had a great time speaking with you and learning from you. We have a ton of more questions to ask. So we do hope there is a second time and we do get to speak with you again in the future. But for today, thank you so much. My pleasure. Thank you, guys. Hey, thank you so much for listening to the show. You can subscribe wherever you get your podcasts and learn more about us at softwaremisadventures.com. You can also write to us at hello at softwaremisadventures.com.

We would love to hear from you. Until next time, take care.