Behind The Tech with Kevin Scott - Neha Narkhede: Author, entrepreneur and Confluent cofounder

Episode Date: September 26, 2019

Neha is cofounder and CTO of Confluent, a company whose mission is to help other companies use Apache Kafka to adopt modern streaming infrastructure.   Check out our other shows about technology, visit: The Intrazone: https://aka.ms/TheIntrazone Sync Up, a OneDrive podcast: http://aka.ms/syncup

So, you know, why we wanted to open source is this was an observation we made is, you know, this trend that technology companies are going to essentially be defined in software, you know, not just use more software, but literally all the actions and decisions in a company happen through software. It's going to Behind the Tech. I'm your host, Kevin Scott, Chief Technology Officer for Microsoft. In this podcast, we're going to get behind the tech. We'll talk with some of the people who made our modern tech world possible and understand what motivated them to create what they did. So join me to maybe learn a little bit about the history of computing and get a few behind-the-scenes insights into what's happening today. Stick around. Hello and welcome to Behind the Tech. I'm Christina Warren, Senior Cloud Advocate at Microsoft. And I'm Kevin Scott. Today our guest is Neha Narkeda, co-founder and CTO of Confluent, one of the initial authors of Apache Kafka and a former colleague of mine.

And, you know, Neha is on a Forbes list of the top 80 self-made women in America. And this is actually kind of an amazing list. It includes women in sports and TV and music. Beyonce is on this list. Serena Williams and Oprah are on this list. But it also has a lot of women in tech and in business like Sheryl Sandberg, who's the COO at Facebook, and Safra Katz from Oracle, Marissa Meyer, and a number of other women in tech and business. And I think it's so cool that women in tech are counted among some of the most successful entrepreneurs in the country. And I'm really curious to hear about Neha and how she got to where she is today. Yeah, I mean, it's no surprise to me at all that Neha is on this sort of list.

It may actually be the closest I'm ever going to get to Beyonce is my association with Neha. But, you know, I got the privilege at LinkedIn of watching her develop as an engineer and as an engineering leader. And she's absolutely spectacular. So I'm just really excited to chat with her today and to have her share with everyone else like this really amazing in some ways, like an archetypical Silicon Valley tech industry story with everyone. She's awesome. All right, let's hear from Neha. Awesome. So today we'll chat with Neha Narketa.

Neha is one of the initial authors of Apache Kafka, an open source software system that lets organizations process data in real time. Neha helped develop Kafka at LinkedIn, where she was the lead of the streams infrastructure team. Neha is co-founder and CTO of Confluent, a company whose mission is to help other companies use Kafka and adopt modern streaming infrastructure. So welcome, Neha. Thank you so much for being here today. Thank you for having me, Kevin. Awesome. So I want to start with your journey through computing. So how did you get interested in computers and programming in the first place? You know, Kevin, I was a very curious sort of self-learner nerd as a child.

And so when I was about eight, my parents brought me my first personal computer. And it was sort of unique in those times back in India to have a computer. And so somewhere deep down, I was very appreciative. And then it sort of became like the tool that fueled my curiosity. So early days was just playing Prince all day. And I was big into art. And so I just drew on it with MS Paint or something and pretty basic. And then it taught me, you know, computing, programming. And so that was sort of the start of the journey. My dad used to tell me a lot about technology and how it's going to have a huge impact in basically everything we do. And so that would be a good career bet for someone who wanted

to learn so much and create new things. And so that sort of planted that seed in my head. And was your dad an engineer? He was a mechanical engineer. And in our family, there were a lot of emerging technologists who moved to the U.S. And one of my uncles started Cirrus Logic, and that went public and sort of, you know, started from nothing. And so I sort of saw that journey and thought, wow, you know, sitting here, I could actually dream about being a technologist. So role models were really important. They were really important. And my parents did this thing where, you know, theyira Nooyi, who basically had origins in India and went on to become CEO of a multinational company, Kiran Dev, the first female head of

Indian police services. And that sort of, you know, somewhere, you know, deep down, I think it sort of developed a sense of empowerment. Now that I look back on, hey, if they could do the impossible, maybe I could try it too. Yeah. You know, and so I think I had a bunch of different sort of role models, but my parents sort of, you know, made me believe that if they could do it, then I could do it too. That's really fantastic. Yeah. So it must have been a big change leaving India and coming to the United States

and going to graduate school at Georgia Tech and being in Atlanta, which I'm guessing is a very different sort of place than where you grew up. Yes. How did you decide to do that? What was that decision-making process? I think it happened over a course of time. So my brother decided to do his bachelor's here. So that was pretty early. He came here when he was 17. And back then we had this, you know, family green card, and it was very hard to get that. But my father got it and my brother got it.

And I said, I don't want it. You know, I don't want to move to the U.S. This was, you know, maybe I was in 10th grade or something. And so we gave it up. So I actually gave that up, the golden ticket to the U.S. And then later on, you know, as I was studying computer science and, you know, I got to sort of see its impact on, you know, basically every aspect of our lives. And, you know, back then I was just sort of doing that by reading up on the internet. And so, you know, when I first thought about taking it seriously is when I said, well, now I want to go to the U.S.

And it is sort of a big deal, you know, if you come from the middle class to sort of go to an expensive university, that was going to be a pretty big leap. And by then, I had earned, you know, scholarships to sort of fund my education, right? So I just had like free education up until my bachelor's of computer engineering. So what I told my parents is, I don't want you to pay, I want to try to get fully funded, right? So I made a list of all the universities where it was possible to get either a research assistantship or maybe a teaching assistantship, and I applied there. And that was my journey.

I think I worked pretty hard. It was pretty hard to do that. And so when I spent the first few weeks, I spent it in Silicon Valley, right? That's where my family lives. And from there, I go to Atlanta. And it was a huge shock. I mean, everything is different.

You know, the way people interact with each other is different. Georgia Tech is in Midtown, and that's right in the heart of Atlanta. And that's very different from suburbia in the valley. And Georgia Tech is like a really good computer science and engineering school. And so, like, not only was it this new city and new culture you're getting accustomed to, but I'm sure that first year was pretty hardcore. It was pretty tough. They sort of have these mandatory operating systems and theory classes. And I'm still wondering like what I actually learned in theory and if it was applicable to what I do right now.

But it was pretty hard, right? And I was ambitious. I wanted to finish it within three semesters. Didn't want to stay all four semesters in order to get the right H-1B, you know, path out. So, yeah, it was a tough, I think, period of struggle, I would say. What was your favorite class, either undergrad or grad, when you were studying computer science? I want to say it was, and I was surprised that it was operating systems because it wasn't just, you know, reading books and taking tests. It was actually reading a bunch of research papers and then in the second half actually building something new, working in groups. And so I think that sort of hands-on experience and using the latest sort of research to learn what's happening in the operating system world was the most interesting part. But I almost flunked, I think.

There was a point when I was like, I need to pull two-nighters. Nice. Well, so, you know, it's really interesting. When I was studying computer science, both undergrad and in grad school, the operating systems course was usually, it was either that or compilers, where it was the first place where you had to write a really big program for the first time. And it was, like, complicated. It was the first place where you're really write a really big program for the first time. And it was complicated. And it's the first place where you're really thinking about software architecture and how all of this stuff fits together

and how you're dealing with all of the modules and decomposition of things. And so was that the first big code that you dealt with was in that OS class? Pretty much. That's why it was hard, and that's why I had to pull like two-nighters. But that was, I think it was interesting and in some ways sort of built the foundation that helped me with Apache Kafka because it sort of teaches you force principles thinking, you know, right from the primitives, right, of how the kernel works and how memory works and how multi-threading works.

It sort of lays that foundation. And that program, the reason it was really hard is because it was the first time I had to actually work in a group where what we decided to do is each write different components and then integrate it at the very end. And that's when I learned that integration is where most of the trouble happens. So it took two whole nights, you know, to resolve that. But I think the first principles, hands-on thinking and programming, was really the magic. Yeah.

Well, I mean, it's interesting that you had that experience and that it was so challenging. Like, everybody learns a lot when you go through sort of a gauntlet like that. But the more interesting thing is, like, the thing that sort of defines your career is, like, you really are a systems engineer. Yes. And so, like, you sort of face this big daunting challenge, and then you decided at some point to lean into it. Like, this is what I'm going to spend all of my time doing is, like, building distributed

systems. Absolutely. That's what I first started thinking about. I want to be a systems, you know, and more importantly, distributed systems programmer. So what did you do after you graduated? You know, I sort of focused on databases more in my master's. So when I graduated, I actually, before I graduated, I interned at Oracle in the database group. And so I decided to take my full-time job there because this was beginning of 2008.

And so we were facing a lot of challenges. You're a lot younger than I am. Yes. Yeah. In some sense, it gives away your age, isn't it? But this was like beginning of 2008. And, you know, just before graduating, I had taken this business economics class where we were putting together a case study about the top five banks. And so we could see it coming, right?

And so I decided to take the job with Oracle rather than go to the startup world where I really wanted to be. It's because I really wanted to make sure I can stay in the country safely, right? So it was a very H-1B safe path into Oracle. So I think Oracle was an interesting experience, I want to say. On one hand, I got to learn what it takes to create high-quality enterprise software. Which team were you in at Oracle? I was in the database search team. So Oracle Text is what it was called.

It's search within the database. And so it was interesting, I thought, but it was also slow. The velocity at which I wanted to learn new technologies, going back to my roots, I had given up a lot to get there. And so I wanted to make sure that I'm learning a lot. And so that's when I spent one year into Oracle. I made up my mind that I want to take the risk anyway and just venture into the startup world

with a particular focus on open source communities because I thought that open source is going to allow me to learn more. And LinkedIn was on the short list because we had a nice page where we advertised our open source projects and the teams that worked on it. So it seemed very cool. Yeah. And I'm going to want to go back to that in just a minute because, like, I think that's one of the very clever things that Igor did. And, like, we'll talk about who Igor is in a second.

So did you have like clarity in your head about how the immigration stuff was going to work then? Or was it still like a big risk leaving stable Oracle? Like when you joined LinkedIn, were, like, I forget, like, were we public yet? No, we were not public. Yeah. So like you're joined in this like private company that you like, you don't know when it's going to go public. Like, there's still all of this uncertainty around it. So, like, how were you thinking about the H-1B?

You know, what was going on in my mind is the H-1B thing was, you know was sort of the main driver for going into a big company, but behind that were years of struggle to get to that point. And so as I was struggling one year down the line to think about, hey, what should I optimize for? I'm here now, and it's a different country, and all my people are far far away and I've given up a lot to learn so much, is I talked to my husband and what he said is, you know, you do what you, you know, you want to minimize your regrets. And so if you're here, you want to think about, you know, what would minimize your regrets in the next five years. And so that's when I said, well, I really need to go dabble in the startup world and just figure it out. And if it goes wrong, you know, I'm going to try and come back to a big company anyway. So back then, when I told my friends, you know, this thing about LinkedIn is people just didn't believe that it was going to be this big behemoth, right, of a company.

They were just like, it's going to be bought by Google and it's, you know, maybe going to be a mid-sized company. And I was like, well, let me just interview there, right? And so when I interviewed people, the people really convinced me. You know, sort of the culture was of inclusion. And when I interviewed at a couple other social network, networking companies, I didn't sort of get that. It was a lot of, like, you know, nerdy, sort of individualistic, move fast.

And LinkedIn, I think, really convinced me that it was about this community and about the open source contribution. And that's kind of why I weighed the risks and said this is the right company to take the leap with. Yep. And so when you joined LinkedIn, you went into Igor Parizik's group. So Igor is now LinkedIn's chief data officer. And Igor worked for me for six years. Wow. Yeah. And Igor is – every time I see him, I tell him he's my favorite Swiss PhD, which he is. And he's also one of my favorite people in the whole world.

Like, Igor is like a wonderful, brilliant technologist and a genuinely good human being. And he was running this group at the time that was called SNA. So, Search Network and Analytics. That's right. And so, he owned the LinkedIn search engine. He owned all of the LinkedIn graph stuff, which is sort of this foundational piece of infrastructure that made LinkedIn possible.

And then he owned all of this sort of analytics stuff, which were like all of the data systems that had to operate at scale. And so it's really interesting that you were attracted to LinkedIn because of the open source stuff, because that's how I was attracted to LinkedIn. No way. So, at my startup, AdMob, we at one point needed to deploy a key value store for our ad system. Yes.

And, like, we evaluated a whole bunch of stuff, including Voldemort, needed to deploy a key value store for our ad system. Yes. And, like, we evaluated a whole bunch of stuff, including Voldemort, which was one of the open source projects that, you know, your co-founder built, Jay Kreps, in Igor's S&A group. And, like, it was the best performing key value store for our particular use case at the time. And, like, we used it. And also because it was open source, like, we not only got to use the software, but we interacted a bunch with some of the engineers at LinkedIn. So you saw, like, it's great code.

It's, like, super good piece of design. And the engineers were great. And so when I was thinking about where do I go, I knew some folks at LinkedIn, but the important thing to me was like, oh, there's some really great engineers there. And whose mindset about open source and how to think about actually doing software development with that open source ethos was aligned with my own. And that's how I chose one of the big, big factors in me choosing to go to LinkedIn. That is amazing. So you might have joined around the same time I joined, right before that. I started in the February of 2011.

Got it. Yeah, I started end of 2009. Yeah, so you substantially preceded me. But yeah, it's sort of fascinating. So like, tell me a little bit about what it was like working in that group. Because one of the things that Igor always encouraged folks to do, and it's sort of set part of like what is now a core value in LinkedIn engineering is like, it's open source first, like you actually have to have

an excuse to say no to open sourcing something. So what was that like as an engineer? It was fascinating. So Igor was super supportive, very technical, and I was just basically extremely fortunate to land in his group. So I started on the search team because I had a search background and I thought, you know, why not? And, you know, when I joined in the search team, what I was doing more of was ETL and data pipeline stuff and how to get data into all the search systems. And I was doing less of, you know, deep distributed systems challenges, which is what I thought I would do on the team. And so that's when I realized that our big problem as a company back then was actually data

pipelines, right? And that was the one thing that was at least slowing or stopping us from building these data-driven products at scale. So I started digging around, you know, I started talking to Chris Riccomini, who was on the analytics side. I talked to Jay, who was thinking about, you know, Hadoop adoption at LinkedIn. And I asked him, like, hey, Jay, what is the biggest challenge you think we need to solve? And he walked me through this data pipeline problem that he was thinking through. And he said that, actually, you know, the main stuff is this Hadoop stuff and the analytics thing. And then the side thing is this data pipeline thing. And I'm just tinkering around,

but no one really wants to work on it. So I'm just going to state it for completion, but no one really wants to work on it. And my experience was the opposite, is that that's all that was important to search and analytics. And so I said, you know, how about I join on that side of the team?

And so I talked to Igor and I said, Igor, I've been doing this and I think this is the biggest problem. And no one is working on such an important problem for the business. And even though it's sort of dirty work, it seems like I want to do that. And he was very supportive. So I thought like a bunch about talking to Igor. Like I thought for maybe an hour and what I'm going to say and like why I'm going to justify this because I'm like a new engineer. I'm only six months in at that time. So I absolutely haven't totally proven myself out.

That's what's going on in my head. But I somehow conjured up the courage to go talk to him and he made it super easy for me. He's like, yep, you know, let's talk to John when he's back. John was the tech lead for the search side. And let's get it going. And that part about LinkedIn's culture really amazed me. And that was super inspiring to be part of, you know, because you don't have to go through a lot of bureaucracy, right? I could just talk to Chris and Jay on one side and Igor on the other side and John. And boom, like in two or three weeks, I'm working on Kafka.

Yeah. And so for the audience who are not all engineers and not all necessarily Silicon Valley folks, like what is Kafka? Kafka started off as a high throughput distributed, super scalable messaging system and has transformed into what's called an event streaming platform. So I'm going to take some time to talk about what an event streaming platform is. But if you think about the database and you sort of unbundle it, you take the transaction log out and you take the thing that creates the tables and the query processing out,

and if you apply it to the scope of an entire company and all its software, then that's basically what Kafka is today. And what we're calling that is an event streaming platform. It has the logs, it has stream processing, which has the ability to create tables on that log, and it has the ability to connect to all the different systems because of that abstraction. And so you sort of described one of the first obvious things that Kafka got used for. So in a modern, large-scale software business, you have lots and lots of things that are producing lots and lots of data. And some of the data is log data. And log data basically is a record of, like,

here's something that happens that you may want to pay attention to at some point. Or you may not. Yeah. Or it could be, you know, sort of events, you know, like some sort of transactional data that is being written into the system. So, like, this is the, you know, the database log, for instance. So, like, here's a thing and, like, you know, we, it's a sale that we made or a click that happened or something, and, like, we need to update some other state

based on this thing that happened, like, somewhere else. And, like, you know, what you describe working on ETL, and so ETL is this process where you take a bunch of this stuff that is flowing out of all of these systems and you gather it all up and you sort of say, oh, I'm going to add up all of the clicks that happen on this particular ad

and I'm going to like store that in a database somewhere that, you know, sort of counts all the clicks so that I can pay the, you know, the advertiser or like charge the advertiser for. Pretty much. Yeah. And it's like a gazillion different things, right? So you need ETL to work so that you can basically reason over what's happened in your system. And so like that's sort of where Kafka

started. It was like, it turns out that, getting all of this data from all of these places where it's being produced and, like, getting it to all of the things that need to consume it, like ETL systems, is actually a pretty hard technical problem. It turns out. It seemed very simple and jaded on the board, which is what got me excited about it. But as I started to look into it, it was pretty complex. You know, back then, LinkedIn was going through its own microservices transition, right? So we were creating a lot of tiny applications. And then we had all these analytical systems and data systems that needed access to the data. And if you think about, you know, what the source of truth was, there actually wasn't one.

You know, the applications sort of were talking through a MQ. And because MQs didn't scale, not a lot of the microservices were talking to the MQs. And we had all this custom tooling and data warehouse-centric tooling that was copying all that data and scraping it at the end of the day. And so the motivation for solving this problem was just like a simple observation, is if you think about LinkedIn, we produce data 24 hours a day, you know, by processes and applications that never stop, that users that were global never stop. But if you think about all the infrastructure that we had to harness that data

and put it to use, it was sort of stuck in this bimodal world where you could either do simple lookups on a relational or NoSQL database, or you could wait for the end of day where you can get like these slow, big batch dumps and query them. And if you think about a global business, what is even end of day? And at the same time, our use of data was changing. It wasn't just the reports that our employees and our exec team needed to look at. It was the problem that you mentioned, which is all this data that was being generated on the site, we wanted a way to take it and supply it to all the software that basically ran the business. You know, every user action we wanted to make a decision on, every user experience we wanted to customize instantly,

not a couple times a week, which is what we were doing. We started by just looking at existing systems, if you remember. It was trying out MQs and looking into why they didn't scale. We invited folks from Scribe, you know, all these different logging systems and tried to see if we could turn that into a real-time thing. And what we realized is it's a fundamental architectural flaw

at the heart of the system. And one of the really interesting bits of genius, I think, about Kafka, like one of the things you all did especially well is, and I think the reason that it's caught on so broadly and gets used for so many things, is that you tried really hard to make as few assumptions as humanly possible about what was going to be producing data and what was going to be consuming it. Because a lot of the failure of these other systems like ActiveMQ and like we had this thing inside the company called DataBus is that they made a bunch of assumptions about

what it is that was producing the data and like all sorts of things like event ordering and... Absolutely. And... The GMS API. And the more constraints that you added, like the less utility you could get out of the messaging system as a whole. And it sort of restricted how it could be used, and it made it cumbersome to reason about and to scale and to manage. Absolutely.

And Kafka is really this brilliant piece of design because it's very, very easy to just take something that makes data and, like, write it into Kafka. And then it's there for the producer doesn't have to think anything at all about the consumers. So anybody can consume the data. I mean, at LinkedIn now, like, there are thousands of unique things that produce data and write it into Kafka. And there are thousands of unique things that produce data and write it into Kafka. And there are thousands of unique things that consume data. And like, they don't have to know anything about each other. And that was the heart of the, you know, design thinking that went in is, A, it needed to be, you know, it needed to serve decoupling, right? So decoupling of the schemas from the producers

and consumers, decoupling of producers from consumers. And the third thing, which is why we really created and made it a log, is this realization that if we wanted to serve this use case where the whole company is going to need to publish and the whole companies need to subscribe to it, is it needed to be a multi-subscriber retentive model for messaging. Right. And that was really the heart of that change is simple APIs, retention, leave it to the user to decide, you know, how to configure that system appropriately,

and then made it super simple and scalable. Yeah. I mean, I think another one of the genius design things about Kafka and, like, one of the things that you all at Confluent are talking a lot about right now is that thinking about systems as like data producers and data consumers with like a messaging and like stream processing mechanism connecting them really is like a way to sort of disaggregate data systems. Very much. You know, intellectually, you could solve a whole bunch of these data transport problems

by just writing all the data to a database. Like, you write them in with SQL, you read them out with SQL. But the problem with that is then you've got this sort of monolithic database thing. It's coupled tightly. It's super complicated. And like you just like all the different pieces of the database system scale differently. And so like pulling the message and stream processing substrate out lets you sort of scale the system in a natural way. It's almost like how you broke up search engines into like different components.

And it's like how we. Absolutely. It is that materialized view is not just in the form of a relational table. It could be in the form of a search index. It could be in the form of a graph engine. It could be in the form of a simple log or a Hadoop file or something. It was that sort of, you know, decoupling to a point where you can actually just create these streaming materialized views in whatever fashion and downstream system that you could conceivably think about. Yep. And it also sort of separates out the failure domains of this thing. So it's easier to operate.

So like, you have all of these independent systems. And like when one fails, it doesn't necessarily mean that everything fails. And it's like easier to debug when debug when, like, the things that are failing independently one another are, like, simpler themselves. Absolutely. And it's this form, you know, it's this idea of accounting applied to software systems where if only you could remember all the events that happened rather than just the right answer at a point in time, you could actually troubleshoot much better. So if you remember, the way we built Kafka and the way applications used it is, it's just sort of like a log that keeps track of all the events. So if you make a mistake in today's deployment and you realize it two days later, you can actually go back and rewind and replay.

And I think the rewind and replay was really the favorite feature that led to Kafka's broad adoption is, oh, wait, I could actually make a mistake and actually go back and it would allow me to go back to a point in time and an offset in the log and replay the source of truth of what happened. Not my interpretation of what happened, but

actually what happened. So, you all built Kafka. Its initial uses were inside of SNA, which is great because that's where some of the most complicated distributed systems were in the company. It worked brilliantly. And so at some point, we started using it for everything inside of the company, like the site reliability engineers, like built all the monitoring and alerting stuff on top of it. It's just basically everywhere. And we decided to open source it.

So, like, talk about, like, why did you all want to open source it? And, like, what did you get from open sourcing it? Wow. So, you know, why we wanted to open sources, this was an observation we made is, you know, this trend that technology companies are going to essentially be defined in software, you know, not just use more software, but literally all the actions and decisions in a company happen through software. It's going to be a trend. It likely is not limited to LinkedIn. LinkedIn is not the only data-driven company. It's that every company in the world is going to become a data-driven company.

And so we wanted to sort of get a validation for that trend and decided to open source Kafka as a result of that. And I think LinkedIn made it very easy to open source Kafka. We were in the group that did that. But we did one more thing, which is we donated it to the Apache Software Foundation. And that was uncommon for LinkedIn, is to give up, you know, the control of the software that was being built and give it to an independent entity. I think that really led to its broad adoption. Although that wasn't a hard decision. Like Igor, you all ask Igor. Igor said, I think it's reasonable. Igor asked me, and I'm like, it's fine.

Yes. And it was done. And it was done, right? And then 2011 is when it was donated, I think, to the Apache Software Foundation. So it was very, very simple to do that. to get built around Kafka because they sort of saw Kafka as not just a LinkedIn thing, as like an independent project that could develop a whole open source community around it. I think that did wonders to Kafka's adoption in the world is it reached beyond Silicon Valley in about two years since we open sourced it to a point where I was on a call with June, my other co-founder, and we were helping this big Fortune 500 just for free about their Kafka problems.

And as I sat there, I realized that, wow, Kafka has gone mainstream. And in order to really fully realize the next level of Kafka's adoption potential, there will be a company around it. And I just sort of thought that, hey, if there is a company around Kafka and it's not three of us, that would be a shame. So I pushed that idea to both of my then teammates, now co-founders, and it turns out that they were also interested. So that's when we sort of began the, you know, brainstorming sessions to start Confluent. This was five years ago. Yeah.

So even though you had this traction with Kafka, it's still a big decision to go start a company. Yes. In fact, I was in three companies in a row that were, you know, sort of private and like ramping up. And it's an intense, intense experience. So, how did you all get the courage to do this? You know, I don't know. I can't speak about my co-founders. But what ran through my mind when I decided to take the leap, and I'm so thankful that they joined me in that endeavor because it's going to be much harder as a single co-founder to do this, is the same thing that ran through my head when I decided to take the leap to LinkedIn is basically, I think, FOMO.

There was just this fear of missing out that, hey, there is a company around the thing we created. That's just not going to work for me. Yeah. No, it's just we had to do it. That was one. And the other thing was it was very clear that it had turned into a phenomenon, right? And it was clear that it needed to have a first-class place in every company's data architecture. And that sort of investment, it really requires like

a business vehicle around it. That was really the second thing that went through my head is, hey, it's like just practically speaking, we're going to need to devote all our time, you know, not just partially, but think about all the companies, you know, beyond LinkedIn that could use this, right? So I think those two things ran through my head is fear of missing out and impact. Yeah. And look, I think you've said two things that are like people should just sort of pay close attention to because it's phenomenally good advice. So there's this fear of missing out or regret minimization. Absolutely.

Like minimize the number of regrets that you're going to have over a period of time. So like that is really good advice. And the other thing is, like, you know, put yourself in situations where you can maximize the amount of stuff that you're learning. Yes. Like, if you do those two things, like, you can accomplish a lot. A lot of things. You can accomplish a lot of things. And at the same time, you are uncomfortable almost most of the time, I would want to say. Yeah, actually, I find the discomfort a sign that you're doing stuff right. Like, a lot of people try to avoid discomfort. Discomfort. And, like, of course, there are flavors of discomfort that you should avoid.

But the discomfort that comes from trying something really, really hard and not actually being perfect at it is good. It is. Because that is the thing that sort of drives progress and that builds skill and that, like, actually, you know, makes change happen. Absolutely. I think, like, discomfort, I realized that through my master's even, it's just, there was a ton of discomfort in coming to a different country and learning about computer science from first principles, very different style of teaching. And that's when I realized that I'm just going to have to be comfortable with that discomfort if I wanted to learn things, which is what I really wanted to do. And so I think that sort of training almost helps a lot in starting a company

because you're just going through one obstacle after the other, after the other, and it just literally never ends. Yeah. And so like, let's talk about Confluent. Like, I still remember the day that Jay walked into my office. And I think he thought he was going to get a different reaction out of me than he did. I came in like almost worried. It's like, okay, I'm delivering bad news. And I told him, I'm like, oh, this is actually pretty cool. That was the other unique thing about LinkedIn's culture. We were out in a month with no drama, a lot of support from everyone, from you, from Jeff, from Reid, and you invested in the company. And that is unheard of.

You know, there were a lot of, there were a couple of other stories that did not look like that back then. Yeah, but what you were doing made sense. Like, we were never going to, like, LinkedIn is a social network. It's very important when you have a company that you have, you have a mission and vision and you focus on the mission and vision. And, like, LinkedIn's mission and vision wasn't building distributed streaming infrastructure for the rest of the world. And so, like, I was completely supportive of you all going out and, like, finding your own mission and vision that, you know, took this thing that you helped build at LinkedIn and, like, now you're going to make it useful for other folks.

I still get asked that question. It's exactly how did that play out internally. I'm like, it was very simple. And that's just, it speaks volumes about LinkedIn's culture, I think. And, like, the thing that I wanted to make sure that we had, and, like, I think by and large it's worked really well, is that there's still a team of people at LinkedIn. Like, a pretty big team of people. Pretty big team, though.

40 people, I think. Who work on Kafka because Kafka is a very important piece of infrastructure for LinkedIn. I just wanted to make sure that, like, you guys and they were going to be able to continue to collaborate with one another. And, like, that's worked pretty well, I think. That's worked pretty well. I think. That's worked pretty well. I mean, it's not always perfect, but it's worked really well. It does. It does.

So how has it been? You co-founded a company. Your start, there's nothing. You've got to figure out the business model. You've got to figure out how are we going to build software? How am I going to hire engineers? How do we, you know, like, where's the office going to be? Like, so talk about this.

Like, what were the big challenges in the first days? I think the big challenges were there was a time when we spent before even we started the company on what the business model was going to be. And the big question is, you know, do you want to have a software business model? We are going to sell subscription on software packages. Are you want to have a software business model? We are going to sell subscription on software packages, or are you going to have a fully managed service? And back then, five years ago, it actually was not that clear, and it wasn't as simple of a decision as it is now for data systems. You just start with a fully managed service. That's it.

And so we thought a lot about that and realized that Kafka is a data integration system after all. It needs to be where most of the data is. And we were told that most of the money is in the enterprise and enterprise's data is still on premises. And so that one was, it turned out to be a very important decision that we made, is to start with a software package so we could address the needs of all these enterprises. And then in time, and this is something we did, I think, well, is in time also transition to the fully managed service part as they're moving to the cloud. But in the early days, hiring engineers was the focus. And I want to say that it was a tad little easier for us

than it is for startups that just start with nothing. Our big pitch was Kafka is everywhere. And so you want to come work on the thing that's everywhere and help us build the best product around it. And so those were the two sort of pivotal things that we did in, I think, year one. And that's how the early customers came. And what I was surprised about is that is how the latter customers also came.

And that is how the cohort after that also came. It's this crazy inbound vehicle, which was another thing that just doesn't work like that in enterprise companies. You have to build demand for the product that you're producing enough so you can commercialize it. And for us, it was sort of the opposite. And that was a great experience, and it still is. Well, look, I don't think every software design project is going to turn into a business quite as successful as Confluent, but everyone should be thinking about this. Like, part of the reason I think that you all are as successful that you have this crazy inbound is because of, like, honestly, software architecture decisions that got made years

and years and years ago. Like, Kafka's just easy to use. Yes. Like, it's no more complicated than it needs to be. And, like, that is a really fantastic thing. Pretty much. You know, in a way it's like MapReduce, right? It sort of abstracts away a bunch of like super complicated, like hard distributed system stuff and like lets you just sort of pay attention to like here's my data, here's my applications, like it solves real problems.

And it just scales. You throw more data at it, a couple more servers, and it just works. And that sort of it just scales and works was sort of the real magic behind that adoption. And we didn't realize this years ago when that adoption was happening is how important it would be to make the company building part easier. Not easy. part easier, not easy, but it sort of turned into a big machine for us to then harvest and commercialize. So there obviously been a bunch of technical things that you all had to go sort out. But like, what are some of the non-technical things that have been interesting challenges

for the company? So the non-technical things are, you know, the first lesson we had to learn is the role of marketing in forming Confluent and Confluent's demand generation engine is just sort of, we had to go through our own learning experience of like just what kind of DNA we need and what was its role in the organization and how technical it needed to be

because our end user persona is still the engineer, the person who first makes the decision to adopt it. And that was a pretty big learning moment for us is this whole inbound motion. It takes first-class investment in what luckily we were good at, which is evangelism and thought leadership. Believe it or not, that is really the investment that actually works with these distributed data systems. No matter how big you get,

that's the way you generate demand for your system. And that was a whole different way to do enterprise marketing than what you imagine enterprise you know, enterprise companies need to do. That was sort of one. And then the other one was the role of sales in enterprise companies, which is, you know, you imagine sort of sales have to go do this big outbound qualification and call into companies. And for us, it was sort of the reverse, which is how do we best harvest all this demand that existed around Kafka and convert it into the best possible vehicle for Confluent. And for the company, those were the two sort of big go-to-market learning lessons for us. And it's sort of unique about, I want to say, open source-based enterprise companies, not so much just about any enterprise company, which just looks very, very different from Confluent.

And then the role of product in particular was another learning lesson. It's a very engineering-driven product and a very engineering-centric adoption cycle. And so the role of what product management should do two or three years in, we hadn't figured it out. And it was so hard and we tripped and fell. So I decided to actually move from engineering and technology into just product is product management and building out, you know, the right product team and doing product marketing. And that whole transformation alone was hard but deeply satisfying. Yeah.

And what were some of the challenges there? So I can imagine what some of them might have been, but I'd love to hear how you made the transition. So the first challenge for me was just learning about product management from first principles. So I followed my playbook, went and read every book there was on product management and recruited a couple of mentors. And that's how I quickly sort of got myself up to speed. What was easy is I was running and had built the engineering team by then. So what I understood is what engineers needed from product managers, right? So we needed to retain the engineering

centric culture and supplement and complement it with product to actually help us prioritize and get the customer feedback in. Because we were not, you know, if you just build for what you think is right technically, you're not really going to end up with the right product that your customers are going to buy. And so that was one. The other one is the DNA of product managers that we needed, not going to succeed because they come in with a certain way of doing things and with not as much of a foundation in technology and systems. And so the second thing I did is make sure we hired product managers who are engineers just by training, who had done computer science, who had done some software programming, and had then learned product management.

So it was a much easier transition. I think that was the second. And the other thing was, how do these teams work together? And that was really, really important to establish is we are not going to come up with plans in a room full of product managers. That's just not how planning happens at Confluent. It's every step of the way engineers are involved, no matter what we do, even if it's prioritization,

even if it's how we are selling. They're there. Yep. And look, I think it's really good to hear you say that you were thinking so much about culture because I think culture is sort of like the, it effectively turns into the playbook for collaboration

inside of organizations. And it's just at some point when the scale of what it is that you're trying to accomplish gets big enough that you can't sort of fake good culture. Pretty much, yeah.

It happens in your first hundred people, I think. Yeah. Yeah, I mean, I think as soon as you get past Dunbar's number, it becomes existentially important. Like, if you don't have a good culture, if you don't have a culture that encourages the types of collaboration that you're going to need to build the things that you need to build, you are screwed. Absolutely.

So that's awesome to hear that, like, that's how, I mean, and this, again, I think is another lesson for deep technology people. Like, at some point, like, this is just sort of the necessary work that you have to go do, even though it's not like figuring out the locking algorithm on this leader election thing that you're trying to do. It's, you know, at certain points in time in the development of a company, like, getting that culture right and, like, figuring out, like, how you're going to get people to, like, play the right roles and to, like, collaborate effectively with people in other roles is, like, the most important thing.

Culture is, like, the shared set of values, right, that you want your people to sort of stand on top of and respect. And I just sort of think of it as, you know, the values that we appreciate that, you know, gets you sort of that just allows you to progress in an organization and the values that don't work, right, that just will not work in an organization. And those sort of boundaries is the way to think about culture. And just like the way to scale a company is by getting it right in the first hundred. Cool. Well, so I want to shift away from tech and business for a second. Let's talk about you. What do you do for fun?

You got this super high-press pressure job, the company's growing like crazy. I'm guessing if your experience is anything like mine, it is probably intense. It is very intense. So how do you have any sort of balance in your life? I strike a balance and my favorite activity to strike a balance with is to travel to new countries and experience new cultures. And that's what me and my husband do. And our crazy hobby together is to go scuba diving. Oh, wow.

Some of the, you know, crazy locations. Oftentimes to see different varieties of sharks. So I can't, I can't. I've been in one of those cages where great whites are on the outside and you're on the inside. I can't confirm if I was scared or not, but I would say I survived it and it turned out to be fun in a very weird kind of way. Yeah. You might be a little bit braver than I am. And you, you mentioned before we started recording that you've got a trip coming up where you're going to do some high altitude stuff in the Himalayas. I'm going back to India for a couple of weeks, mostly for work, but we're going to go up to the Himalayas, hopefully to get a very good view of the Everest from Kalapathar.

Awesome. Well, definitely take some pictures and be safe. Thank you. Absolutely. Awesome. Well, definitely take some pictures and be safe. Thank you. Absolutely. Awesome. Well, thank you so much for spending some time with us today. This was fantastic. Thank you for having me, Kevin. It was great. Awesome. Well, we hope you enjoyed Kevin's interview with Neha Narkeda. That was such a great conversation. Like you said, she kind of has that archetypical Silicon Valley story.

I thought it was really interesting how she read biographies of famous women and seemed almost to kind of prepare herself for what she wanted. She's had this vision in her mind for a long time. Yeah, I think it's one of the really consistent things that we see with some of the folks who've accomplished big things that we've chatted with on the podcast is that at some point you actually do have to have a vision of where you want to go. It doesn't necessarily need to be like a super clear vision where you've got every last little, you know, waypoint on the journey that you're about to take mapped out. But like, you do have to know approximately where it is that you want to go. And like, one of the really impressive

things about Neha is like, I think she, over and over and over again, throughout the course of her career, as like we heard in this conversation, has had just this very clear notion of where it is that she wanted to go. Yeah, I love that. I love that vision and that drive. What I also think was really interesting is, you know, the story of Kafka, you know, and how, you know, she works on this and the story of how it goes from being this internal tool to being something that's contributed to the community, that's open source, that's used all over the place. But then it's also, you know, she's able to kind of, you know, spin it out and make her

own company based on that. That's in some ways kind of the archetypical modern Silicon Valley story when it comes to open source. Yeah, it is. And, you know, it's one of those things where I'm really glad that we open sourced the technology. Like this is the way open source really should actually work. Like, LinkedIn and now Microsoft at a pretty large scale are investing heavily in contributing back to the open source community. Like, we consume a lot of open source software ourselves. And so, like, you know, it's the right karmic balance to contribute back. And there's more than one way to contribute. Like, one way is, like, obviously,

it's sort of the way that most contributions happen, which is, like, you sort of open source a project and you do your development work partially or completely out in the open, which has a huge number of benefits, some of which we chatted about in this conversation with Neha. But, like, another way to contribute that is, like, I think, you know, as you pointed out, that's really powerful is, like, allowing companies to form around these open source

ecosystems that are getting a lot of traction. And, you know, if you look at Kafka's traction inside of LinkedIn, it is staggering. Like, there is a pre-Kafka world for LinkedIn infrastructure, and then there is a post-Kafka world for LinkedIn infrastructure, and like it is starkly different. Kafka, in a sense, lets you sort of think about how you build infrastructure and how you build infrastructure teams and their culture in a different way. It sort of decouples everything and just sort of like lets a lot of creativity happen because it removes like one of these control points that exist in many sorts of infrastructure stacks.

You know, the separation of concerns that sort of let people who have data publish it without having to imagine every possible use and to allow folks who need the data to sort of let people who have data publish it without having to imagine every possible use and to allow folks who need the data to sort of consume it for the application or use case that they're building, like, it's a super powerful notion. And so it's awesome to me to sort of see more people benefiting from that, like more companies, and I'm glad that Confluent exists to help make that happen. Definitely. I love, like you said, it's one thing, A, it's amazing when these internal tools are open sourced, but then when it goes to the next level and you are able to see companies built off

of that technology and are able to make real businesses for themselves, that's kind of the goal, right? Because for many years, that was sort of the unspoken rule about open sources. Oh, there's not a way to make money. And you had a few examples, you know, the red kind of the goal, right? Because for many years, that was sort of the unspoken rule about open source is, oh, there's not a way to make money. And you had a few examples. You know, the Red Hats of the world would kind of prove that wrong, but it was less common. And now we're seeing, I think, in this kind of, I guess, maybe third wave of open source that I would kind of include, you know, like the Kafka's of the world, the React's of the world, things like that. Nodes, certainly, where you can start to see these real businesses form around the technology. Yeah, it's really exciting because sometimes open source is hard.

Sometimes you can have a really, really great idea and you can even have a really involved community and you still need a little bit more than that to like help tip everything over into like like, more widespread adoption. And so, like, these companies are fulfilling a very, very important role in getting more technology in the hands of more people, which, in my opinion, and I'm biased, is, like, a really good thing. I'm biased, too. That's one of the things I advocate for in my day job. So, I'm glad to see more of this. All right. Well, we are at the end of our show today. Thanks to Neha. And last show,

we asked you to let us know what technologies you think are inspiring kids in the next generation. And survey says? Well, we got some really interesting responses. The most prevalent one centered around augmented reality, which is kind of not a surprise. And that is due to all the gaming with Fortnite and Minecraft. Awesome. We shall see. I certainly hope so. And as always, we would love to hear from you about ideas for future guests or anything on your mind. So you can email us at BehindTheTech at Microsoft.com.

And be sure to tell all of your friends and colleagues and fellow Beyonce fans about Behind the Tech so that you can geek out together. Thank you for listening. Indeed. Thanks, and see you next time.