Lenny's Podcast: Product | Career | Growth - “Engineers are becoming sorcerers” | The future of software development with OpenAI’s Sherwin Wu

Starting point is 00:00:00 95% of engineers use Codex. 100% of our PRs are reviewed by Codex. For engineers, I don't know what job has changed more in the past couple years. Engineers are becoming tech leads. They're managing fleets and fleets of agents. It literally feels like we're wizards casting all these spells. And these spells are kind of like going out and doing things for you. What do you think people aren't pricing in yet?

Starting point is 00:00:19 The second or third order effects of the one-person billion dollar startup. To enable a one-person billion dollar startup, there might be 100 other small startups building bespoke software. So I think we might actually enter into a gold and name. age of B2B SaaS. I've been hearing more and more, there's this stress people feel when their agents aren't working. There's a team that's actually doing an experiment right now with an Open AI where they are maintaining a 100% codex written code base. They run into the exact problems that you're describing.

Starting point is 00:00:43 And so usually you're like, all right, I'll roll up my sleeves and figure it out. This team doesn't have that escape hatch. You've shared that listening to customers not always the right strategy in AI. The field and the models themselves are just changing so, so quickly. They tend to like disrupt themselves. The models will eat your scaffolding for breakfast. What's your advice to folks that are like, okay, I don't want to miss the boat? Make sure you're building for where the models are going and not where they are today.

Starting point is 00:01:05 There's a quote from Kevin Whale, our VP of Science here. You like saying this is the worst the models will ever be. Today, my guest is Sherwin Wu, head of engineering for OpenAI's API and developer platform. Considering that essentially every AI startup integrates with OpenAIs APIs, Sherwin has an incredibly unique and broad view into what is going on and where things are heading. Let's get into it after a short word from our wonderful special. Today's episode is brought to you by DX, the developer intelligence platform designed by leading researchers. To thrive in the AI era, organizations need to adapt quickly. But many organization leaders struggle

Starting point is 00:01:42 to answer pressing questions like, which tools are working? How are they being used? What's actually driving value? DX provides the data and insights that leaders need to navigate this shift. With DX, companies like Dropbox, booking.com, adion, and intercom get a deep understanding of how AI is providing value to their developers and what impact AI is having on engineering productivity. To learn more, visit DX's website at getdX.com slash Lenny. That's getdX.com slash Lenny. Applications break in all kinds of ways, crashes, slowdowns, regressions, and the stuff that you only see once real users show up. Century catches at all. See what happened, where, and why. Down to the commit that introduced the error, the developer who shipped it,

Starting point is 00:02:29 and the exact line of code all in one connected view. I've definitely tried the five tabs and Slack thread approach to debugging. This is better. Sentry shows you how the request moved, what ran, what slowed down, and what users saw. Sear, Sentry's AI debugging agent, takes it from there. It uses all of that Sentry context to tell you the root cause, suggest a fix, and even opens a PR for you. It also reviews your PRs and flags any breaking changes with fixes ready to go. Try Century and Sear for free at Century.com.com.

Starting point is 00:03:03 and use code Lenny for $100 in Century Credits. That's S-E-N-T-R-Y.com. Sherwin, thank you so much for being here and welcome to the podcast. Thank you. Thank you for having me. I want to start with what's feeling like a barometer of progress in AI, especially in engineering. What percentage of your code, if you even write code anymore, and your team's code, is written by AI at this point.

Starting point is 00:03:33 I do write code occasionally now still. I'd actually say for managers like myself, it's way easier to use these AI tools than to manually code at this point. And so I know for myself and some of the other EMs engineering managers at OpenAI, all of our code is written by Codex at this point. But more broadly, there's just been this,

Starting point is 00:03:52 there's just so much energy. There's like a tangible energy internally around just how far these tools have gotten, how good codex as a tool has gotten for us. and it's a little hard for us to exactly measure how much of the code is written, because the vast majority of it, I'd say close to 100% is usually generated by AI first. What we do track, though, is, you know, at this point, the vast majority of engineers use codex on a daily basis,

Starting point is 00:04:16 so 95% of engineers use codex. 100% of our PRs are reviewed by Codex daily as well. So basically any code that goes into production that's merged in, Kodas kind of has its eyes on and suggests, improvements, suggest changes in the PRs. And so that's kind of what we're seeing internally. But by and large, the most exciting is just the energy that there is. Another observation that we've had is engineers who tend to use Codex more open way more PRs. So they're actually opening 70% more PRs than the engineers who aren't using Codex as much. And the gap is widening. So I feel like,

Starting point is 00:04:55 you know, the people who are opening more PRs are starting to, you know, learn how to use the tool more and more, get more efficient, and that 70% gap keeps growing over time. And so it might have actually increased since I last looked at the number. Okay. So just to make sure we hear what you're saying, you're saying all of the code of these 95% engineers at Open AI is written by AI. It's written and then they review it. Yep. Yep. It's like crazy that that's almost like not crazy anymore, that we're just like getting used to this. I think there's still some getting used to, to be clear. There's also, I think, some engineers who I think trust codex a little bit less, but basically every day I talk to someone who is blown away by something that I can do and kind of like their bar of trust kind of or like how much they trust the model to do on its own goes up over and over time. And there's a quote from Kevin Whale, our VP of Science here.

Starting point is 00:05:54 You like saying this is the worst the models will ever be. And so this is the worst that the models ever be for software engineering as well. And so over time, we just see people trusting it more and more. And then we'll see the models get better and better as well. Yeah, Kevin Will, former podcast guest, he said exactly that line on this podcast. Yeah, yeah, a few times. Yeah. Peter, the Claudebot slash molt bot slash open claw is what it's called now.

Starting point is 00:06:15 Developer recently shared that he uses codex for his work. And he feels like anytime it does things, he just trusts that it has done the right job. but he's just like almost certain he could just commit it to master and it'll be great. Yeah, yeah, he's a great user of codex. I know he's in close touch with the team, gives us great feedback. I'm not surprised that he uses it. I mean, sorry, it's called Open Claw. Open Claw.

Starting point is 00:06:38 Yeah, open Clause is a great product. And then I saw that this more, I mean, this is very recent, but this morning, I think, Moldt book kind of like with Sherrod as well. And seeing all of the AI agents talk to each other is pretty surreal. It's basically her is happening in real life as what I'm hearing. Yeah, yeah. So just like coming back to this crazy moment we are living through four engineers in particular, we've gone from you write every line of code to now AI is writing all of your code.

Starting point is 00:07:05 I don't know what job has changed more in the past couple years. Like job that we didn't expect to change this much. We're just like the job of an engineer is so different in the entire lifespan of an engineer. Like in the past couple years, it's now shifted to I don't write any more code. How do you imagine the role of an engineer in the way? job of a software engineer looks in the next couple of years, just like, what is that job? Yeah, I mean, it's honestly been really cool to see. And it's part of where the excitement is because, like, the job is likely going to change

Starting point is 00:07:35 pretty significantly over the next one or two years. It kind of feels like we're still figuring things out, though. And so there's like this excitement I know, especially from some of the software engineers, of like, we're in this rare moment, you know, maybe over the next 12 to 24 months where we'll kind of get to figure things out ourselves and set our standards for ourselves. in terms of where I see this moving. So I think there's a common thing that everyone's saying, which is people are generally, like,

Starting point is 00:08:00 I see engineers are becoming tech leads. They're basically like managers now. They're managing fleets and fleets of agents. I know many of the engineers on my team basically have like 10 to 20 threads kind of being pulled on at the same time. Obviously not active running codex jobs, but just a lot of parallel threads.

Starting point is 00:08:19 They're checking in on what they're doing. They're steering the, agents and codex and and and giving it feedback. And so their job has kind of really changed from just writing the code itself into being almost like a manager. In terms of where I think this will go one to two years from now. So one kind of metaphor that I kind of always come back to here is actually from this is from this programming textbook that I read back in college called SickP. I don't know if you've heard of it. Structure and interpretation of computer programs. So S-I-C-P. At MIT, it was really popular, and it was actually used as the introductory.

Starting point is 00:08:57 It was the textbook for the intro programming course for a very long time. And it kind of has this cult following. It teaches you programming, it teaches you a dialect of LIS called Scheme. And so it introduces you to functional programs, like very mind-opening that way. But the thing that was memorable for me about that book, so I kind of read it in college, the very beginning of it kind of describes programming as a discipline and draws this metaphor to basically like sorcery. Like it says like software engineers are like wizards and you're like programming languages are like incantations and you're like you're saying you're issuing these spells and these

Starting point is 00:09:35 spells are kind of like going out and doing things for you and the challenge is like what incantation do you have to say to make the program do what you want. And this book was written in 1980. So this is a while ago. And I think that metaphor is actually like kind of persisted over time. And I think it's actually playing out as we move into this new era of vibe coding or just like what software engineering will look like. Because programming languages were basically these incantations. They've changed over time. And the challenge is always, and the trend has been that these, it's been easier and easier to kind of get the computer to do what you want via programming. And I think the current wave of AI is probably the next stage of that evolution. It is now literally incantations because

Starting point is 00:10:14 you can tell, you know, you can tell Codex, you can tell cursor exactly what you want to do, and then it'll go do it for you. And I particularly like the wizard and like the, the, the source analogy, because I think our current state is starting to move towards kind of like the sorcerer's apprentice, you know, from Fantasia, where Mickey Mouse is like, you know, he finds the sorcerer's hat and he tries to do all these things. And I actually think it's a really apt analogy because, one, it's just, it's really powerful now. These incantations you can do can, is extremely high leverage. But you kind of have to know what you're doing, right?

Starting point is 00:10:47 Like in Sorcerer's Apprentice, the whole plot is like, Mickey goes wild, the brooms like go crazy and everything's flooding. I think he literally sets the, like, sets the brooms off on a task and then goes asleep. And so, you know, it's like vibe coding at its greatest. And then eventually the old sorcerer comes back and like cleans everything up. And, you know, when I see engineers kind of like doing these, these 20 different codex thread it's at a time, there is some skill and there's some seniority and, like, you know, a lot of thought that needs to go into this because you want to make sure that the models

Starting point is 00:11:20 aren't going off the rails. You definitely don't want to just, like, completely go away and, you know, like ignore the thing. But it's also extremely high leverage. Like, you know, a very senior engineer who's really proficient with these tools can now just do way more things via what they're doing. And I think this is also what makes it fun. Like it literally feels like we're wizards. It feels like we're closer to having, to making, making it feel like this magical experience where we're, you know, casting all these spells and having software do all these things for you. I was thinking of the Sorcercerer's Apprentice exactly as the metaphor as you were describing that, so I'm glad you went there. A previous podcast guest described

Starting point is 00:11:57 it as you have a genie that grants he wishes and it's a useful frame because you have to be very clear about the wish you want. Like if you want to be big. Yeah, or it might be like the monkey's paw type thing where, you know, it's like you call. way you want, but what are the side effects. Yeah, yeah, I think that in the analogy is great. And yeah, the crazy thing for me is just the staying power of that book. Sick B, like, it's called the wizard book. You know, people call it the wizard book because that is the metaphor that they kind of

Starting point is 00:12:21 weave throughout the book. And we're, we've basically reached that point now, which is, which is really cool. There's two kind of threads I want to follow here. One is, I've been hearing one more. There's this like stress that people feel when their agents aren't working. You fire off all these, you know, codex agents and then you have to keep stand top of them. Oh shit. one's not working, I'm wasting time.

Starting point is 00:12:41 Do you feel that? Do you feel that across your team at all? Yeah, yeah. I mean, it happens all the time. And I actually think, like, this is where the interesting part of all of this lies right now, because these models aren't perfect, these tools aren't perfect, and we're still trying to figure out how to best interact with these, with codex or with these AI agents to get work done. We see this come up all the time. There's a particularly interesting team that we have internally.

Starting point is 00:13:05 So there's a team that's actually doing an experiment right now with an open AI, where they are basically maintaining a 100% codex written codebase. So, you know, like, you know, you'll have the AI write code, but you'll obviously end up, like, rewriting a lot of it, and you might need to, like, double track and change things. But this theme is just fully codex-pilled and just, like, leaning in entirely. And they run into the exact problems that you're describing, which is, like, you know, their challenge is, you know,

Starting point is 00:13:31 I want to get this thing, this feature built, but I can't get the agent to do it. And so usually there's an escape hatch where, you know, But then you're like, all right, I'll roll up my sleeves and like figure it out. And then instead of using codex, I might use like tab complete and cursor and things like that. But this team, for the experiment, this team doesn't have that escape hatch. And so then the challenge, like, how do I get the agent to do this? And I actually think we're going to be publishing a blog post from some of our learnings here.

Starting point is 00:13:57 But a lot of fascinating, like, paradigms and best practices are falling out of this. One interesting thing that we've noticed, I don't know if this is what you kind of feel, but we definitely feel it here is a lot of the time when the coding agent is not doing what you want, it's usually a problem with context and just like information that you've given it. It's just either under-specified or there's just not enough information

Starting point is 00:14:20 around how to do something available to the agent, available to codex. And so when you have to solve it through that, the challenge is then to add documentation and actually work around this limitation and basically encode more tribal knowledge that's in your head somehow into the code base, either via code comment itself or code structure itself,

Starting point is 00:14:42 or via text files, like dot-md files, skills, any type of additional resources within the repository so that the model can better do its task. There's a whole bunch of other learnings from this group, which I think is fascinating to explore. But yeah, removing that escape hatch of no longer using the AI has allowed them to start piecing together, a lot of the problems that we'll have to solve

Starting point is 00:15:06 if we really want to lean into agents. Another issue people run into, you talked about how people are shipping PRs like crazy, a lot more PRs if they're working with AI. Obviously, code review is becoming a bigger challenge. Is there anything you've figured out of your team to help speed that up to make that scale and not just create this terrible job for people

Starting point is 00:15:25 where they're just sitting there reviewing PRs all day? Yeah, I mean, one thing is Codex reviews 100% of all of our PRs at this point. And so I actually think, so one really interesting thing, that's happened is the things that tend to, we tend to hand to the models immediately tend to be the things that annoy us are like the most boring parts of software engineering. It's also why it's more fun now because we get to do more, you know, more of the fun things. For me, speaking more for myself, I really hated code reviews. It was like one of the worst things for me. And then I

Starting point is 00:15:56 remember in my first job out of college, it was that Quora. I owned, I was working on the news feed. And so I owned the code for the news feed. And so I was a reviewer for News Feed. And it was just like the central piece of code that everyone would touch. And so I would just, every morning I'd log in and be like 20 to 30 code reviews. I just like, oh my goodness, I got to like, you know, get through all of these. I would procrastinate and then it grows to like 50. And so there's just like a lot of code reviews. Codex is really good at reviewing code. So actually one thing that we've noticed that 5-2 in particular has gotten extremely, strongly adept at is reviewing code and especially when you kind of steered in the right direction.

Starting point is 00:16:34 And so for code reviews, yeah, we create a lot of PR. but Codex reviews all of them. And it makes, you know, code reviews go from a, you know, I don't know, 10, 15 minute task to sometimes even just like a two to three minute task because you have a bunch of suggestions already already baked in. A lot of the times people will, especially for small PRs, like you actually don't even need people to review. We kind of trust Codex in this way. The original author kind of website Codex, it is, you know, the benefit of code reviews to have a second pair of eyes to make sure that you're not doing anything dumb. Codex is a pretty smart second pair of eyes at this point. That's something that we've heavily leaned into.

Starting point is 00:17:10 The general CI process and the post kind of push and deployment process has also been heavily automated via Codex internally at this point. If you talk to a lot of engineers, the thing that annoys the most is after you've written your beautiful code, like, how do you get it into production? You know, you've got to run through all these tests. You've got to like, you know, Lint errors. You have the code review. There's a lot of automated stuff you can do with Codex. And so we've actually built some tools internally that help automate that process, automate the Lint. If there's like a link to error, it's a very easy codex fix,

Starting point is 00:17:39 and then it could just patch it and then kind of restart the CI process. So all of that is we're trying to collapse into as little work for an engineer as possible, and the byproduct of which is they can now merge and push out a lot more peers. Codex writing the code, codex reviewing its own code. I'm curious if you are open to using other models to review your models work. Is that a path or is it just, it's good enough? We don't need anything else. So I will say there's definitely a circular thing here.

Starting point is 00:18:05 And like going back to Sorcerer's Apprentice, like you want to make sure you're not letting the brooms go crazy here. And so, you know, we're very thoughtful, I'd say, around which PRs kind of are completely just Codex reviewed. Most people still obviously take a look at their PRs. And so it's not like it's going to zero. It's more like going from, you know, 100% attention to like 30% attention, which just helps things push through.

Starting point is 00:18:30 In terms of like multiple models, so we obviously test a lot of models internally. and so we have a lot of those. We use external models less. We think it's important to kind of dog food our own models and kind of like get feedback there. But you can also, you know, there are a lot of like internal variants of models

Starting point is 00:18:47 that you can use to give you different perspectives here as well and we found that to work quite well. Okay, so just to make sure we get a barometer of today's world at OpenAI in terms of AI in code, just so I understand, and then I want to move on to different topic, 100% of code across opening,

Starting point is 00:19:04 AI is written by Codex at this point? Is that the way to frame it? I wouldn't make the statement that 100% of code running in production today is written by AI. And it's kind of hard to do attribution there. But almost every engineer heavily uses Codex in all of their tasks at this point. And so if I were to guess I'm just like the vast majority of code at this point, it was probably authored by AI. Incredible. Okay.

Starting point is 00:19:30 So there's a lot of talk and we've been talking about kind of the IC role, the work of an ICN. engineer. There's less talk about the changing role of a manager, especially an engineering manager. How is your life as a manager changed with the rise of AI and just what do you, where do you think managers, what's the role of a manager in the future? It's definitely changed less than an engineer. There's no, you know, codex for managers just, yes yet. However, I use codex quite a bit for some of the, some of the, like, kind of more managery task that I do. I'd say a couple things are changing. They're like some trends.

Starting point is 00:20:05 So I don't think it's changed that much yet, but I see trends. And I think if you play it out, you can kind of see where a lot of this is going. One thing that's becoming increasingly clear is Codex really empowers, like, top performers to get a lot. I like to be a lot more productive. And so it really like, and I think this may be true for AI more broadly, like across society, which is like the people who really lean in or like the people who have high agency or like, will really get good at these tools will kind of supercharged themselves. And so I'm kind of noticing this now as well, which is like the top performers kind of end up being a lot more, a lot more productive.

Starting point is 00:20:46 And so you see a broader spread in team productivity in this way. So one thing that I've always done as a management philosophy is to spend actually the majority of my time with top performers, just like make sure they're unblocked, make sure they're happy, make sure they feel productive and they feel hurt. I think this is even more true in an AI world where your top farmers are going to just like really be shooting ahead using these tools. I think one example is the team that's maintaining a 100% codex generated code base, like just letting them kind of rip and see what's happening there is something that's paid dividends. So I think that's kind of one trend that I'm seeing where spending even more time with top performers for managers I think is likely going to continue. The other thing is, so this is more an observation, but my sense is with a lot of these AI tools available to managers, so less like writing code, but just things like ChadGBT with organizational knowledge,

Starting point is 00:21:45 like being able to do research and understanding organizational context a lot better. Another good example is we're doing performance reviews right now, and it's actually really easy to use chat GPU with internal knowledge, hooked up to GitHub and Micronotion Docs and Google Docs. To get a really good sense of what this person has done over the last 12 months in writing a little, you know, deep research report for it. My sense is I think managers will be able to manage much larger teams in this world. Kind of like how, you know, like software engineers are managing 20 to 30 codexes. My sense of these tools will allow managers, people manage to be higher leverage and will allow them to, to manage, you know, teams of way more than the current best practice.

Starting point is 00:22:27 So I think it's like six to eight, right, for software engineers. you kind of see this applied to, you know, like the non-engineering domains, like support or operations where it's like, you know, previously, like the size of a support team might be limited, but like as you can pass off more things to agents, you can actually do more work and also manage more people this way. I think the same thing might happen for people management as well, especially in tech companies. And we're already seeing this. There's some teams where their EMs managing quite a few people. And they're doing it pretty adeptly because of some of these tools where they can get higher leverage and understand what their team's doing, understand organizational context a little bit better, and operate in that way.

Starting point is 00:23:09 I love this advice that the way you described is you've always leaned into top performers and spent more time with them and blocked them and make sure they're happy. The way Mark Endreson is just in the podcast, the way you phrased it is AI makes good people better and it makes great people exceptional. Yeah, yeah. And what you're saying here is just doing this more. and more is probably the right move, spending more time with the best people on your team to unblock them, make sure they have everything they need. Yeah, a very good example right now is there are, I would say, like a group of engineers internally who are really codex-filled and are thinking through what the best practices are for interacting with this model. And that is just an extremely high-leverage thing

Starting point is 00:23:47 for them to do. And so, just like, as a manager, I'm just like, yeah, go explore this, you know, whatever best practices come out of this, you know, we have to share with the org. Well, we'll, you know, we do all these knowledge sharing sessions, we'll share documents and best practices everywhere. So things like that, just elevate everyone. And I view that as like, you know, another example of this trend that we're seeing where the top farmers really get exceptional. People just like have a sense.

Starting point is 00:24:15 This is big. AI is changing so much. The world is changing. It's going to be a huge deal. What do you think people aren't pricing in yet into what will change into where things are heading just like what's an example of something you think are like, okay, we're not realizing this yet. So one of my favorite kind of like phrases or like things that have come out of this whole

Starting point is 00:24:36 AI wave is the idea of the one person billion dollar startup. I think I should think Sam may have keyed it or like Sam may have been the first one to say it, but it's fascinating to think about, right? It's like, yeah, if people are so high leverage, at some point there will likely be a one person billion dollar startup. And while I think that's really, really, really cool. cool. I think people aren't really pricing in the second or third order effects of this. And really what, you know, because what the one person billion dollar startup implies is that

Starting point is 00:25:04 there's, you know, one person can just have so much more agency and so much more leverage using one of these tools that it is just super easy for them to get everything done that they need to for their business to, you know, ultimately create something that's a billion dollars. But I think there are a couple other implications of this. So one of them is, if it's easy for a person to create a one person, or if it's possible for a person to create a one person billion dollar startup, it also means it's way easier for people to just create startups in general. Like I actually think

Starting point is 00:25:32 this will, like, once I can order of fact of this is I think there's just going to be a huge like startup boom and like small, like SMB style boom where anyone can build software for anything, right? Like one, you're kind of starting to see this play out in

Starting point is 00:25:48 the AI startup scene where software's became a lot more vertical oriented where like these verticals, like creating some AI tool for some vertical tends to work quite well because, you know, you really lean into that particular domain. You like really understand the use case for it. And so if you play out AI, there's no reason why you can't have like 100x more of these startups. And so I think I think one world that we might end up seeing happen is in order to enable a one person billion dollar startup, there might be like a hundred other small

Starting point is 00:26:21 startups building bespoke software that works extremely well to support other types of, you know, small, small one-person, you know, billion-dollar startups. And so I think we might actually enter into a golden age of like B2B SaaS and just like software and startups in general. And so I think, I think that's a really interesting trend to kind of see because as it's as it's really, as it gets easier and easier to build software, as it's easier and easier to, you know, run a company, you might actually just end up seeing way more of these these these startups and so the way i've i've been thinking about is like yeah there might be one uh one person billion dollar startup or there might be like a hundred you know a hundred million dollar startups there might be tens of thousands of 10 million dollar

Starting point is 00:27:05 startups and as an individual it's actually pretty great to have a 10 million dollar business like that's like enough for your stuff for life at that point and so you know we might really see seen an explosion in that way and i and i feel like people aren't aren't really you know pressing that in um there's another kind of like third order effect of this. And again, all of these, as you get to the further and further out, predictions, I think, there's a lot of uncertainty. I think if we end up moving to this world

Starting point is 00:27:31 where you end up with these, like, kind of micro companies building software that works for one or two people who own the company and are working there, I think the startup ecosystem will change. I think the VC ecosystem will change. You know, we might end up in a world where there's just like a handful of big players that are offering platforms

Starting point is 00:27:48 and supporting all of these startups. but, you know, the types of venture scale return startups that can really 100 or 1,000 X your investment might actually end up shrinking if you end up having a bunch of these, you know, smaller $10 to $50 million companies, which are not great for venture salary returns, but are great for the individuals, the high-agency individuals who are now, you know, really leaning to AI to build these businesses for themselves. I love how many order, like order effects we've been through. When I hear the fourth order effect now, sure.

Starting point is 00:28:18 I'm just joking. I can't, it's two, fourth order is two, two gigabrain for me. I can't, I can't think that far ahead. It's like inception

Starting point is 00:28:27 where just everything gets slower every time you go deeper into someone. Yeah, every layer. Okay, so the billion dollar startup, I've been, I think about this a lot

Starting point is 00:28:36 because I, I'm not going to be a billion dollar startup because what I'm doing is not venture skill in any way and not super high leverage, but just could see how many support tickets I get from just like the most ridiculous things.

Starting point is 00:28:48 It's hard. for me to imagine one person. Like, I'm bearish on this billion-dollar startup. I just want to share this thought. Simply because of the support costs, even if AI is helping you at a billion dollars, just like, unless your ACVs are, you know, very high and you have very few customers, it's just dealing with support. And people are like, you know, like, they can solve their own problems, but they're like,

Starting point is 00:29:10 I'll email support to ask about this thing. Just dealing with that is hard to scale, in my experience. So unless you have, in my opinion, unless you have a bunch of contractors, which I I don't know. Does that count as a single person company? I feel like it's very difficult to scale a billion dollar startup and not have someone helping you with at least the support work. And AI, I think will take you so far. So I think that's true. And actually, I think my view on it is is slightly different, which is I think that your, you know, Lenny's podcast might end up becoming a billion dollar startup. But what I think might happen is instead of you kind of being the one person who has to dispatch an

Starting point is 00:29:48 to solve and fix those support tickets, I think what might end up happening is there might be a whole smattering of other startups that are building software and super, and like super tailored towards what you might need. And so, you know, there might be like 10 or 20 startups that build support software for podcasts and newsletters. And that might be a one-person startup.

Starting point is 00:30:13 Like it doesn't need to be a big one. And it's, and, you know, they might be able to just code up this product very, very, very easily. They're able to kind of build their own thing. And because it's so tailored and unique and hopefully, you know, useful for you, it might be something that you purchase as the one-person billion dollar startup. I would buy that. I would buy that. Yeah, there's like a question of like what you in-house and what you like kind of outsource. And what I think might happen is because the cost of writing software and building products is, is collapsing so much.

Starting point is 00:30:39 You might end up outsourcing a lot of this. And in doing so, reducing the size of your company. And so that's kind of the world that I think might end up happening. Again, there's like high and or a need in what might play out here. But the end result still might be a one, like one person driving this like high, high massive leverage company that might actually reach a billion dollars. I could see that.

Starting point is 00:30:57 I also think about Peter at Claudebot slash mold bot slash open claw of just like how he barrage is right now by all these asks and emails and pings and DMs and PR is just like, I'm curious to, and he's not even making any money out of this thing. Yeah, I can't imagine what it's like to be him right now. It must be like absolutely insane.

Starting point is 00:31:14 It's probably like, you know, like the months after we launched chat GVT, the craziness that was... As one man. He's coming on the pot, by the way, in a week. Oh, that's exciting. Yeah.

Starting point is 00:31:26 Maybe the fourth order effect is distribution becomes increasingly important because there are so many freaking things trying to get your attention. So people with an audience and platform, I think, become more and more valuable, which is good stuff. Okay.

Starting point is 00:31:41 I wanted to come back, actually, to your management stuff. So I really loved your insight about spending more time with our performers has been really successful to you. Just thinking about you as a manager of a team that is building the platform that powers basically the entire AI economy, like every AI startup is building on your API. Clearly you're doing a great job. What other kind of core management lessons have you learned?

Starting point is 00:32:06 What do you find is really important and key to your success as a manager of engineers and just people? Yeah. I think a lot of the lessons that I've learned here, I don't know how specific it is to the opening API or some of our enterprise products in particular. I think my management philosophy is obviously changed over time, but I think it's probably stayed the same more than it's changed over time. One of these principles is kind of what I talked to you about before,

Starting point is 00:32:33 which is spending a lot of time with top performers, like actually spending, and like to be very concrete, like it's like more than 50% of your time with your top performers, with maybe your top like 10% performers. and really, really trying your best to empower them. The way that I think about it is kind of come back to this analogy of software engineer as a surgeon, which comes from the mythical manmunk book. So it's actually, it's funny.

Starting point is 00:33:00 So I pull it from the book, but in the book, they actually described this world where I think they were like predicting the future, because I think the book was written like in the 70s or something. They said that software engineering might end up moving into a world where the software and nears are like surgeons or like in a surgery room there's like one person doing the work um and uh you know there's the one person like cutting or whatever and like doing all the surgery and everyone else in the room is there to just support them right it's like the nurse and like the assistant and the resident and the fellow and then the surgeon's like i need a scalpel and they give them a scalpel and then uh

Starting point is 00:33:33 they're like i need you know this tool and it's machine and they'll bring it over everyone's there to just like you know support the one uh surgeon and so the the mythful mammoth actually predicted that that that is kind of the direction that software engineering is going to go. I don't think that's exactly played out where it's much more collaborative and like, it's not only one person doing the work. But I've always really liked that analogy. And that analogy is actually what I strive to kind of like emulate in my own management philosophy, which is software engineering isn't really like surgery where it's not just

Starting point is 00:34:03 one person doing work, but the way in which I like treating the people on my team and the way that I act as a manager is I want to empower them, make them feel like they're a surgeon. And insofar as like making sure that I'm supporting them and making sure they have everything that they need to do their work. And it feels like they have an army of people kind of supporting them and looking around corners and giving them everything that they need when it's really just me as the manager. And so like the example that I give is is looking around corners and unblocking people, especially from an organizational perspective, is extremely extremely useful. And again, going back to the AI conversation, it's even more important nowadays. Right. like if people are just like cranking PR after PR, the main thing bottlenecking progress and,

Starting point is 00:34:45 you know, shipping something tends to be organizational or like process oriented. And if you as a manager can kind of look around corners and kind of unblock the team, if you can, you know, like if the surgeon needs scalpel, but, you know, the manager kind of already has a scalpel ready for them, that's the best case scenario. That's kind of the way that I approach management and especially engineering management. And so that's something that's really, really stuck with me over time. And even though, you know, software engineers aren't exactly surgeons, that metaphor is always kind of stayed in my mind as of the rest of my career.

Starting point is 00:35:18 I love that. And I feel like I wonder if that's something AI can help with is look around corners and predict here this engineer is going to be blocked by this decision. We need to figure this out. We need to get the long. Yeah, that's actually a really good point. I haven't tried this yet. But I wonder what would happen if I ask Chad GPT hooked up to company knowledge,

Starting point is 00:35:34 you know, like, what are the active blockers? look through all the Notion Docs, maybe Slack messages. You know, it's probably in Slack somewhere. What are the active blockers on my team? And is there something I can do to help? Now, that's very interesting. I have not thought about that, but you're right.

Starting point is 00:35:48 We just had an insight right here. Yeah, yeah, yeah. And it's, I think even more interestingly, what do you anticipate will be a blocker for this engineer or this team in the coming months? Yeah, you ask the model, you asked the AI to do the second and third order. There we go. Anticipate that, man, anticipate what the bloggers will be next month, too. Great.

Starting point is 00:36:05 I think we've got a good idea. idea right here. Yeah, yeah. This episode is brought to you by Data Dog, now home to Epo, the leading experimentation and feature flagging platform. Product managers at the world's best companies use Data Dog. The same platform their engineers rely on every day to connect product insights to product issues like bugs, UX friction, and business impact. It starts with product analytics, where PMs can watch replays, review funnels, dive into retention, and explore their growth metrics. Where other tools stop, Data Dog goes even further.

Starting point is 00:36:37 It helps you actually diagnose the impact of funnel drop-offs and bugs and UX friction. Once you know where to focus, experiments proved what works. I saw this firsthand when I was at Airbnb, where our experimentation platform was critical for analyzing what worked and where things went wrong. And the same team that built experimentation at Airbnb built Epo. Data Dog then lets you go beyond the numbers with session replay. Watch exactly how users interact with heat maps and scroll maps to truly understand their behavior.

Starting point is 00:37:06 And all of this is powered by feature flags that are tied to real-time data so that you can roll out safely, target precisely, and learn continuously. Datadog is more than engineering metrics. It's where great product teams learn faster, fix smarter, and ship with confidence. Request a demo at datadoghq.cue.com slash lenny.

Starting point is 00:37:26 That's datadoghq.cue.com slash Lenny. Okay, I'm going to shift to talking about the API and the platform that you all build. So you work with a lot of companies implementing your API, your platform building on your tools. You told me that you find that a lot of companies actually have negative ROI on their AI deployments, which I think is what a lot of people you read about and feel and think. And it's interesting you're actually seeing that. What's going on there?

Starting point is 00:37:55 What are they doing wrong? What's happening in the world of AI and deployments in RY? Yeah. So to be clear, I don't like explicitly see. quantitative numbers around this. You know, it's actually really hard to measure these things. But especially from observing some companies kind of trying to do AI, I would not be surprised if a lot of AI deployments are actually, you know, negative ROI.

Starting point is 00:38:18 I mean, part of this too is I think there's also general sentiment from folks around the country, like basically outside of tech that AI is being forced onto them. And I think part of this is probably a symptom of some. some negative ROI AI deployments. A couple of things I've observed around this. So one thing is, and I think I come back to this again and again, like I think we in Silicon Valley just forget that we live in a bubble. Like we are so, like Twitter is a bubble, sorry, X is a bubble.

Starting point is 00:38:49 Silicon Valley is a bubble. Software engineering is a bubble. Most people in the world, most people in the U.S. are not software engineers, are not very AI killed, are not following every single model release. And so we're just like highly, out of the loop on how to use this technology. And so, you know, like, we always talk about all these, like, best practices for codex, all these, like, codex-filled people within OpenAI.

Starting point is 00:39:11 I'm sure everyone on X who posts are, like, crazy power users of these AI tools. You know, they lean into skills. They lean into agents.md. MCPs. Yes, yeah, all of that. And when I talk to some of these companies and I talk to the actual employees using these, it's like the most basic thing that they're trying to do. and they, like, have very little understanding of exactly how the technology works.

Starting point is 00:39:37 And so that's kind of, like, one big observation for me, which is, like, they're asking very simple questions of these things. They're really not pushing it just yet. And so that kind of goes back to, that kind of ties into what I think more companies do or, like, what should do or what a more ideal AI deployment setup looks like. And this is kind of how we've run things with an open AI too. the companies where I think it started to work really well have a combination of both top-down buy-in. So it's like the C-suite, it's like, you know, we want to become an AI-AI first company. So there's buy-in, they buy the tools, they have, you know, exec support. But it also has bottoms up adoption and buy-in.

Starting point is 00:40:18 And so what I mean by that is it has, like, actual employees doing the work who are really excited about this technology and are willing to learn, evangelize, build best practices, and kind of like knowledge share within the organization. we've seen this a lot internally. So, like, obviously Open AI has always wanted to be a very AI-centric company, but when it really started taking off was with the introduction of Codex and these tools where, like, people, like actual employees themselves could start applying it to their work. And I think you really need this because at the end of the day, everyone's work is, like, very different.

Starting point is 00:40:52 It's, like, very unique. Software engineering is different than finance is different than operations, different than go-to-market and sales. And so there's, like, a lot of these, like, last-mile intricacies. of work that needs to really be done in a bottoms up fashion. And so my sense is a lot of these air deployments don't have like don't have bottoms up adoption. Like it was like an exact mandate and it's extremely top down and it's very divorced from what

Starting point is 00:41:18 the actual work looks like. And as an result, you end up with a giant workforce that doesn't really understand the technology is like, I know I'm supposed to use this and maybe it's like on my performance review too, but I'm not sure what to do. And they look around. No one else is doing it. there's no one else to learn from. And so my, you know, my recommendation for companies kind of pushing this is,

Starting point is 00:41:36 is find, or maybe even staff a full-time team internally that is this kind of Tiger team internally that can explore the full extent of the capabilities, apply to specific workflows, do the knowledge sharing, create excitement within folks who might want to use this technology. Because in the absence of that, it's very difficult to, it's actually very difficult to pick up. And who would you put on this Tiger team? Is it like engineer-led, do you find in your experience? is a cross-functional sort of team?

Starting point is 00:42:03 Yeah, it's interesting. Also, a lot of companies don't have software engineers. And so the pattern I've seen is it tends to be these like software engineering adjacent, like basically technical people but are not software engineers. I think those are the ones who tend to get most excited around this. It's like, you know, maybe the, it's like maybe the like, you know, support team operations lead who doesn't code but loves you. using these tools and, you know, is like an Excel wizard or something. And so it's like

Starting point is 00:42:34 technical adjacent or like coding adjacent and like, you know, pretty technical. Those are the times, like, those are the kinds of people I've seen in these companies who just like really light up and get excited around this. And you can usually build a team, a team around that. But yeah, it's like oftentimes not software engineers. Software engineers, I think will understand this, but not every company has software engineers is actually kind of a rarity. They're hard to find. They're expensive. And so it's these other types of folks. What I'm here is the anti-patternattern is top down, this is very, the CEO found an exact team just like, we are going to go AI first, we're going to lead into AI. Everyone's going to be judged on their performance using AI

Starting point is 00:43:09 tools, how much your productivity is increasing things to AI. And without, with that being just top down and not creating a team that is bottom up, spreading the gospel, you find that doesn't work. Yeah, yeah, exactly. And the advice is find the people that are most excited. And instead of kind of having spread out through the organization. What you find works is create a little AI kind of evangelist team that finds ways to use it and kind of spreads it across the work. Yeah, I mean, another, it's kind of like hearing you play back to me, another way to think about it, kind of tying back to my own management philosophy is find the high performers in AI adoption and empower them. You know, let them build hackathons, let them, you know, hold seminars,

Starting point is 00:43:52 do knowledge sharing, kind of create the seeds of excitement internally. Okay, amazing. There's a couple hot take. So when a, here from you, something that I've seen you talk about and share. One is you've shared that talking to customers and listening to customers is not always the right strategy in AI and it might often lead you astray. I don't know if it's that hot of the take. I think the main thing here is so obviously you should talk to your customers. It's like you still talk to customers. I just think the AI field, especially what I've seen over the last kind of like three years, working on the API and seeing kind of all that evolves,

Starting point is 00:44:31 is the field and the models themselves are just changing so, so quickly. They tend to, like, disrupt themselves, especially around the, like, tooling and the scaffolding space. So there's this quote that I read, actually, earlier this week from an X article, by this guy named Nicholas,

Starting point is 00:44:48 who's the founder of a startup called FinTool, where I think he was sharing a lot of the best practices that he has learned through building, AI agents for financial services, I think I had to start FinTool. And I had this phrase that I thought was really good, which is the models will eat your scaffolding for breakfast. Like if you look, if you rewind back to 2022, right when Chad GPT launched, these models are pretty raw. And there was like all this product scaffolding and things, especially in the developer space, to basically try and steer the model and build a scaffolding around it to get it to do what you want. Like agent

Starting point is 00:45:23 frameworks. There's like like vector stores, I think was like, really popular back then. And just like a whole smattering of tools here. And as you've kind of seen the feel play out, that the models have just changed so much that and gotten so much better, that they ended up, yeah, literally eating some of the scaffolding. And I think this is even true today.

Starting point is 00:45:42 So I think the article from Nicholas actually, you know, the current scaffolding, which is fashionable, is skills, files-based context management. I could see a world where at some point, you know, that's no longer useful, where the model can actually, you know, manage all that themselves or like, you know, or there might be, you know, it's hard to predict, but like might move on to some new paradigm where you know all irony is file-based skills type thing.

Starting point is 00:46:05 You have literally seen this play out, right? Like the agent framers, I think, are a little less useful now. There's a period of time like 2023 where we thought vector stores and is going to be like the main way for you to, you know, bring organizational context into the models. And you need to, you know, vectorize and embed every bit of your question. corpus is, and they need to do all this work to, like, figure out the vector search, to optimize that, to fill out the right information, the right time. All of that is scaffolding, because the model, you know, was not good enough. And it turns out, you know, in this case,

Starting point is 00:46:35 it turns out, as the models get better, a better approach is actually to take out a lot of that logic and trust the model and give it a set of tools for search. It doesn't need to be a vector store. You could actually just hook it up to any type of search. It could literally be files on a file system like skills and agents MD to kind of steer it as well. Obviously, there's still a place for vector stores. I know a lot of companies are still using it, but the entire scaffolding around that and building an entire ecosystem around that and assuming that's the only scaffolding that you need has really changed. And so tying this back to the like, you know, you don't always have to listen to your customers. Because the field is changing so much at any point in time, you know,

Starting point is 00:47:13 a lot of people are kind of in this local maximum. And if you just blindly listen to your customers, they'll be like, yeah, I want a better vector store. Like, I want a better, you know, agent framework for this. And if you had just kind of only chased down that path, it actually would have led you to, you know, build something that, again, is the local maxima. Whereas as the models get better, we've had to reinvent and kind of rethink the right, right abstractions

Starting point is 00:47:38 and the right tools and frameworks to build around these models. And the cool slash exciting slash kind of crazy, annoying part is it's a moving target. And so, yeah, like the current, current smattering of tools and frameworks right now will likely need to evolve and change pretty significantly over time as the models get smarter and better. But that is just the nature of building in the space. I think that's what makes it exciting.

Starting point is 00:48:01 But it also means when you talk to customers, you kind of need to balance the exact feedback that they want with where you think the models are going and where you think things will trend over the next one or two years. It's interesting how this is the bitter lesson is, you know, this big lesson that AI and ML folks learned, which is just like, the less you overcomplicate,

Starting point is 00:48:20 the less logic you add to machine learning to AI, the more it'll be able to scale and grow and just take it all the way and let it just compute basically, just give it more power to get smarter on its own. There's literally a version of the bitter lesson applied to building with AI where we were trying to architect all this stuff around

Starting point is 00:48:39 and turns out the models will just kind of heat it all away. And honestly, like, OpenAI API team has been guilty of this, where we kind of, like took some left and right turns when we shouldn't have. But yeah, the models still end up. Models get better. And we're all learning the bitter lesson day in and day out.

Starting point is 00:48:57 So what would be the key takeaway for folks building on, say, the API or just building agents and having to build a little bit of surround for now? Is it just, yeah, what would be the advice? My general advice, and I've been giving this to people for a while and I think it's still true today is make sure you're building for where the models are going and not where they are today. you know, the, it's clearly moving target. And I think a lot of the companies that I've seen,

Starting point is 00:49:22 startups that I've seen really, really do well is they build a product for an ideal type of capability that is like maybe 80% of the way there today. And it like, they end up, you know, having a product that kind of works, but it's like just almost there. But then as the models get better, you know, suddenly it might click. And then their product now is incredible because it works. you know, like, maybe with like, oh, 3 at some point, it suddenly works with 5.1, 5.2, suddenly it unlocks it.

Starting point is 00:49:51 But they're building these products with the, like, the model capability improvements in mind. And with that, you end up creating an experience. That's way better than if you had assumed that it's static in the first place. And so that would be my general advice, which is, you know, build for where the models are going and not where they are today. You end up building a better product. You may need a, you know, like, wait a little bit,

Starting point is 00:50:12 but like, you know, the models are getting so much better so quickly, you often don't need a weight that long. So to follow that thread, where are, like in the next six to 12 months, where is the API heading, where's the platform heading, where are the models heading as much as you can share? I know there's a lot of secrets here that maybe you're more excited about, or do you think that people should start to prepare for it, however much you can share? I mean, so the obvious one is how long of a task these models can do coherently. So there's like the meter benchmark that, that I think track software engineering tasks and how long, you know, like how long of a task can these models do 50% of the time, 80% of the time. I think we're at something like multi-hour tasks being able to be done by software engineering tasks being able to be done by these frontier

Starting point is 00:51:00 models 50% of the time. And then I think 80% is something like just under an hour. But the sobering thing about that chart is they plot all the previous models on this chart as well. So you can really see the trend of this. That's something that I'm really excited about, which is, you know, I actually think products today really optimized for tasks that the model can do for, like, minutes at a time. Like, even codex and, like, the coding tools, I'd say, like, you know, it's in the cli. You're kind of like seeing it be interactive. It's really, you know, quite optimized well for, like, maybe at most 10-minute type tasks. I have seen people push codex to the limit into, like, multi-hour-long tasks. But again, I think that's more of the

Starting point is 00:51:41 exception. But if you follow this trend, like I think like in the next 12, 18 months, we could see models that could do multi-hour long tasks very, very coherently. At some point, it might reach like, you know, six hours a day-long task where you kind of like dispatch it and have it do, you know, do things on the zone for a while. The types of products you build around that will look very different. You want to give the model feedback. You obviously don't want it to completely run wild for a day. Maybe you do, but you probably don't. And then the universe of things you have the model do really expand. So that's something that I'm really, really excited about seeing. Another thing over the next 12, 18 months where I think it'd be really cool is improvements in the

Starting point is 00:52:20 multimodal models. So, and actually by multimodality, I'm mostly thinking about audio here where the models are pretty good at audio. I think they're going to get a lot better at audio over the next six to 12 months, especially the like, you know, the native multimodal model, the speech to speech ones. I think there's also interesting work being done around new types of models and architectures on the multimodal audio side as well. But audio, especially in the enterprise and in a business setting, I think, is a hugely underrated domain still. Like everyone talks about coding. It's all text. But we're talking in audio.

Starting point is 00:52:59 A lot of the world's business is done via audio. A lot of services and operations are done via talking and audio. And so I think that area is going to look very exciting in the next 12 of 18 months, and I think there will be even more unlock for what we can do with audio models there as well. Amazing. So quick summary, expect agents and AI tools to run longer to that trajectory to continue to increase. And then audio and speech becoming a bigger deal, more first party and native and better and core to the experience. Yeah.

Starting point is 00:53:35 Extremely cool. Okay, I want to go back to one of your hot takes, another hot take that I've seen you discuss. You're very bullish on business process automation as an opportunity in the world of AI. Talk about that. Yeah, this goes back to the thing that I said previously, which is we live in a bubble in Silicon Valley. And a lot of the work that we do that we're used to, software engineering, you know, product management, building products is very differently shaped than the work that goes on that runs our entire economy.

Starting point is 00:54:07 And I see the same in and now when I talk to customers. If you talk to any company that's not based in, it's not a tech company, there's a lot of business processes. And so what I mean by this is, I generally delineate it as, you know, there's like software engineering is kind of like open-ended knowledge work.

Starting point is 00:54:26 And this is why I think tools like Codex tend to be quite good because it's exploring and you're giving it these like open-ended things. But software engineering is fundamentally like pretty open-ended and is not very repeatable, right? So like you build a feature. You're not trying to build the exact same feature over and over again. And a lot of like tech jobs are in the space. I think like data science is kind of in the space as well, even some of the like strategic finance stuff. But as you move further and further away from software engineering and like what is core in tech,

Starting point is 00:54:56 a lot of jobs are just business processes. They're like repeatable things, repeatable operations that some manager at a company has kind of like iterated on. There's usually a standard operating procedure that people want to do and you don't want to deviate from it that much. You know, in software engineering, the ingenuity isn't deviating.

Starting point is 00:55:18 But a lot of the work being done in the world is actually just running through these procedures and operations. Like if I call a support line, they're running through one of these. If I call my utility company, there's a bunch of processes and things that they can and cannot do for me.

Starting point is 00:55:35 And so I'm just extremely bullish on this general category of like, and I think it's underrated because it's so different from what we think about in Silicon Valley, people tend to not think about it. But how can we apply AI and some of the tools and frameworks that we have towards this business process automation, towards automated and making easier repeatable business processes with high determinism that is fully integrated with business data and business decisions and different systems within an enterprise,

Starting point is 00:56:07 and how can we actually make that process better? Because I actually think there's a lot of opportunity and a lot of work to be done in that area. And we just don't talk about it because it's a little bit less in our wheelhouse. So your take here just to make sure I fully understand it is you think there's a much bigger opportunity outside of engineering for AI to impact productivity,

Starting point is 00:56:29 companies and also jobs of these folks that are doing these kind of repetitive, easily automated tasks. Impact jobs and also just impact how work is done. So much of work is done in this way. Like you think about, you know, like what a, like basically we, I talk to customers all the time, big enterprise. It's like, like, how will AI transfer my company? Like, how will it run in a world with AI in like 20 years? And, and, you know, software engineering is part of the story, but there's so much more on the business process side. And I actually think it might look even more different on the business process side and and the work there is pretty substantial. It's actually interesting. I don't know like from an absolute percentage or absolute base,

Starting point is 00:57:07 I don't know if it's bigger or smaller than software engineering. Like software is pretty huge and pretty expensive as well, but it is pretty massive and it's definitely bigger than, you know, it's bigger than you would think it is based off of how how people talk about it or don't talk about it on X or Twitter. Okay. Going in a slightly different direction, having built the platform building the API, people building on the API, the biggest question on people's minds is always just, how do I not have Open AI squashed my idea

Starting point is 00:57:37 and build their own thing and then, you know, destroy this market I created? What's the general policy? What's the general philosophy of how startups should think about where open AI is unlikely to go? My general answer here is, the market is so big and so massive. I actually think

Starting point is 00:57:57 startups should just not overly think about where OpenAI or these labs are going. I've talked to a lot of startups that have not worked out, startups that are doing really well. Every startup that I've seen that is kind of fizzled out

Starting point is 00:58:11 is not because Open AI or a big lab or Google or something has come to swatch them. It's because they built something and it really didn't resonate with the customers. Whereas the ones that take off, even in very competitive spaces like coding,

Starting point is 00:58:23 like cursor is huge at this point. And it's because they built something that people really love. And so my general advice is like, don't, you know, don't overly stress about this. Just build something that people like and you will have a space in this.

Starting point is 00:58:34 I can't overstate how big of an opportunity there is right now. Like, the opportunity space of building with AI so big. A good example of this is, like, the space is so big that the Overton window of what is acceptable and not acceptable for VCs to do has completely changed here. VCs are like investing in like competitive companies left and right. It's just like the space is so big

Starting point is 00:58:55 because the opportunity is unlike anything that we've seen before. And while that affects how VCs operate, from a startup perspective, it's like the most empowering thing in the world because even if you just build something that some people really, really love, you will end up with a massive, massively valuable business. And so that's why I tell me, like, don't know what you think about it. The other thing, like I also think is important to remember,

Starting point is 00:59:19 at least from an open-AI perspective, one thing that we've always held very near a dear, which both Sam and Greg help reinforce from the top as well, is we actually view ourselves fundamentally as a ecosystem platform company. The API was our first product. We think it's really important for us to foster this ecosystem and continue to support it and not squash it. And so if you kind of look at the decisions we make,

Starting point is 00:59:42 this is all we've through it. Every single model we've released in one of our products gets released in the API. Like even, you know, we release these codex models now that are a little bit more optimized for the codex harness. but they always find their way into the API and all of our customers end up using those. We don't hold back on any of that. We think it's really important to keep our platform neutral

Starting point is 01:00:02 and so we don't block competitors. We allow people to have access to our models. We also want, you know, like we've recently been testing more of like the sign-in with Chad GPT product as well. And so we want to foster this ecosystem. I think it's really important that we do so. The general like thinking about this is like, you know, rising tide lifts all boats. And, you know, we might be an aircraft carrier. We're like pretty

Starting point is 01:00:26 big at this point. But we think it's important to raise the tide because everyone kind of benefits. And I think we'll benefit as well. Like our API itself has grown pretty significantly because we, we act in this way. And so I'd really encourage people not to view open AI as this kind of like, you know, thing that'll just shove people out of the way. But instead, focus on building something valuable. And we, you know, remain committed to providing an open ecosystem. term. Why is that important to Open AI just this focus on building a platform, creating a way for people to build businesses? Just like, is that just that's been the vision from the beginning? We want this to be a platform. It's been the vision from the beginning. It comes, goes back to our charter actually, like our mission. So the open eyes mission has always been to want to build AGI. So we know where I was seeing that. But then the second thing is to like spread the benefits of it to all of humanity. And there's kind of like a lot of, you know, The main part there is all of humanity. And obviously, Chad GeBD is trying to do this.

Starting point is 01:01:24 We're trying to reach however many, the whole world. But very early on, and this is why we launched the API back in, I think it's like 2020 or something like really early, we don't think we as a company will be able to reach all of humanity, right? Like there's, I don't know, every corner of the world is like pretty, pretty deep. And so we actually feel like in order for us to fulfill our mission, we need to have some platform-style thinking here where we can empower other people to build, you know, the customer support bought for podcasters and newsletter hosts, because we're not going to be able to do it ourselves.

Starting point is 01:01:57 And so we've largely seen this play out with the API. This is why we talk to so many of our customers and really love seeing the diversity of things built on. But yeah, it's been there to say one because we view it as an expression of our mission. And you haven't even mentioned the app store that you guys are launching the ChatjipT App Store. Yeah.

Starting point is 01:02:17 Is that under your umbrella, by the way, or is that a different Oregon team? It's a different team. So it's under chat. We obviously collaborate very closely with them. And, you know, they built like an apps SDK, which is a built in close collaboration with our team. But that is more within the chatGBT umbrella. But that is also another, like, that's another example of this, right?

Starting point is 01:02:34 It's like, Chad ChiBT BT is like, we kind of like have these 800 million weekly active users who are just coming over and over again. Like, it's a great asset to have as a business. But like, man, would it be better if we could somehow allow. other companies to come in and and take advantage of this as well and build for this audience as well. And then ultimately we think it will help us expand that that group as well, right? And so it all kind of comes back to the mission and we find that being a platform being open tends to help here. Just that number, 800 million, I think it's MAs. Just like almost weekly weekly. Weekly active. Yeah, it's crazy. Billion people using weekly. It's absurd happening

Starting point is 01:03:17 out how these numbers were just used to now, but that's insane, unprecedented. Yeah, it's mind-boggling for me to think about from a scale perspective, honestly. And the way I think about it is like 10% of the world and growing, by the way, like it's just, it's shooting up, come to chat GPT and use it every day, or sorry, every week. In this point, I just want to double down on this point. You're making OpenAI's mission was to make AI available to all humanity. and I think some people just that, they're like, oh, you know, it costs money.

Starting point is 01:03:48 And it's like, like the fact that it's, there's a free version of chat GPT that anybody can use that is not so different from the most powerful AI model that exists in the world for free. That's not gated that anyone can use. Like if you have, if you're a billionaire, there's only so much more you can get out of AI than what someone, you know, in a village in Africa can can get. And I know that's always been really important to open AI. Yeah, yeah. I mean, look, uh, that's, uh, that's,

Starting point is 01:04:14 That's why I think we've leaned into the health work. We've leaned into like education is going to be a very interesting here. The other insane kind of trend here is the free model has gotten so smart over time. Like the free model back in 2022 was, you know, like, well, it's good at the time, but it's like nothing compared to what you get today because you get you 55 today. And so the like, you know, raising the floor across the world is kind of, you know, something that we're really trying to do. And we view it as part of our mission.

Starting point is 01:04:42 The other flip side of this, by the way, is like, you know, kind of, talking about like the billionaires or whatever. I know people love saying like you're using the same iPhone that like, you know, Steve or sorry, like Mark Zuckerberg's probably using or like the billionaires are using. Like for like $20 a month, you're basically using, you know, like using the same AI that, you know, the billionaires are using. For like $200 a month, you get the same pro model that, you know, all the billionaires are using. But they're probably not using pro for everything.

Starting point is 01:05:07 They're probably just using the plus tier ones for their day in and day out. And so, yeah, this kind of like democratization and just like spreading of this. this benefit across all of the world is something that's really meaningful to us and something that drives a lot of what we do. One last question. Just for folks that are thinking about building on the API are just like, oh, wait, I could do cool stuff with open-A-is models and APIs.

Starting point is 01:05:29 What does your API and platform allow people to do? Like, I know you can build agents on top of the platform, just talk about what you allow. So fundamentally, the API offers a bunch of developer endpoints. And these developer emberts basically let you sample from our models. The most popular one that we have right now is one called Responses API. And so this is an endpoint. And it's optimized for building long-running

Starting point is 01:05:53 agents, so agents that'll work for a while. So at a very, you know, at a very, you know, low level, you're basically just giving the model text. The model will work for a while. You can kind of, you know, pull it to see what it'll do. And then you'll get the model response back at at some point. That's like the lowest level primitive. That's, like, the lowest level primitive. we have for people. And that's actually what a lot of people use. That's the most popular way of building on top of API. With that, it is like super unimpinionated.

Starting point is 01:06:21 And you can do basically whatever you want, it's like the lowest level thing. We've also started building more and more kind of like layers of abstraction on top to help people build some of these. And so next layer up, we have this thing called the agents SDK, which has also gotten extremely, extremely popular. This allows you to use the response to API or some other API endpoints that we have to build what you

Starting point is 01:06:42 more traditionally think of as an agent, like an AI kind of working in an infinite loop. It might have sub-agent agents that it delegates to. It starts building all this framework, all the scaffolding, actually. We'll see where this all goes. But it makes it a lot easier for you to build these kind of agents, giving it guardrails, allowing it to like farm out sub-tasks to other agents and kind of like orchestrate a swarm of agents. The agents' SECK kind of allows you to do that. And then above that, we've now started building tools to help also with kind of like the meta level of deploying an agent. So we have this product called agent kits and widgets, which are basically a bunch of UI components that you can use to very easily build a very beautiful UI on top of either our API or agents SDK. Because a lot of the times these agents kind of look very similar from a UI perspective.

Starting point is 01:07:36 And so there's agent kit. We also have a smattering of like eVALs products, like EVALs API, where if you want to test and like, you know, see if your models or your agent or your workflows working, you can test it in a very quantitative way using our Eval's product. And so, yeah, I view it as like these various layers. They're all kind of helping you build what you want with our AI, with our models and with increasing levels of abstraction and, you know,

Starting point is 01:08:03 how opinionated it is. And so you can start, you can do, you can use a whole. stack and it very quickly allows you to build an agent, or you can go down the stack as low as you want to basically the responses API and build whatever you want because of how low upload is. Sherwin, is there anything else that you want to share anything else you want to leave listeners with anything we haven't touched on that you think might be helpful before we get to a very exciting lightning round? The only thing I'd leave folks with is, yeah, I think the next two to three years are

Starting point is 01:08:32 going to be some of the most fun in tech and in the startup world. that that we'll have in a very long time. And I would just encourage people to not, not take it for granted. Like I, I entered the workforce in 2014. It was great for like a couple of years. I felt like there was like a period of like five to six years where it wasn't very exciting in tech. And then in the last three years, it's just been the most insanely exciting, energizing period of my career. And I think the next two to three is going to be continuation of that. And so I would encourage people not take it for granted. I'm trying to not take it for granted. At some point, you know, this wave's going to play out.

Starting point is 01:09:06 out and it's going to be a lot more, you know, incremental. But in the meantime, we're going to get to explore a lot of really cool things and invent a lot of new things and change the world and change how we work. And so that's the main thing I'd leave folks with. I love this message. I want to spend a little more time on it. When you say don't miss it, is it, what do you recommend people do? Is it just build, lean in, learn, join a company building really interesting things? Like, what's your advice to folks that are like, okay, I don't want to miss the boat?

Starting point is 01:09:32 Yeah, I would just say engage with it. So it's basically like what you said, lean in building. tools on top of this is part of the story. Just using the tools. You don't need to be a software engineer to lean into this. I think a lot of jobs are going to change here. So just using the tools, understanding the limitations of what it can and cannot do so that you can kind of watch the trend of what it can start to do as the models improve.

Starting point is 01:09:58 And yeah, and so it's basically like getting used to this technology and getting familiar with it instead of kind of like laying back and letting it pass you. On the flip side of that, there's a lot of, I think, stress and just anxiety around, like, there's so much happening. How do I keep up? I got to learn out a cloud bot this week. Oh, God. Yeah. Is there something you learned about it just not, like, you're at the center of this? How do you not get overly stressed and worried about missing things that are going on and just keep you stay on top of news? What are some things you've done learned? Yeah, so I think I'm personally a bad example of this because I'm basically chronically online on X and our company Slack. So I actually try and absorb. I end up absorbing a lot of it.

Starting point is 01:10:40 What I will say, though, is just like from observing other folks who are less, you know, addicted to this stuff like I am. Yeah, a lot of it is noise. Like, you don't need to have like 110% of this kind of pass your mind, like, like going to your mind. Honestly, just leaning into like one or two different tools, starting small is already like, you know, more than you need here. I think just the combination of like the frenetic pace of the industry.

Starting point is 01:11:06 X as a product just creates like this insane kind of like, yeah, this insane like pace of of news, which is honestly very overwhelming. The main thing is like you don't need to be, you don't need to know all of that to really engage with what's happening right now. And even something in simple is just like install the Codex client, play around with it. Install chat GPU machine connected to a couple of your, you know, internal data sources, Notion, Slack, GitHub, and see what it can and cannot do. All of that, I think, is a part of it. Amazing. Sherwin, with that, we reached our very exciting lightning round. I've got five questions

Starting point is 01:11:43 for you. Are you ready? Yeah, yeah, absolutely. First question, what are two or three books that you find yourself recommending most other people? I'll talk about one-on-one-one fiction book. The fiction book was, I just finished reading it. I really, I've really recommended it. It's, there is no anti-mimetics division by QNTM. It's a, I think he's like an online author, but I saw it being shared on X. This, it's like a science fictiony kind of book. And it was, I basically devoured it in like two days. It was, it's super, super well written, super fascinating.

Starting point is 01:12:17 It's about a government agency that's fighting, you know, things that make you forget it. And so it's just a very like smart, like creative book that, that and fresh, honestly, in terms of like source material that that I really like. So I'd recommend that one. The book is also unintentionally hilarious. So like it's like meant to be like this like sci-fi, almost like horror-style book, but it was it made me laugh a couple times. So that's the fiction book.

Starting point is 01:12:43 Non-fiction, I'm going to cheat and I'm going to recommend two of them. So in the last year I've been reading a lot more about China and kind of like the US-China relations. And I think there are two books that came on in the last year that have been really, really eye-opening for me in that regard. first one is the Dan Wing book Breakneck. That one was really, really good. I really liked his analogy of, like, the lawyerly.

Starting point is 01:13:01 US is the lawyerly society. China is the engineering society, and they're pros and cons to each. I read it, and I was like, hmm, yeah, it does seem like we're run by lawyers in the U.S. So then that's one. And the other one is the Patrick McGee book on Apple and China.

Starting point is 01:13:17 It was super, super interesting. I'm a huge Apple fanboy. Like, if you could see my desk right now, it's all Apple stuff. But just like, one, was just super fascinating learning about Apple's relationship to China. And then two, it just like had a lot of inside information about Apple as a company that I found fascinating. So it was also quite a page turner and also, you know, very, very timely, a timely book as well. The anti-mimetics book sounds amazing. I'm

Starting point is 01:13:41 buying it right now as you're talking. Yeah. Yeah, yeah. It's like, I think it's only like a couple hundred pages. I literally finished into things. It was just like so, so good. Okay, great tip. Okay. Favorite recent movie or TV show you have really enjoyed? Yeah, that one's tough because, you know, with, I've two kids and a busy job, and so I really haven't had much time to watch TV shows. I will say in the last couple weeks, I watched a couple episodes.

Starting point is 01:14:07 I'm actually a big anime guy, and so I watched a couple episodes. There's a new season of this anime called Jiu-Jutsu Kaysen. That's out. So season three of JJK was really good. In general, I'm a huge fan of Japanese anime. I think they create the most novel and unique plots in universes that Western media has shied away from. And so generally a big fan of that.

Starting point is 01:14:36 But yeah, I haven't really watched much, but saw a couple upsets with JJK recently. Extremely understandable in your role. Yeah. Favorite product you recently discovered that you really love. Yeah. Okay. So I recently had to set up Wi-Fi and like home networking. And I went all in on ubiquity routers and security cameras.

Starting point is 01:14:52 and security cameras. I'd never heard of it before I had to do this. I always just had a very simple setup. And it's just such a well-built product. I don't know if you used it before, but it's basically like the Apple of like home networking. So beautiful products. But the thing that actually makes it extremely good

Starting point is 01:15:11 is that software is good. And so they have a really great mobile app to help manage, you know, all of the home networking. And so basically ubiquity, you can use it to buy wireless, routers, you need Ethernet wiring throughout your house to use it. But I actually think what makes it really

Starting point is 01:15:28 good are security cameras. So if you have security cameras that are plugged at the ubiquity ecosystem, they have an incredible mobile app, and Apple TV app, an iPad app, to kind of see the live feed of your cameras. And so they're a little pricey, but not that pricey, but it's been

Starting point is 01:15:44 just an incredible product experience. All right, I went ERO, so I made a mistake. Eros are pretty good, too, but it's not ubiquity. Fully converted to ubiquity. Okay, good tip. Okay, two more questions. Do you have a favorite life motto that you find yourself coming back to in work or in life?

Starting point is 01:15:59 Yeah, the one that I always, you know, repeat to myself is never feel sorry for yourself. There's a lot of things that are going to happen, you know, at work in life and reminding yourself to never feel sorry and that you always have a sense of agency to kind of pull yourself up. It's something that I've had to tell myself a lot and also something that I repeat to a lot of other folks as well. Last question. So in your previous life, you worked at Open Door, where you led work on basically figuring out how much to pay for houses, you basically built the model that told the company here's how much we'll pay for this house. What's like a variable in the price of a house that you didn't expect is really important and impacts the price of a house? There's a bunch that we're surprising. I'll maybe list the couple of most interesting ones. Power lines and like high voltage power lines. like are super, super, it actually impact your price quite a lot. I didn't really fully internalize this

Starting point is 01:16:57 until I went to, like, Dallas and observed, like, when your house sits next to one of these giant, like, you know, voltage lines is, like, buzzing. And most people have families. You don't want your kids kind of near there. So I think that was one that really, really kind of surprised me. And that makes sense. Yeah.

Starting point is 01:17:11 And then the other one, which was something that was always something really difficult for us to quantify was floor plans. And so it is very important. Like, yes, of course, it's really important. But just, like, quantifying what a good floor plan is like and what a really bad floor plan is like. We were doing all these things of, like, how wide is the kitchen? And, like, is it a, what style of kitchen is it?

Starting point is 01:17:33 And then, like, where's the master bedroom? And so it was just really, really hard to quantify. But I remember floor plan was a big one because, like, we'd have a home that, like, wouldn't sell. And then our ops team would go in and be like, yeah, that's the floor plan issue. So, like, how do you tell? It's like, you go inside. You just feel it.

Starting point is 01:17:47 It feels, you know, the floor pint feels off. So, yeah, those are. ones that were surrising. And then the last one that was more impactful than I thought is general like curb appeal and like even like the front door. And so I actually think there's a Zillow book on this where the front door replacement tends to be the highest ROI for homes. But just like the feel of like as you walk up to the home as a buyer, what you're interacting with and the first moments of the house I think was I'd underrated its importance. That is extremely interesting. And I love that you have to figure how to do all.

Starting point is 01:18:21 all this in code and not walking around. Yeah, yeah, and floor plans, I have a bunch of stories around. Like, for floor plans, there's like, there's like, it's not digitized. There's like a handful of people who have like paper floor plans of like all these homes and like Phoenix and Dallas. Yeah, a lot of fun, fun stories from the Open Door Days. Okay, Sherwin. Thank you so much for doing this.

Starting point is 01:18:40 This was incredible. Working folks finding online and, and how can listeners be useful to you? Yeah, so I'm online on Twitter on X. I'm just at Sherwin Wu. And, yeah, I mostly just tweet about, opening I and the API and some of the products that we're launching. And then how folks can be useful to me, I love hearing about things that people are building. And so if you're working on a startup, if you're hacking on an idea, you know, would love to

Starting point is 01:19:04 just reach out to me on X. I would love to hear about what you're building and learn about how opening I can help support you. Amazing. Sherwin, thank you so much for being here. Yeah. Thank you, Leni. Bye, everyone. Thank you so much for listening.

Starting point is 01:19:18 If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotter, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lenniespodcast.com. See you in the next episode.

Lenny's Podcast: Product | Career | Growth - “Engineers are becoming sorcerers” | The future of software development with OpenAI’s Sherwin Wu

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.