Lenny's Podcast: Product | Career | Growth - Failure

Episode Date: December 13, 2023

In this special compilation episode, we delve into failure—an overlooked source of wisdom. From freezing onstage in front of thousands of people, to coworkers staging an intervention, to huge product investments that went to zero, we’ve pulled our favorite stories of failure from 100+ podcast episodes. I hope these stories serve as a gentle nudge to view failure not as a setback but as a crucial detour toward growth.—Brought to you by Sendbird—The (all-in-one) communications API platform for mobile apps | Eppo—Run reliable, impactful experiments—Find the full transcript at: https://www.lennysnewsletter.com/p/failure—Where to find Lenny:• Newsletter: https://www.lennysnewsletter.com• X: https://twitter.com/lennysan• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/—In this episode, you’ll hear from:(00:00) Lenny: Why I’m focusing on failure(03:25) Katie Dill: The single meeting that changed how Katie leads forever(08:03) Paul Adams: Freezing onstage in front of thousands of people(18:38) Tom Conrad: Lessons from Pets.com and Quibi—two of the most famous product disasters of all time(33:19) Sri Batchu: When you fail, make sure you fail conclusively(39:00) Jiaona Zhang (JZ): One of the biggest product misses at Airbnb(44:32) Gina Gotthilf: Everyone has an “A side” and a “B side,” and we should all share our B sides more(57:57) Maggie Crowley: Her favorite interview question about failure, and lessons from a personal failure(1:00:33) Thanks for listening—Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email [e-mail].—Lenny may be an investor in the companies discussed. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.lennysnewsletter.com/subscribe

Today we've got another very special compilation episode. Something I've been pulling on more and more with the podcast and the newsletter, in case you've noticed, is failure. Normally, I spend a lot of time researching how the best companies and the best product leaders operate, but you can learn a lot and often a lot more from failure. And so what we've done with this episode is we've looked at all of the past episodes we've done and pulled out all the most interesting and insightful stories of failure and turned it into this very focused episode on failure. I hope that you find this useful and interesting.

Let us know what you think in the comments on YouTube or on lennies newsletter.com or just let me know on Twitter. If you like this, we'll keep doing this. If not, we'll probably not. Either way, I hope you enjoy. Before we dive in, here is a short word from our wonderful sponsors. Let me tell you about our product called Send Bird, the all-in-one communications API platform designed for both web and mobile apps.

In a world saturated with multi-channel communication, product teams are discovering the effectiveness of in-app communication. With Sendbird, businesses can elevate their in-app experience with decluttered and branded communication, featuring AI-powered chat bots, one-way messages, chat, video calls, and live-stream capabilities, all tailored for commerce, marketing, and top-tier support. Forward-thinking companies such as Hinge, Patreon, Yahoo, Accolate, and more. Use Sendbird to build in-app communication experiences that drive engagement, conversion, and retention. In-app communication has the highest conversion, highest engagement, and highest satisfaction of any communication channel. And when it comes to investing in this channel, trust Sendbert to take your in-up communication experience to the next level.

Start today with Sendbert's free plan. And as a listener of Lenny's podcast, you'll get an additional two months of unlimited usage and access to all premium features, including creating your very own generative AI chat bot. Visit sendbird.com slash Lenny to begin your free journey. That's sendbird.com slash Lenny. This episode is brought to you by Epo. Epo is a next generation AB testing and feature management platform built by alums of Airbnb and Snowflake for modern growth teams. Companies like Twitch, Miro, ClickUp, and Draft Kings rely on Epo to power their experiments. Experimentation is increasingly essential for driving growth and for understanding the performance of new features. And Epo helps you increase experimentation velocity while unlocking rigorous,

deep analysis in a way that no other commercial tool does. When I was at Airbnb, one of the things that I left most was our experimentation platform, where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own. Epo does all that and more, with advanced statistical methods that can help you shave weeks off experiment time, an accessible UI for diving deeper into performance and out-of-the-box reporting that helps you avoid annoying, prolonged analytics cycles. Epo also makes it easy for you to share experiment insights with your team, sparking new ideas for the AB testing flywheel. Epio powers experimentation across every use case, including product, growth, machine learning, monetization, and email marketing.

Check out Epo at getepo.com.com slash Lenny and 10x your experiment velocity. That's get-epppo.com slash Lenny. All righty. First up, we've got Katie Dill, who is Head of Design at Stripe and former head of design at Airbnb and Lyft, sharing this amazing story of how the entire design team at Airbnb basically rebelled against her soon after she joined and what she learned from that experience. I'm happy to talk about it because, frankly, it was the biggest learning experience of my leadership career, or at least that happened in one moment. And it happened in my early

days at Airbnb. So I was hired to take on the design organization or the experience design organization. That's basically the product design team, which was 10 people at the time. And so they had been reporting directly to one of the founders. And they were going to start reporting to me. And during my interview process, I learned a lot about what was working and what wasn't working and some of the trials and tribulations with the design organization and its collaboration with others. So it seemed like there was room for improvement in how engineering, product management, and design all worked together. And there was also really low engagement scores in the design team. And so I kind of came in, ready to go and, you know, excited to try to help make some change

based on all the things that I had learned from various leaders and people across the company. And I, you know, came in swinging, ready to go. And then about a month into my time there, I got a meeting on my calendar Thursday, 830 a.m. It was an hour and a half with half of the design teams. That was five people and our HR partner. Oh, no. That's never a good sign. Yeah. And I remember this so vividly and remember walking into the office and all the rooms in Airbnb's office are very unique spaces that look like Airbnb's. But of course, this was the one room with like all white walls and just like, a gray, you know, flat, rectangle table. And I walked into the room and there were five of them seated around the table and they had a pack of papers in front of them. And they went on, taking turns, you know, quietly reading from the papers, all the things that they saw that I was doing wrong and all the things that they didn't like about me. And, you know, it was a really hard

moment. I went through all the usual kind of like stages of grief when one hears feedback, which is just like, you know, immediate want to like respond to be like, oh, like, you know, well, there was a good reason for that. And like, that's not how it actually was. And this is why I did that. But luckily, I had, you know, goodness, I had the sense to just like listen and not respond in that way. I mean, clearly what they were telling me is that, you know, that was one of the things that was missing. And so I heard them out and, you know, took it all in. And, And regardless of each individual saying, what was very clear was that, you know, the missing piece, the theme that was across all of that is that I hadn't earned their trust. So whether,

you know, how right or how wrong what I was doing was is like the key piece is that I wasn't bringing the team along with me. They had no idea that they could trust in, you know, what I was trying to build and what I was trying to shape and that I cared about them and that I had, you know, their best interest and shared goals at heart. And that was absolutely. my fault. And in retrospect, you know, as hard as that was, I'm like very grateful and very amazed that, you know, that they could kind of come together and, and share that with me. It can be hard to, you know, bring, you know, feedback forward like that. And so it was an extremely valuable learning experience. And I took from that to then immediately shift how I was operating. And

really a key part in building trust was to, to listen, to hear out, you know, what, you know, the individuals and the team were, you know, setting out to do what they cared about that, what motivated them. And so I started to make pretty fast change and still moving in the direction that, you know, was necessary for the org to make like the, you know, the really large impact in how we were operating, but bringing folks along with me is you can inflict change on people, but if you want to do it with them, you really, you know, trust is the key element there. And then, you know, a couple months later, we had the best engagement scores in the company. So, like, it actually, like, you know, it did, you know, objectively improve the situation.

And, you know, since then, taken that on into next steps and other companies that I've joined and just think about, like, instead of coming and swinging, you know, come in listening, so that you can really, you know, set out to make change that actually has, like, true, positive impact on the folks around you and you bring along with you. Next up, Paul Adams, chief product officer at Intercom, sharing the nightmare experience of freezing on stage in front of thousands of people having to walk off stage and then people hearing him curse because the mic was still on. And then how he recovered. Plus, a few stories of building some of Google's most infamous product failures. Yeah, some things that happen in work,

every member of the time. They don't really scar you. This goes in the book that have scarred for life. Yeah, it's good a long story short. I was at Facebook just over a decade ago. Loved it at the time. I think it was a great place to be at the time. And at San Francisco, I did a lot of talks for Facebook, internally and externally. Facebook had a keynote slot. Always had a keynote slot at Cannes, the world's biggest advertising festival. And the year prior, Zuck had been interviewed. He was the speaker.

He'd been interviewed. He'd gotten a hard time on privacy. It didn't go well as well as they'd hoped. So the next year, they asked me to do it. Maybe it was the Irish accent, you know, that made the offer come my way. And, yeah, I got out in some sense of state. the world's biggest advertising stage. And I'd say I was like three, four minutes into the talk.

A talk I'd given, a very similar talk to what I'd given lots of times. And I just froze. I couldn't remember what I was supposed to say. It was the first ever time of my life I'd rehearse to talk word for word. You know, usually like I have talking points and I'd live and, you know, things get mixed around and it's kind of informal. This was like, you know, media trained, like, don't, do not say the wrong thing kind of talk. And I just could not remember what to say. had some version of a panic attack, walked off stage, I was still miced up, cursed,

everyone started laughing. I was like, she's, are they laughing at me? You know, oh my God, this is, but I, I managed to turn it around. I walked back out. I'd kind of been disarmed internally in my head and the mess of it went well. But it was, and I was famous that night, you know, out in Cannes afterwards, like on the, on the, whatever, the sea front, it's just like rosé everywhere. And yeah, I was famous and infamous for my performance. I feel like you live the worst nightmare that everybody has when they're thinking about giving a talk. And I think what's interesting is you survived. And I think that's a really interesting lesson. It's like you could freeze in front of thousands of people walk off stage.

And then it works out okay. Yeah. And it all happened kind of organically, I guess, very naturally, you know. But yeah, ever since then, every time I walk out on to a conference talk stage. Still, today, I ask myself, I have this tiny doubt in the back of my head. It's never happened since, but yeah, you just, I think you have to go with it with these things, you know, like when life kind of throws you these whatever curveballs, you have got to kind of adapt and it's not that big a deal. None of these things are that big a deal at the end of the day, you know, you kind of move on, live and learn. So, yeah, but I still hope it doesn't happen again. I also hate public speaking and I always fear this is exactly what's going to happen to me. And so

I think this is nice to hear that even when the worst possible thing basically happens, things can survive. You can turn it around, yeah. A second area I wanted to hear from is your time at Google, and there's a couple of products you worked on at Google. Both of them were not what you call big successes. And then there's a kind of a transition to Facebook, which was also kind of messy. Can you just share a couple stories from that time? Yeah, similar to the kind of like, you know, walking our stage thing, you live in a learn and I was at Google for four years and I was on Facebook for kind of two and a half years or so

at Google I worked on a lot of failed social projects like you mentioned Google buzz and then later Google Plus I think a lot of the motivation for those projects came for a place of fear you know it didn't come up for a place of let's make a great product for people let's like really understand the things people struggle with when communicating with family and friends like that's really really try and create something wonderful it came from a place of fear. And so during those times, I learned, I think, how not to lead in places. And by the I should say, you know, at the time in Google, it was other things happening that

were amazing. Like Google were building Google Maps. Incredible product, one of my favorite products. I think one of the best products ever made. They were building Android. You know, I was kind of in the mobile team, in the mobile apps team at the time that Android came out. So I can make an incredibly good product.

So I just happened to be in the social side, which was in this cut. And yeah, we, Google Bose is kind of a privacy disaster. And Google Plus is similar. And so kind of halfway through, I kind of published research about groups and how I've done a ton of research. An interesting kind of side note there is at the time I was been asked, I was working in the UX team as a researcher. I was been asked to do a lot of tactical research, like usability study type stuff. I can people use these products? and I ended up doing a lot of formative research as well in the same session.

So I'd kind of say to the team like, hey, I'll do the research, I'll answer your questions. But also, I'm going to do this other thing. I'm going to take 20 minutes doing that. And so what I used to do is, what I used to do with people was map out their social network, all the people in it, their family, their friends, how they communicate. We'd map on all with channels. We'd talk about what worked well, what didn't. And we did this with dozens and dozens of people over the course, maybe 18 months.

and the same pattern emerged every single time, which was people need way better ways to communicate with small groups of family and friends. I kind of look back now and go like, WhatsApp, you know, or like, maybe like I message if everyone's on Apple, but like really obvious in hindsight, but at the time not obvious.

And so we kind of tried to build a product around that called Google Plus. But again, it was kind of motivated from the wrong place. And so halfway through the research that had kind of done, all this research, public to a conference talk and suck and Facebook notice got in touch. One thing led to another and I left and joined

Facebook, which was an amazing thing for me personally. Facebook was amazing, an amazing place at the time and exciting and they were trying to do things for the other reasons, the kind of good reasons, like, hey, let's build an amazing product for people. And this was during Google Plus being built, you basically

shifted. Yeah, Midway. I'm stressed it even telling you about. The project hadn't been launched. it was still under wraps, you know, it was highly confidential. Google had done a lot of things at the time that were the first for them. I don't know if they've done them since. But things like everyone worked in Google Plus was sent to a different building, that building had a different key card.

If you didn't work in Google Plus, you could not get in. All sorts of like kind of countercultural things at the time. And as a result, there was a lot of, you know, antagonism internally for Google Plus. And so when I left in the middle of the project, kind of leaving with all of the plans in my head, to the enemy. Some people saw me as a traitor, understandably. Other people thought I was enlightened, you know, it's a bad thing you talked to. But it was, like, it was the right thing for me to do. But at the time, you know, it was a hard thing to do.

I know there's also like a lot of scrutiny in what you took with you and the process. Yeah, when I left Google kind of assumed that I was one of the spies, you know, I was quarantined. I told them I was leaving. they, you know, forensically analyzed my laptop, like all sorts of stuff like that. So it was pretty intense. You know, looking back, I can understand why that happened. But the root cause for me is that the project has been run from a place of fear, competitive fear, which I don't think leads to good things.

So one of the themes through the stories you just shared is, let's say failure, I don't want to make it that harsh, but just things not working out. And I'm curious as a product leader, how important you think that is for people to go through if you think that's something that is almost a good thing. And I guess just is there anything there that you find helpful as a coach, as a mentor, as someone, two people that are trying to become basically you. Very, very. It still is. You know, like, I've personally failed so many times, you know, like there are two stories and the Google one is like long deep tentacles. Like, there are two stories.

I failed a ton of times. Like at Intercom, I remember, like, you know, when I was at Facebook, I was very happy, and Owen, I knew Owen and does to the co-founders of Intercom, and they were trying to persuade me to join Intercom. We were like, it was like 10-person company at the time. But Owen said something to me at that time, which has stuck with me ever since, he said,

you know, at Facebook, you can design the product, but at Intercom, you can design the company. And that was extremely appealing to me, like a great pitch. he's like just design the company with us that you want to work in. And so part of that was a company that embraces failure that says it's okay to try things. I'm a big believer in like big bets, you know, higher risk, higher reward.

I don't get as excited about incremental things. Now, I haven't said that, of course, a place for that too, especially as companies get bigger. But I get excited about like big bets. And if you make big bets, you're going to get a lot of it wrong. So a lot of the principles that we built here at Intercom or in building software, like we have a principle called ship to learn. And we've actually changed it since over the wall here.

Ship fast, ship early, ship often is what it says now. You say ship to learn. Ship fast, ship early, ship often. It's like in that idea is the idea of failure. You know, it's not going to go right. And it's going to go wrong more often than not. But if you ship early and fast and learn fast, you can change fast and you can improve fast. And that's kind of how we, that's the kind of culture that we, as much the most

as much as possible try to embrace and teach people, but it's much easier said than done. Especially when you're in the moment, like, God damn it, everything's going to fall apart. I really, I really mess this one out. Yeah, and there's a tradeoff with quality that people really struggle with. Like, you know, we've high standards of ourselves. A lot of intercom comes from a kind of design founder background. We value the craft a lot. We never want to be embarrassed by what we ship.

So there's a real tension there, a real tradeoff where people have these high standards. which we encourage, are we encouraged to ship fast and learn and make mistakes? It's a constant kind of tension that we're navigating. Next up, we have Tom Conrad, who is chief product officer at Quibi, and engineering leader at Pets.com, two of the most memorable failures in product history, sharing his lessons from those wild experiences. Tom is currently CEO of Zero Longevity Science,

which is a killer business and an app in case you haven't come across it. I'd definitely check it out. Here's Tom. You brought up this phrase of notable disasters, and I want to talk about that. You worked at two of the most famous notable disasters of product companies, pets.com and quibby. I think it's really rare. Someone sees the inside of so much hype and then such a fall at a company. And so I just want to spend some time in these two areas. And maybe the way to set it up is just what's a lesson you took away from each of these two experiences that you've taken with you to future? work and maybe advice you share with people.

Probably the biggest lesson. It's not really about the specifics of the business. The biggest lesson, really, is these things make you better. In some instances, actually, I think in both instances, they became kind of dominoes that opened doors for me in my own ambition and my own sort of professional life that maybe just wouldn't have opened at all if I hadn't gone to those companies and learned to those things, had those experiences.

And frankly, even in the case of Pets.com, like even the high profile nature of it. I could have worked at, you know, one of a thousand e-commerce websites in 1999. And when I went on to some subsequent job interview or something and talked about my experience, who would look like I would never heard of the thing that you worked on. But everybody certainly heard about pets.com.

It's a pretty funny example, too, of how some struggles are like timeless, you know, that was 23, 24 years ago now. And while as a leadership team, we made, I'm sure, all kinds of mistakes. One of the things that happened was that there were three kind of overfunded pet e-commerce sites. And we all raised in excess of $50 million, which is a tremendous amount of money now. It was a tremendous amount of money then. And we all thought it was a zero-sum game and that we,

as one player started to spend on promotion or to spend irrationally on, you know, national broadcast television advertising, we all did. And it became this kind of unwinnable arms race. So there was like a, there is like, I think, a fundamental lesson about the like the, you know, having an excess of, of investment can be its own arbitras or or, or, uh, lead you to make decisions that maybe would be or unwise. And then of course, it's just like, you know, timing is really important. You know, the, you know, Chewy is a online pet store for $9 billion today. They were a private company and bought by PetSmart and then spun back out,

but when they were bought by PetSmart, they were required for $3 billion. You know, biggest e-commerce acquisition of all time. And while I think it's probably unfair to compare Chewy, who, executed exceptionally well over a decade, grew their business brick by brick and turned it into something really remarkable to Pest.com, which was in a very, very different moment in time and tried to go to market in a really different way. The critique that is often leveled at Pest.com, or at least at the time, was like, this is just a stupid business. They're shipping dog food around. You could never make that work. And like, that's just wrong. Like, you absolutely can make it work. Probably can't make it work when 80% of the country on the internet is still on dial-up.

It was really, really early. I saw a stat. I think you shared somewhere that you took Pets.com from nothing to a public company to completely out of business in 19 months. Yeah. Yeah, the thing that's about right. You know, the other thing that's forgotten in the tale is that we, we actually didn't go bankrupt. We shut the company down and returned the remaining balance to the investors. where no public company had ever done before.

And the leadership team just, like, reached the conclusion that given the way market conditions had evolved, there was just no way we were going to be able to get more capital into the company. And it was a company that required, you know, additional investment to get to profitability. And so it was better to sort of wind down early, take the money that we had in the bank and get it back to investors than to just spend every last penny on, like, you know, it was sort of a fruitless attempt to salvage it. Did not know that.

Let's talk about Quibi. What went wrong there? Do you think there was a path to Quibi having worked out any big lessons they took away from that experience that you bring with you? The kind of miraculous thing about Quibi for me was it relit my enthusiasm for the industry for doing this work. I had left in, I think it was December of 2000. And I thought that maybe I was just done making software. I had done it for a really long time. I done for like 25 years or something.

And I had changed a lot. The industry had changed a lot. And I thought maybe I just didn't have the same passion for it that I had, you know, a decade before. It also seemed like maybe it would be fun to have like another chapter of my life. That was just completely different. And I had a whole list of things that I thought I might want to do.

I mean, they were really, they're kind of ridiculous. It's like, maybe I want to be a pastry chef. Maybe I want to be a landscape photographer. Maybe I want to, you know, learn to make bad music to put up on SoundCloud or something. And really the only thing they had in common were they were all things that I knew nothing about. People would be like, oh, you think you might want to be a pastry chef? Do you like to bake?

And I'd be like, no, I don't know anything about baking. You know, oh, you think you might have landscape photography. Do you take photos? No, I don't make photos. But I was kind of committed to the bit, actually to the point where when TechCrunch interviewed me about my departure from Snapchat, I was like, you know, I'm out. I'm going to do something else entirely. That story is very much out there.

But a few months after my last day at Snap, I got a call from Meg Whitman and Jeffrey Katzenberg, who were starting up, it was called New Teat at the time. And, you know, the pitch was we're going to try to take the best of mobile and Silicon Valley and consumer tech and sort of weld it to the best of sort of Hollywood style. content production to build something like completely bespoke and purpose made for consumption on the phone. They were looking for both technology leadership and product leadership and wanted

to know if I was interested in one or both. And I took the meeting, even though I wasn't only taking these kinds of calls from anybody, it just seemed like who's going to pass up the opportunity to have lunch with the two of them. So I listened to the pitch and politely declined and told them that I was going to be a pastry chef or something. And we kept doing that every couple of months for like seven months. We'd go to lunch. They would give me an update on the progress they were making and I would decline, you know, the invitation to get involved somehow. And then late in that year, I went to lunch one more time and Meg explained that they brought on someone to lead technology and they brought another person to lead product and both of them

for really truly for reasons that are completely disconnected from from quibby itself both of them had left after about six weeks and meg's like we've raised all this money and we've told the world that we're shipping this product in about a year we got an awful lot to do and i really could use some help and i would like consider it a personal favor if you would come and spend just a couple days a week helping. You know, she's like, I'll continue to look for someone who actually wants the job, but it would really be, it would be, you know, great if you could help me get this off the ground. And my wife is a freelance writer, marketing strategist, and loves her life as a freelance contributor. And she's like, you should do this. Like, why not? It's two days a week. It's just a few months.

What's the worst thing that could happen? Like, maybe you'll like it. And I'm like, no, no, the word, here's the thing that will happen. Like, I won't do it two days a week. It will like immediately be three days, then four days, then five days, and six days. Like, I just know myself. And she's like, no, she's like, you know, just on Wednesday night at six o'clock, close your Quibi laptop and be like, all they're paying me for is for Tuesday and Wednesday and then open it back up on Tuesday morning. Like, that's all you've got to do. Well, you know, she's right about most things. If she's wrong about this, I fell deeply into it right away. And, it was just so fun to get to build a team from scratch and to design and build a product from

scratch and to take advantage of all of the sort of modern software architecture stuff that had come into being over the course of the 15 years since we had started Pandora. And, you know, I'm embarrassed about some of what happened with Quibi for sure. but I'm super grateful for the experience because I just really fell in love with the industry again and was reminded of just how rewarding it can be to build something and to try to put it out there even if you stumble pretty mightily along the way. Is there something that you took away from that experience that taught you what to try to avoid, what to try to pull towards?

I think I'd sort of misunderstood or misunderstood, or misjudged companies sometimes by thinking about them, like really focused on the product execution. You know, kind of if you find an interesting problem that people, you know, people care about and you, you solve that problem in a really beautiful,

elegant, delightful way that's 10 times better than anything else that they can get in that same space. They'll tell their friends and like all the rest of it will sort of take care of itself. And so that was always my ambition. Find a thing that I could about building. Do a great job building it a really delightful way. Go really deep on listening to people and their feedback and iterate your way to success

and sort of braving through that membrane that we all strive to get across the kind of really great word of mouth. But I think the thing I've come to better appreciate is that companies are also kind of like a, they're kind of a math problem that describes how you take investment. and pour them into the equation, and out the other side comes returns on some time horizon. And yes, there are variables in that equation that are influenced by the product that you build and all of the little details and decisions that you make about making that product great. But if the equation is fundamentally broken or a big swing in and of itself, no amount of like iteration and execution can like kind of get you out of the the failed outputs of the broken equation.

And I think, you know, Quibi made a bet that you could build an entirely bespoke content library that was sufficiently scaled to get people to subscribe and retain. for a couple billion dollars. I mean, it was a huge amount of money. But, you know, we made, we made 70 shows in 18 months, which is like more content than all of the major broadcast networks combined, made in a single year. So it was a pretty major accomplishment. And we made a bet that we would augment those sort of episodic and serialized

for Hollywood style shows with a bunch of daily content that we produce at the level of like network television, nightly news and so forth, that would be an alternative to some of the sort of daily content that you might otherwise get on YouTube. And that was going to be about a third of the content spent. One like super interesting thing that no one talks about

is that all of that content was designed to be made like day of or day before it aired. So there was no back catalog of it. And it was all designed to be shot in these professional studios that we built out. And it was really expensive. Like I said, it was like a third of the investment we were going to make in content, almost half the investment we were going to make making content.

And we launched two weeks into COVID and we couldn't make any of that content except literally in the garages of the host homes. And so we had this thing that was supposed to seem like really set apart from YouTube that literally now was being made exactly like YouTube content. which is sort of like self-produced at home with, you know, very little sort of of the support infrastructure of Hollywood. Now, you can argue, I mean, I think the content on YouTube is really, really exceptional in this category.

And maybe we were never going to do better than that. But I think it was really fundamentally broken with Quibi was that the actual foundational equation of can you make enough premium content that's totally bespoke and made for the service and takes advantage of the unique nature of the phone. It is that enough content to get people to sign up and retain. And can you do that for a couple billion dollars? And I think the answer is no. The library has to be much, much bigger.

And you have to have, like any company, you have to have sufficient time and energy to iterate on the content format itself. Because we were really, our roadmap really wanted to innovate on the content format. And so I think part of what happened is pretty quickly it became clear. that the math was just wrong. It wasn't going to take $2 billion. It was going to take $6 or $8 or $10 billion. And the risk-reward profile of betting $10 billion on the format was just more than anyone can stony.

Next up, we've got Shri Bachu, former head of growth at Ramp, who shares something that I've thought about ever since we had this conversation, which is this idea that when you fail, make sure you fail conclusively. to make this failure an actual learning that you can build off of versus just a waste of time. Here's Shri. Growth experiments in my history are typically like 30%-ish success rate. So the vast majority of things that you try don't work. And so you want to create a culture where people aren't afraid to take risks and aren't afraid to fail.

And for me, failure is not that you didn't drive revenue. Failure is not learning. So it's really important that you learn when you fail. And so we celebrate failure as long as you're learning. And you can only learn if you've designed the right test and you failed conclusively. Because otherwise, I think many of us have been in situations where there's intuition that something might work and it doesn't work. And then you end up doing it over and over for years. Because every time a new executive where somebody else has the same idea, you try it again.

And it's because you haven't been able to design the test to fail conclusively. And it's hard to do, but at the end of the day, there's only two ways to, you know, make an experiment successful. Either you have a very large N or you have a very significant treatment, which is like, you know, what you're doing in the experiment itself. And, you know, in B2B, you don't usually have the luxury of large N, which you do in consumer. Facebook can get stats, say, in two hours. You know, a B2B company could take two years to get to get to get. the same number of, you know, touchpoints. And, and so to counteract that, I recommend people, like, just trying to maximize the treatment effect, which is like, if you have a hypothesis

that you're testing, just throw all of the possible tactics and resources that you think would move that needle. Because you can always cost rationalize later if it works, right? And so just maximize the treatment effect. And if with all of that, it didn't work, then you can saying, hey, like, we're not going to try this again because we literally did try everything that we could to test this hypothesis. And if it doesn't work in the best version and it's expensive as it is, this is not worth spending more time on. But if it does work great, then you do another version of the test with like half the tactics or whichever taxes you think work better or worse and you optimize over time. Is there an example you could share when you did that?

I mean, account-based marketing is something that is very common in enterprise software, right, where you selected certain customers that you think are high priority. And you're saying, I want to touch them in as many nuanced ways as possible to see if that drives conversion. And this is something I've seen tried many times where people do it, but they kind of, you know, do it halfway where they're like, okay, tried these three things, conversion of the control route, like, wasn't higher. And so we're, you know, we think it's, does not going to work. And then a new go-to-market executive comes, and they have to do it again. They have to do it again. It's like a very common one, wherever this happens. And, and so when we

did it at RAMP, we did exactly, you know, what I just described, which is like, let's really, be thoughtful about the experiment design, both in terms of like maximizing the number of people as well as, you know, maximizing the number of ways and types of ways that were effectively touching these target customers to show the value one or the other. So what it sounds like is the hypothesis isn't like this email will have a big impact on conversion. It's like this strategy of coming after customers is what we're testing. That's the example there, right? And I think like, For example, if you have the, it's a, this kind of framework is more important for cross-functional, larger scale, bigger tests, rather than like, you know, an email modification, right?

But we can even use it on a micro example, like an email modification, right? Where you are like, okay, I think this particular email is underperforming because it's not talking to this part of the customer's paypoint or journey. or whatever. And you could just, the simplest test would be, okay, let me make some tweaks and to the text and edit that. And that could be the end of that test. And if that doesn't work, you're like, oh, maybe those weren't the right text edits. Let me do different text edits or whatever. And that's fine. That's low cost. It's not the end of the world. And it's for you to be wrong there. But an alternative that you could do is like, oh, what are all of the things that I could change about this email in the same test? Is it the trigger of the email? Is it the text? Is it the

text content of the email? Is it, you know, additional personalization? Is it the design of the email? Like trying to think about what are all of the various levers that you think could be wrong and put them all together to test your hypothesis of this touch point is wrong. And how do I improve that? Well, obviously the downside of that is you don't, if it doesn't work, you don't know if it's like, oh, maybe this thing could have worked in the subject. Yeah, yeah. So there's always tradeoffs on this. And so, but what you're hoping is like you've done a complete refresh. where you did all the things that you thought were intuitive, that should work. And if it doesn't work, then you're like, okay, maybe my hypothesis is wrong.

But you're right. It's always going to be a challenge of maybe the execution is wrong. And I did too many things, potentially in that case. Our next story is from Jay-Z, who is a colleague of mine at Airbnb, head of product at Webflow when we recorded this episode. And this is her sharing the story of one of the biggest product misses at Airbnb. You've seen a lot of new PMs. And you've seen these PMs succeed, you've seen some fail.

What are the most common mistakes that you find new PMs make in this experience of kind of helping new PMs get into the field? I think something that is really hard to untrain, but I think every human does it is you jump to solutions. And so one of the biggest things I see not just in my course, but also just as a PM and like some of the mistakes that you make as a PM, is the idea of get really attached to a solution, a way of implementing something, something that you can see in your head that you want to build. And so that's the first thing I really want to un-teach in our course. And so a lot of people will literally come in. They'll be like, I want to build X startup or I want to do this thing. Or I'm in blank school and I've been doing a lot of research on this particular area. And so like untraining that and being like, hey, we're going to go out there.

We are not going to think at all about the thing that you want to build. But instead we're going to be focused on users and people in the real world and their problems. And the first step is to understand their problems and then to understand if there's an opportunity here. as opposed to, hey, you want to build X thing for Y person. So that's the biggest mistake that you really have to unteach and like retrain thinking around. So let's kind of go to the other side of this question. We talked about what mistakes new teams make. I'm curious, what's the biggest product mistake that you've made?

Wow, that's a good one. It's so interesting. I feel like as product people, we're always making mistakes and we're always learning. Maybe I'll give an example from Airbnb since you and I were both there. And this one does stand out to me. So, you know, we're working on this concept called Airbnb Plus. You know, if you took a step back, we're really trying to do is to be like, hey, not everyone trusts Airbnb in terms of, you know, it's a platform.

It's not like it's managed inventory. It's not a hotel. How do you go in and really make sure that we're like all the Airbnb's aren't meeting the quality bar? But I do think we were very solution first. And I think we're also a competitor afraid at the time, right? So it was during a time where there were managed marketplaces. There were the saunders out there. And I think that as a company,

we're very much like, oh, good this, like, what are we going to do in the world of managed marketplaces? And so we went really hard down the solution space. We essentially were like, let's go inspect our inventory. Let's actually try to manage our inventory more. And really what we should have done is taking a step back and be like, what's the real problem? The real problem is people want to know what they're getting themselves into. We need to represent the homes a lot better. And I think the other piece here that's really important is what as a company is their strategic strength and like what's in your wheelhouse. So for example, Airbnb, we weren't that strong in operations. Again, we're this platform with this marketplace, right? And so if you don't have that

muscle and then you're asking the company, the teams to essentially build it from the ground up, that's really, really difficult. Not to mention the unit economics. Are the unit economics actually going to work, even as you scale? Yeah. I feel like Airbnb Plus is an untold story that somebody should tell and that could be its own podcast, I guess. You and I can tell it. We could tell this could be Airbnb plus the hidden the story. As you said, the problem is trying to solve was people don't really trust. They don't want to even consider Airbnb because it's like, no, I don't want to stay in someone's home. I don't know what it'll be. It's unpredictable. And so as an outsider, it felt like a really clever approach. We're going to vet them.

We're going to make sure they're awesome. There's a minimum bar. I think the, and I guess this is the question is, do you think it was just like, this is never possible because we'll never make money as a business doing this because we don't make it that much per bowl. booking and investing time, resources, sending people pillows, all that stuff is ever going to be economical? Or do you think there was a path and was just not executed well? I think there wasn't really a clear path. I think there's less than execution. Exactly. And it's more just like if you understood, again, this is my point around unit economics, there are things where I think you have like magical thinking around unit economics. You're like,

well, when we get to the scale of X, it's all going to work out. We can make these things happen. I think you actually need to really make sure the unit economics work quite at the beginning. So that is definitely one lesson. And I think the other thing is, and going back to the spirit of what are you trying to achieve, if you're trying to achieve this idea of like really knowing the quality of the place and for a platform like Airbnb, the right way to go about doing is through our reviews, right? Through our guest reviews, which are essentially free as opposed to literally sending out inspectors. And I think the other things are if you can get signal on what are the things around quality that people care about.

Is it cleaning? Is it the, hey, I'm locked out? And I think that there are other solutions besides inspection that then get at that. So, for example, it is actually cheaper to go send everyone a lockbox than to deploy an inspector and go look at your property, right? It is actually cheaper to maybe do a partnership with a bunch of cleaners in different local areas and then get that as part of the feet, like, as opposed to doing the inspection. So again, it's really about what are you really trying to achieve? what is the user problem in each of these areas? And can you target that problem with the particular listing that is, you know, that you're looking at?

And so, yeah, I personally don't believe the unit economics ever would have really worked out. I think we should have known that. We should have dug into that more at the very beginning. And then to get very tailored instead of like one blunt instrument to solve it all, hey, we're going to go inspect. It's like, what is the problem for this listing and what's the best solution to fix that problem? Our second to last story is from Gina Godhillif, who is an early growth leader at Duolingo. She's currently C-O-O-O-of-Latitude.

And this is her sharing a wide-ranging and important point about how everyone has both an A-side and a B-side to their career. And people often only share their A-side. So this is Gina sharing her B-side. We are very encouraged in our lives, especially professionally, to talk about our A-sides all the time, because that's what impresses people, that's what open. stores, that's what allows us to keep growing and it's so important. So it means that a lot of what you hear in podcasts and on stage ends up being the Instagramable version of someone or a company or a country's trajectory. It's just the highlights. You know, and when I talk about

my A-side, it's very impressive. You know, I did things like we'll talk about. I met President Obama. I worked on the Mike Bloomberg presidential campaign. I helped dual-lingo scale from 3 to 200 million users. I worked with Tumblr, helping them scale in America. and Reese and Horowitz invested in my company, et cetera. But between all of those highlights, there were so many B moments that get sort of like shoved under the rug because it's just easier for me and it's more impressive for others. But I really like to highlight those

because I think that most of us have a lot of B moments every day, every week, every month, and every period of our lives. And it's easy to think that things aren't just not going to work out for us because we're in one of those B moments if we don't recognize them as moments. I love this concept. We're going to talk as you expected about a lot of your A side stuff. Is there any example of a B side story of your life that would be interesting to share? Look, I think those are the most interesting because they're funny or ridiculous, you know?

I had a lot of B sides and I still do. For example, I, you know, I had no idea what I wanted to do. I actually wanted to be an, I thought I wanted to be an actress. I either wanted to be that person in SeaWorld who goes like this with the dolphin. This is before Sea World was canceled. or I wanted to be an actress. I applied to schools. I didn't get into any Ivy League.

I didn't get into any of the top schools I wanted to go to. When I got to college, I actually ended up dropping out because I got so depressed. Like incredibly depressed, couldn't get out of bed, depressed. Ironically, I dropped out of Reed College, which is the same college that Steve Jobs dropped out of. So, you know, I was just destined for greatness. I knew it at that moment.

It all makes sense looking backwards, as he said. Totally. I was dropping out being like, yes, this is exactly the past. No, I was miserable. I thought there was no path forward. And I finally went back and graduated, the college counselor looked at my curriculum and said, what have you even done with your life? There's nothing to show for. And it was shocking because I was always like the overachiever who wanted to do the maximum curriculum and ace all of my classes and do whatever. I did three diplomas in high school, the internationally American, the Brazilian. And so that for me, I think my one learning there that has stuck with me and I think can work for other people too. is that it's not just about doing things that actually matter and learning. It's about being able to tell the story. And it's about understanding what other people perceive as valuable. I applied to 100 companies.

I didn't hear back from most of them. I finally got an internship at kind of like a Tier B slash C digital marketing agency in New York City because I wanted to live in New York so badly. And they forgot to apply for my visa on time. so I lost my visa and had to go back to Brazil. And then I ended up leaving that organization to go work for another one. And I won't even go into the details of the shadiness of that company that I worked for. But then they ended up laying me off.

So I lost my visa again, had to go back home, found another opportunity, got fired that time. So there was just a lot of rockiness in my start that I don't think he would imagine when you see someone up on stage, like leading a conference for 5,000 people that I think is important. And even when I started working for Tumblr, I was like, this is it. Like, I made it. This is a really interesting company. This is going to work out. That was super rocky because it was an early stage startup.

So, for example, you know, they couldn't figure out how to wire money to Brazil. So I was not paid for six months. And at one point me and like my colleagues were trying to get money out of the teller to pay contractors because we had no money to pay them. And like, we borrowed money from people. And finally, they also laid me off because they decided to sell to Yahoo. And then I had to figure out like, what am I going to do? do, no one's going to hire me. I've been fired and laid off so many times. So this is all before I started an agency to help US-based tech companies and startups grow in Latin America because I figured

I was in like this really great place to make that happen. And it eventually worked for well-known companies such as Duolingo. At the time, they weren't well-known. They were a tiny little startup. They didn't have an Android app. And that's how I started working with Duolingo because their head of marketing connected with someone they had worked with at Flickr and said, I noticed Tumblr, grew a lot in Brazil last year. Can you recommend like a company or an agency to help? And they said, this girl. And that's how they connected me with Duolingo. And I started helping them grow in Brazil as a consultant. They were like, this is great. Can you help us grow in Chile, Argentina? And I was like, yes. They were like, how about Mexico? And I was like, yes. Did I know anything about these places,

Lenny? That I know people there? No, but like you can figure it out. And then they ended up asking me to come on full time, do that across the world, Japan, China, Korea. Turkey, Spain, France, etc. And then to own growth, which ended up meaning communication, social media, government partnerships, anything to grow. And then eventually became an EB testing growth engine with engineers and PMs and designers, of which I knew nothing about.

And even after that, like, I left Duolingo five years later, didn't know what to do with my life. You'd think, oh, wow, you have it figured out. Now you left Duolingo. You have the world in front of you. And I'm like, maybe I can finally go work for nonprofits, which is what I actually wanted to do in the first place. Try to hand at that, like had a couple of experiences

before going to work for the Mike Bloomberg campaign. Working for the Mike Bloomberg campaign is impressive. But you know what? Mike Bloomberg didn't win. He's not the president. So that was not a successful campaign, you know, if you really look at it. And, yeah, latitude seems like it's a really promising path. But there's A days and B days.

So it's just a lot of that. and just staying resilient and believing in yourself and getting back on the horse when you fall on your face. Amazing. That's such an important message. I think one of the threads from what you're describing, something that I think about a lot is, people kind of underestimate how long their career is. There's just so much time to do stuff and for things to start to work.

This is going to sound really fancy, but I think Marcus Aurelius has this quote about how our life is actually very long. We just use it really badly and we just waste a lot of our time. I think you're so right, Lenny, and I love that because people are going around being like, life is short, life is short. But that's so true. We waste so much time. But also, I think we don't, we don't recognize how much opportunity we have in front of us. And as a 26 year old, I definitely thought my career was over. You know, I was like, I blew it. And like, looking back, it's funny,

you know? Yeah, I know exactly what you mean. Like, I spent nine years at my first job at a random company in San Diego and a startup. I was like, what am I doing here? So long. And then it turned out that was really useful for the thing I did next. And then eventually, wow, things started to really take off. So I think that's a really good lesson for people just for a long time. Like, this is my fourth career. Like I've switched careers many times. I was a engineer, then I was a founder, then I was a product manager and I'm whatever this is, whatever you call this thing. I guess me too. I was an operator. I was a consultant. Well, I was an employee. I was a consultant. Then I was an operator, which is a fancy way to say employee at a startup. And then now I'm a founder and a VC and an

angel and whatever this is. Awesome. Awesome. So I think. I think that's a really important takeaway. It's just there's a lot of time to do stuff and don't stress if things are moving as fast as you want. I'm curious, what's like a mess up or a big mistake maybe that you made or your teammate that was like, oh, wow, that was a big waste of time? Yeah, look, a lot of things didn't work out. More than 50% of our A-B tests didn't work out. We made bets that didn't make sense. I will say, though, that like in the spirit of A and B sides, I and I think in general, we are really good at forgetting the B stuff.

Like I talk so much about all the stuff that worked that it's hard to remember like all of those moments that didn't actually work. And the moment and the thing that I tend to talk about, which is like this mistake that we made as the growth team is almost like one of those like when you get asked in an interview like what's your what's like your biggest weakness and you're like, I'm a perfectionist. You know, it's like one of those things that actually makes you sound good because it's the story about how my team really wanted to implement badges. We spent a lot of time playing all the games that were popular at the time, trying to understand how those gamification growth hacks that we could find in those apps would potentially overlay onto Duolingo and how we would do that. And Badges was just pervasive in all of the top games. And so it seemed like a no-brainer.

But since we ranked all of our experiments in terms of ROI and return being like how many users we think we're going to get from this, DAUs and time investments, it was never, it never made sense to focus on this because we thought that the time sync would be too high. So I actually ended up not letting the team run this experiment for like six months so that we focused on lower hanging fruit. So that's like a mistake on my end. Then we decided to, we decided to run this experiment in the most lean way possible. We're like, you know what? Like there's MVP's there's like minimum viable experiments. We don't have to run a whole badges thing. We can just like do something more simple and actually see if that like

leads to growth in an interesting way, and then we'll know. And we ran this very simple experiment that was like, you signed up and then you get a badge. And it was like, this girl with a balloon, I don't know, she was like happy or whatever. And of course, of course, in retrospect, it led to no results because no one is proud of signing up. It's not an exciting moment. And you don't even have badges to collect. You can't show it to other people. Like, none of the things that make badges compelling were there. But we were like, okay, well, we does it. It didn't work. And then we moved on. So we moved on for like another, I don't know, eight months. and we didn't look back.

And then when we did look back, first of all, at that point, we discovered that we hadn't been dog fooding, which also was embarrassing looking back. We hadn't been dog fooding in the growth scene. We'd just come up with hypotheses. We were super careful about prioritizing them and making sure that we were doing the best possible

write-ups and all these things. But the dog-fooding piece, it just, I didn't come from a product background. I was a marketer, and I didn't really understand the term dog-fooding. But when we thought, we had a conversation, we were like, you know what, if we had just tested that, we would have all known that this was a super lame badge. And I was like, why are we not testing our experiments?

And so like that became part of our of our practice. It's still relevant. I just had a conversation yesterday with engineers at Latitude. I haven't explained what we built yet where we're building it. Maybe we'll get there. But I was talking yesterday to engineers at Latitude. And they're awesome. Like in terms of product team, we have like the number eight employee at NewBank.

You might have heard of NewBank, but it's this like massive banking like fintech. in Latin America. And we have people from like other fintechs and then we have this guy who was a lead PM at Twilio. And I was explaining to them why we should be dog fooding. And they were all like, oh yeah, we should dog food. It's just like easy to forget stuff like that. So that was a mistake that, you know,

we could have probably gotten to the growth that we got to with badges much earlier on. And not only that we get to growth with badges, but it became like this amazing treasure trove of opportunity because once you have badges and people want them, you can now ask people to do anything. like go find friends go buy things like you know whatever it is and so like we we impacted almost all metrics across the company positively including some we hadn't expected but it's like easy to

talk about a mistake that ended up being a win so that's why I compared it to the interview thing in the beginning but you know we we tried making dualingo a social app really early on and failed it was called duels dual duels like you could duel you know very clever Yeah, I know. We were clever, but people didn't use it. We didn't figure out why. We tried making a Duolingo for Schools platform. We couldn't get it to pick up. I went and launched Duolingo in China and it got downloaded by a million people in the first day and then the app got blocked. And then, you know, because of the government. And then we couldn't figure out what to do. And then everyone rated the app like one star because it didn't work. And so then we had a really like a lot of trouble actually recovering from that. We launched Duolingo. in India and didn't realize because we couldn't have unless we went there, which we finally did, that most people set their phones, phone UI in India to English because typing in Hindi is hard. And of course, there's a lot of languages throughout India. And we were making it so that when you downloaded Duolingo, whatever UI, you open your app,

your phone was set to, we offered not that language for you to learn. That was your base language. So we were telling people, learn French, Spanish, German, from English. And they were all trying to learn English. so they didn't find what they were looking for. They left. There were so many mistakes, you know, and luckily I think we were able to bounce back

for most of them in terms of how dueling was doing today. And our final story is from Maggie Crowley, VP of Product at Toast, and one of the most beloved episodes of the podcast, this is Maggie sharing something a little bit different, her favorite interview question about failure and what it tells you about the person you're interviewing, plus a story of her own product failure.

Here's Meg. A question I ask in every product interview is, what's the worst product you've ever shipped? And that's because I don't think you're a good PM if you haven't shipped something that's really shitty. Like, you just haven't had enough reps. You haven't done enough time. And it's not only that you've done it, but that you can admit it and you know which one it is. Like, that's so important. And I remember, this is so dumb.

I'm still so mad about this that we did this. It was, I won't name which team, which company I'm not going to call that out, but we wanted, we decided we needed to do a rewrite red flag number one of a existing product and, you know, engineer who I'd work with many times. We had a really good relationship. And this person was like, yeah, yeah, it's going to take six months. No problem. Core part of the product, been around for forever. Like one of those things that the code is still is still the code written by the founders kind of thing. It didn't take six months. It took two and a half years. It still wasn't. wasn't done. It almost never, like, it went on for so much longer than it should have. It took us forever to get to feature parody. It was the worst project. Like so many people rotated in and out of it. Everyone thought it was dumb. Like sunk cost fallacy. Just the worst. And it's because, A, we got arrogant and we thought we could do it. B, we skipped discovery. We didn't really write a one pager. We just went for it. We didn't do enough technical and design research into what the requirements would actually have to be.

And there you have it. And did not work out. Where was it a huge success in the end? It changed the trajectory of the business. Absolutely not. But you know what? I didn't get fired. So like it's fine.

I feel like I've gone through those experiences. And then like three or four years later, it's like another, maybe this rewrite and redesign may work. Because we haven't updated this thing in a long time. Just don't do it. Don't rewrite. If anyone ever tells you to do a rewrite, don't do it. A side-by-side rewrite.

Nope. Yeah, I've never had a, like what I run into is once you get too far down a redesign slash rewrite, everyone's building in that new world, and then you launch it and the experiment's negative, and then it's just like, oh, we just got to launch it, we're going to claw it back. We're going to get to figure out how to get back to neutral someday. Yeah. Yeah. Yeah. Don't do that.

Good times. And that is a wrap. I hope you enjoy these stories of failure. I want to give a huge special thank you to all of our amazing guests for being vulnerable and sharing these stories of failure in their career. I hope you leave this episode with a new perspective on how setbacks and challenges and failure can often be exactly what you need

to get to the next step of your career or your life. If you've got a great story to tell about failure, I'd love to hear it. Leave a comment either on YouTube or on lenny's newsletter.com or just DM me on Twitter, or you could reach out at lennyurjitsky.com and click the big contact button. Thank you for listening.

Bye everyone.