Lenny's Podcast: Product | Career | Growth - The UX research reckoning is here | Judd Antin (Airbnb, Meta)

Starting point is 00:00:00 User-centered performance refers to customer obsession or user-centered practice that is symbolic rather than focused on learning. It's hugely common, I would argue. It's work we do to signal to each other how customer-obsessed we are, not because we want to make a different decision. If you're like, listeners are like, I don't do that. I'm kind of like, think about it for a second. This is extremely common.

Starting point is 00:00:24 Every time a PM comes to a researcher at the end of a product process and says, can you just run a quick user study, you know, just to validate our assumptions. That's user-centered performance. It's too late to matter. We got to ship it, right? What they want is to check the box. One of my big kind of mantras was, we don't validate, we falsify. We are looking to be wrong. Many PMs, many designers are not in that place. They do not want to be wrong. They're looking to validate. I mean, that's user-centered performance. Today, my guest is Judd-Anton. Judd helped build the user research practice at Facebook. He was a long time head of research at Airbnb, and his direct reports have gone on to lead research teams

Starting point is 00:01:06 at Figma, Notion, Slack, Robin Hood, Duolingo, Fair, and other amazing companies. These days, Judd spends his time consulting, helping companies with organizational challenges, product strategy, design, research, hiring, onboarding, and crisis management. In our conversation, we unpack a conclusion that Judd has come to recently about how the user research field is going through a reckoning. and what needs to change, both within the user research field and how companies leverage user research going forward. Judge shares what the user research field has gotten wrong over the last decade, how PMs and designers rely on user research too often and to answer the wrong questions,

Starting point is 00:01:47 where user research will continue to provide significant value and how to best leverage your researchers, why it's important for researchers to think about the business goals more versus just what the users need, what to look for when you're hiring you user researcher, how PMs can be better partners to researchers, and also a phenomenon that I love that Judd describes and often witnesses, that he calls user-centered performance, where everyone acts like they care about the user, but they're just doing it for show and already know what they want to do.

Starting point is 00:02:14 This episode has a lot of spicy takes and will probably upset some people, but Judd is sharing some real talk here that I think we all need to hear. With that, I bring you Judd Anton, after a short word from our sponsors. This time of year is prime for career reflection, and setting goals for professional growth. I always like to spend this time reflecting on what I accomplished the previous year, what I hope to accomplish the next year,

Starting point is 00:02:37 and whether this is the year I look for a new opportunity. That's where today's sponsor, Teal, comes in. Teal provides you with the tools to run an amazing job search. With an AI-powered resume builder, job tracker, cover letter generator, and Chrome extension that integrates with over 40 job boards, Teal is the all-in-one platform you need to run a more streamlined and efficient job search

Starting point is 00:02:57 and stand out in this competitive market. There's a reason nearly one million people have trusted Teal to run their job search. If you're thinking of making a change in the new year, leverage Teal to grow your career on your own terms. Get started for free at tealhq.com slash Lenny. That's tealhq.com slash Lenny. This episode is brought to you by Vantta, helping you streamline your security compliance to accelerate your growth.

Starting point is 00:03:24 Thousands of fast-growing companies like Gusto, Com, Cora, and modern treasury, trust VANTA to help build, scale, manage, and demonstrate their security and compliance programs and get ready for audits in weeks, not months. By offering the most in-demand security and privacy frameworks such as SOC2, ISO-271, GDPR, HIPAA, and many more, VANTA helps companies obtain the reports they need to accelerate growth, build efficient compliance processes, mitigate risks to their businesses, and build trust with external stakeholders. Over 5,000 fast-growing companies use Vanta to automate up. up to 90% of the work involved with SOC2 and these other frameworks.

Starting point is 00:04:02 For limited time, Lenny's podcast listeners get $1,000 off Vanta. Go to vanta.com slash Lenny. That's V-A-N-T-A.com slash Lenny to learn more and to claim your discounts. Get started today. Judd, thank you so much for being here. Welcome to the podcast. Lenny, thanks for having me. It's my pleasure.

Starting point is 00:04:25 So we actually worked together at Airbnb for many years. And as I was preparing for this, I realized. I realized how many of the people that you managed went on to do amazing things. I'm just going to read a list of people that work for you and what they do now. So we had Matt Gallivan, who now leads research at Slack. We have Jana Bray, who leads research at Notion. Celeste Ridlin, who leads research at Robin Hood. Rebecca Gray, who leads research at Fair.

Starting point is 00:04:51 Hannah Pelagie, who I think was leading research at Duolingo. Luis Barrell, who leads research at Figma, and then Nome, who was leading research at Wellfront. I think he moved on to something else. What a freaking crazy alumni community and group from this one team that you hired and incubated. No, I've never looked at that list, but I'll tell you, I've been so privileged to work with all these amazing humans. I can't take credit for it. They're just outstanding people. And I'm glad the diaspora is out there because, like, these people rock stars.

Starting point is 00:05:22 Okay, so the main reason that I wanted to do a podcast episode with you is that you wrote this piece that was titled, The User Research Reckoning is Here, which I understand caused quite a stir in the research community and I think adjacent communities. And let me just read one of your takeaways at the top of your post to give people a sense of what it was about. So you wrote, the user research discipline over the last 15 years is dying. The reckoning is here. The discipline can still survive and thrive. but we'd better adapt and quick. Before we get into the meat of the piece,

Starting point is 00:05:58 could you share a bit about just the reaction to this piece and maybe if it was a surprise and what you expected would happen when you put this out? Yeah, I mean, I was definitely surprised. I wrote it because I wanted to start a conversation about something I was thinking about. I didn't really know who would read it, and in the end it turned out a lot of people read it.

Starting point is 00:06:18 I learned that using the word reckoning may have been a mistake because it inspires a lot of drama in a conversation that I wanted to be really productive and positive. Overall, I would say, though, that, like, the response was very positive. It seemed to resonate with a lot of people who reached out to me. I spent a lot of time talking to teams, to designers, to researchers. But there were also a ton of critiques. I would say some of it was, like, people thought I was throwing research or researchers under the bus. Like, it's researchers' fault.

Starting point is 00:06:49 We're doing it wrong, which I don't believe at all. and that I wasn't taking responsibility as a research leader or a design leader myself. And like a real, the most interesting one, I would say, was the anti-capitalist crew. Because one of my points that we'll talk about is that I think researchers need to be more profit-focused. And there are a lot of people out there who think that's not, I think they think that's not cool or not research's job. And I'm kind of like, well, what are we doing that? And if we're not helping businesses succeed. But that was the most surprising critique for sure.

Starting point is 00:07:22 I've worked with some of those people who are just like, why are we growing? Why do we focus so much on growth? Why do we need to grow this business? Maybe it's the wrong industry for them. Yeah, I'm not a fan of that. Okay, so let's actually dig into the meat of your message and the big takeaway and kind of the conclusion of what you're finding is happening in user research. And I know a lot of this comes from, a lot of user researchers have been laid off at a lot of companies. It was one of the hardest hit teams.

Starting point is 00:07:48 And so I think a lot of this comes from that. So yeah, so let's just start big and then see where it goes. So, yeah, I mean, everybody who's paying attention has noticed that there have been a bunch of layoffs. And I think back in the summer, I was thinking, listen, this seems to have been hitting UX and UX research particularly hard. Is there something going on? Is there a bigger picture? Is it maybe the reason I use the word reckoning is because to me that's like, hey, a moment to take stock. And triggered by the fact that a lot of wonderful humans may have lost their jobs.

Starting point is 00:08:18 and many more are afraid of losing their jobs. And so if it's a sign, the fact that research has been hit so hard, it's a sign of what? And so the thesis of my article is really, it's a sign that maybe the system is a little more broken than we think. And that research is not driving the value or impact that it should or could. And that's for a bunch of reasons, I think. Some of it is stuff that research can do better.

Starting point is 00:08:42 And a lot of it is how research is integrated and positioned in companies. And at the root of all that, I think, is, that we're just doing too much of what I would consider the wrong type of research. And what I mean by the wrong type of research is I have this framework that it's in the article, macro, middle range, and micro research, at least three ways to talk about it. And it's pretty simple, the intuition of what those are. So macro research is like big picture, strategic business focus, forward-looking innovation, you know, look at the market, look at competitors, you know, long-term research to

Starting point is 00:09:16 understand where the product should go now. stuff like that. And then you have micro research, which a lot of really technical usability falls into this, all the beautiful stuff that researchers do to enable like a really high quality, excellent pixel perfect thing to go out the door. Laser focused research to understand AB test results, stuff like that. And then you have this middle range, which is this globular place where the research questions are sort of middle altitude. And a lot of the core kind of, let's say, user understanding questions fall here. And a lot of what research is doing is research in that space.

Starting point is 00:09:53 It's sort of, let's take a group of people and ask some questions about how they think, feel, behave, how they're using a product or not using a product. And it's just this devastating mix of really interesting to many, including researchers, and not impactful enough for the business. That's kind of the core thesis. Researchers do it because it's interesting, but honestly, and I think we should talk about, Lenny, is researchers also do it because it's the kind of work we most often get asked to do. Yeah, that's exactly what I was thinking. That's what, as a PM, like, that's,

Starting point is 00:10:25 that's what I want to get answers to is like, how should we think about this one product? And I totally get this. Yeah. It's like the questions turn out to be really interesting. And there are many cases and many companies where it's super impactful. But the problem with that, those types of questions, is they tend to be really, like, they trigger all the worst stuff that researchers experience. Right. So they yield results which are interesting, but sometimes hard to operationalize. They trigger the post hoc bias really, really, really well where like a lot of people can say confidently like, oh, that was kind of obvious. We knew that already. And they fulfill this kind of need for us to feel and be customer obsessed, user centered without changing anything. And so doing

Starting point is 00:11:10 doing too much of that research to me is a symptom of a broken system, right? And where, like, companies are really different from each other. I heard from so many after this article and they're like, well, my company and my industry is like this or not like this. But in tech, we spent the last many years hiring, hiring, hiring researchers. But maybe I'm sure most your listeners are familiar with the idea of a ZERP, you know, like maybe it was a zero interest J's phenomenon where it was okay when the money was easy to hire researchers, even though we were not setting them up properly. We're going to set them up to fail. We set them up as a service function. We didn't know what research was for. We didn't know how to really drive impact with it. And

Starting point is 00:11:52 that's where the reckoning comes from. It's like that era is over. Research, I think, is more crucial than ever. Researchers, great researchers are more impactful than ever, but it's in a new space. We're in a new space now. I want to make sure people understand this framework. And specifically, how would you best describe the difference between this middle range research and macro research? So middle range research is usually focused on a more specific set of research questions or a constituency. So if macro is like, let's understand the overall competitive landscape, let's do a concept car type project where we really look ahead. let's get involved with strategic planning,

Starting point is 00:12:34 which is a wonderful thing for researchers to do. You know, do TAM studies, other things like that. That stuff lives in the macro space. The middle range space is like, what's a good example? We want to know how Airbnb hosts feel about their payment options. That's like a really interesting, reasonable question, right? And we can go out and do research on that, but it's not that specific.

Starting point is 00:12:58 It's not really targeted at a business business. problem yet. It could be, right? Maybe that's a result of the research, but it yields these kind of like middle range insights in which we've learned things like, well, hosts want flexibility about their payment options. I'm making this up. But, you know, and that's a good example where it's like, it's not that that's not an interesting set of questions. It's just not quite pointed enough in order to like, and it's not framed in the language of like the funnel or the business strategy or the okay ours. It's not quite enough aligned enough to that. It's too globular in that middle level, and it ends up not driving impact. I think it also leads to a lot of the things, as you

Starting point is 00:13:40 describe people don't like about research. It delays everything. You have to wait for the research to be done to have an answer to make a clear decision. It also creates this issue that people complain about that like PMs and product teams don't want to just make a decision on their own. They're like, I will get this additional data point and make sure research tells us this is the right or instead of just trusting their got, I guess, maybe along those lines, this may be going off a little track, but what do you, what's your advice there for, say, product managers or PMs or product teams to not necessarily rely on research for that middle research? I think the reason why so many PMs ask for those middle range questions is because they

Starting point is 00:14:19 haven't really gotten deep with their researcher in a way which can leverage it for maximum impact. So if the question is like, hey, Judd, you just pointed out like a bunch of problems, like, can you be more solutions oriented? Well, the solution is simple but not easy to me. It's that we need to restructure the way we make products in a way which integrates research much more fully. It looks like consistent relationships in which researchers and the work and the insights they provide

Starting point is 00:14:48 are a part of the process from beginning to end. And like I think, Lenny, you as a PM, that's how you worked. You know, I remember you. Like, I know who you worked with. You worked with great researchers. but honestly, most product processes are not that way. And so that's when you like research is a service function. It gets called in right at the end.

Starting point is 00:15:08 It's reactive in the sense that a researcher in the room listening and participating in the conversation could have a ton of impact on framing exactly the right question that will drive maximum business impact, maximum product improvement at that moment, and then go do it quick and get back and we're on to the next. But they weren't there. The relationship wasn't there. they're not engaged in the project from the beginning, and that's the number one root of the problem.

Starting point is 00:15:33 As long as research is a service discipline, I think we're going to be stuck in this spot. When people might be hearing this, on the one hand, it's research has been not as helpful to teams as they thought, and researchers have been spending time on the wrong thing. On the other hand, your advice is integrate research from the beginning, make them more involved throughout, and I think that might confuse people.

Starting point is 00:15:54 How should people think about, like research is actually more important, and you should integrate them more deeply. There's a vicious cycle that's been happening, right? It is from where I sit. And this is what I hear from, you know, many, many researchers and research leaders, which is a lot of companies hired a lot of researchers with great intentions, didn't quite know how to integrate them, you know.

Starting point is 00:16:13 And research is a, U.X research is a kind of a newer discipline. So maybe that's not surprising. We're still learning how to use it. Cool. Let's evolve. But a lot of companies hired these people, but they hired them into kind of like a service discipline. very reactive, not in the room, not integrated in the way I said. And so they had less input on the questions to ask, or they're included, but only at the end, right?

Starting point is 00:16:35 And then they're unable to build those direct relationships to be there in the room to actually, like, drive the questions and insert insights. Because a good researcher is like the repository of insights you need for growth, but they're not there. They don't participate in the decision. So they end up doing research. They have jobs to do. So they do research that is too reactive. It doesn't matter. and then it doesn't, you know, it's less impactful.

Starting point is 00:16:58 Researchers kind of conclude that therefore researchers are not as impactful, and then they get sidelined or laid off, and the cycle continues. So I think the short circuit is the constant engagement. If you take a great researcher and you insert them consistently in a product process, I feel confident that researcher will drive product improvement, metrics impact, growth, all the things that you want to see as a PM and a product leader, that's the exception, not the norm these days. This may be a hard question to answer, but when people hear, if you have a great researcher,

Starting point is 00:17:35 here's how you approach it, what are signals that your researcher is great versus not great? What are some things people could look for it to tell them, like, maybe I have the wrong researcher in my team? So the best researchers, I think, are, first of all, multi-method. the first iteration of user research was primarily a qualitative discipline, but a strong opinion that I have is that is largely one of those models that needs to evolve. It's not that qualitative user research is no longer important. It's that the best researchers have five tools. I think they have five tools. And those five tools are number one, what we would call

Starting point is 00:18:11 formative or generative user experience research. So looking ahead, innovation focus, really open-ended, more ethnographic. Let's go out into the field and talk to host and guests on Airbnb. Let's see people using our product in the field, stuff like that. So that's formative. The second type is evaluative, right? So more like usability testing. The third tool is a basic rigorous survey design, right? It's the best scaled way to get responses from communities, small and large. You can get a lot out of really well-crafted surveys. But to do that, you have to have the fourth tool, which is applied statistics. The best research know a little bit of stats. You can't interact in a world of AB testing without knowing basic statistics. And then in the old version of this, the fifth

Starting point is 00:18:58 tool was SQL because I think good researchers need to be able to run their own queries. These days, so much of that is dashboarded that the fifth tool may now be prompt engineering, which is the thing we could talk about. But I think it's some, maybe that's the fifth tool. is it technical skills that fall in between querying your own data, understanding it very well in companies that are washed with data, and then interacting with generative AI. Amazing. That's such a cool list. Okay, so just to play back, formative, generative, innovative skills to think bigger

Starting point is 00:19:30 and come up with new ideas. Usability. Yep. Yeah, usability. How did you describe it? I have a different word here. Evaluate. Evaluative, right.

Starting point is 00:19:41 So we're evaluating products and doing more really. That's kind of the micro level. Yeah. Survey design being really rigorous about it, applied statistics, and then SQL slash dashboard slash prompt engineering. Right. Maybe just one last question along this thread. Also, big question, but any advice for how to evaluate these skills slash interview for

Starting point is 00:20:01 them? I know this is its own, like, deep topic, but any advice for someone trying to find this person? You know, I've interviewed hundreds or thousands of researchers. And the way I usually approach that is, you know, you want a researcher who's got a Swiss Army knife because if all you have is a hammer, then everything looks like a nail. And so if you give in the context of an interview, let's say, a researcher, a pretty juicy, open-ended research question, and you want to see how they handle it. And a good answer is usually multi-method, right? We're not going to handle it in any one way. We're going to say, well, here's a couple of ways we could deal with

Starting point is 00:20:34 this. Here's that we could do this in a day or a week or a month. I mean, we usually don't have a month, but sometimes big research projects go on for that long. And here are the different sets of methods that we can use. So see where they go. It's actually pretty simple. Most researchers are deeper in one than the other. And sometimes you can make up for those five tools with the team, right? So you have experts who are kind of T-shaped, but maybe deeper in one or several of those ways. But when I built a team at at meta and at Airbnb, that was my goal is individually as researchers build up those tools and then as a team build deep expertise that would fill all the gaps. Coming back to the main premise of your post, one of your big takeaway,

Starting point is 00:21:13 is researchers need to be much more business-oriented thinking about what helps the business versus the user, which I think to a lot of researchers will feel really weird. You can just talk about your kind of takeaways there. So much of user experience practice, not just research, but design too, is focused on empathy, right? And very user-centered. This is beautiful. I'm not saying that we should abandon that.

Starting point is 00:21:38 I think what I'm saying is there's an overlapping then where you have the user and profit or the business. And what researchers need to do is be way more explicit about finding that overlap. So one thing I often, when researchers ask for advice, they're like, well, what should I do to be more business or profit focused? I say something like, did you read the last quarterly report if it's a public company? Did you listen to the shareholder call? You know, and they're probably like, no, you know, it's full of a bunch of, you know,

Starting point is 00:22:11 language I didn't quite get. And I'm like, mm-hmm. Yeah. So there you go. That's the language you need to learn. Scour, scour your Google Drive folder, your internal folder, and look for all of the documents that are about this quarter or this haves or next half strategy. What are the OKRs? Understand the metrics and the funnel, the conversion funnel, like know it back and forward because then what you're doing is you're proposing, if you're in the active conversation, you're saying, cool, I hear you asking that research question. I've identified this is exactly the spot in the funnel where I think we need to do work, right? There's an, there's an opportunity here, or that competitor is eating our lunch with this group of users. Like, I know that because I read

Starting point is 00:22:51 the competitive report and I understand it deeply. So like, those are skills that many, some researchers have and a lot are building these days, but historically, like last 15 years, it hasn't been a thing we've been as focused on. And I think that's an evolution that needs to happen. I think a lot of PMs listening to this are going to be like, hallelujah. This is exactly what I've been trying to convince people of. It's what I've been trying to convince my researchers of. Design often falls into this. But Lenny, the opposite is true too. Like, because you got to take the average PM who lives in that land all day, every day. And what they do is not in the Venn. You know, I think those are people who are also like performing customer centricity and

Starting point is 00:23:34 performing user-centeredness a lot when they're really not interested. And so this is like not about researcher, this takes two sides. Fixing this broken system takes everyone, researchers, PMs, designers, everyone at a company, but also the way that organization is structured and integrating itself in a different way. Everybody's got to come to the table. Such a good point. And you have this actual term that you call user-centered performance, where the performance of being user-centered. Can you talk about that? And then just what advice you'd give to PMs that hearing this are like, yes, I love everything you're saying and then not realizing maybe they're too far in that extreme.

Starting point is 00:24:11 So user-centered performance is a term I made up because it's fun to make up terms. And it refers to customer obsession or user-centered practice that is symbolic rather than focused on learning, right? So it's hugely common, I would argue. It's work we do to signal to each other how customer-obsessed we are, not because we want to make a different decision. And like, if you're like listeners are like, I don't do that, I'm kind of like, think about it for a second, right? Because they're, this is extremely common.

Starting point is 00:24:48 It shows up in explicit ways and implicit ways. So explicitly, I would say every time a PM comes to a researcher at the end of a product process and says, can you just run a quick user study, you know, just to validate our assumptions? That's user-centered performance. It's too late to matter. that PM is not interested in being wrong at all. It's too late in the game for that. We got to ship it, right? What they want is to check the box.

Starting point is 00:25:16 So any check the box style research is a wild example of user-centered performance. I would argue every researcher has probably had to do executive listening sessions, you know, because a lot of PMs, founders, product people, but designers too, they want to get close to the customer, right? And so, like, can I do some focus, groups, I want to be there. I want to ask them questions. This is 97% performance. It's well intentioned, but it isn't focused on learning. It isn't going to drive better outcomes or more impact. And then there's all these implicit ways that people engage in that kind of user performance

Starting point is 00:25:52 too. A lot of it comes down to cognitive biases, confirmation bias, ego. Like one of my big kind of mantras was we don't validate, we falsify. Right? We are looking. to be wrong. That is the mindset you should use when you're approaching insights and research. I want to be wrong. I want you to do research that shows we were off base in the following ways. Tell me exactly how and why in a way that allows me to fix it quickly. But many PMs, many designers are not in that place. They do not want to be wrong. They're looking to validate. And that's user-centered performance. Oh, man, I think a lot of people are hearing this and feeling exposed. Exposed.

Starting point is 00:26:34 I feel like you're like this deep throat person coming from sharing these things people don't want to talk about. There's this quote in your post I'm going to read. Product managers love to ask for middle range research that they can use to justify decisions they're reluctant to make on their own. User designers love to ask for middle range research because it fits their model of what proper design process should look like. Executives love to ask for middle range because they don't really understand what research is for. It helps them do performative user-centric. In the end, they will decide based on their own opinions. There is an important place for intuition in product development.

Starting point is 00:27:12 Of course. Like the best designers, researchers, product people develop strong intuition for the product. But you got to understand, intuition is where all of those biases lie. It's where all your blind spots are. And what great insights people do, what great researchers do when you're next to them all the time is they'll expose you. I don't have to be the deep throat, right? Because you have somebody whose professional job is keeping you honest is probably the wrong way to put it. But, you know, somebody whose capabilities are about expanding your horizons, making it so that your intuition

Starting point is 00:27:48 is constantly improving. You don't have to rely on it when your intuition and the evidence sort of collide in a way that either affirms or falsifies the product decision you made and how something really good is happening. So, you know, and the other thing that is inherent in that quote is, you know, I at Airbnb wore many hats over the years. I was head of research two different times. I was head of design for guest products. And my last job was I was head of the design studio. So UX research, UX design, writing, localization, they all reported it to me. So I've seen this from many disciplinary angles in the UX field. And researchers aren't the only ones who are guilty of this. I would say design, it has a ton of performance.

Starting point is 00:28:30 And it comes from the fact that we have like figured out user-centered design, this process or design thinking, which IDEO popularized. Like, that's what we're supposed to do, right? Right. Bezos told us that we as PMs had to be customer-obsessed, right? So that's what we're supposed to do. It's a really common and damaging thing when we are not genuinely, like, we don't genuinely have that growth learning mindset.

Starting point is 00:28:54 and it's like easy to sideline researchers. Like we don't need them in that situation. We've got our guts. Isn't the gut where like a great PM, a great founder needs to have that gut? And they do. But they need to be open to the fact that your gut is limited and biased and narrow and wrong sometimes. The two sides of this is trust your gut opinion. I don't need research.

Starting point is 00:29:16 I don't need data. I have opinions and my own experience and I'm going to use the product. And let's just go with what feels right to me versus pure data-driven research. driven for designers that are maybe listening for product managers. Do you have any advice for just like where to fall on that spectrum and just how to best leverage research to inform that opinion? Yeah, I taught a class at UC Berkeley this semester on leadership and we talk about that a lot because, you know, great leaders develop intuition, right? That's part, it's the, it's the, it's the pattern matching part of experience, right? Where you're, you develop heuristics,

Starting point is 00:29:51 which allow you to make good judgments, even if you can't quote, explain where that judgment came from, right? That's what the gut is. But it's also, like I said, where bias comes from, where we're cognitive, like all the cognitive biases, there's a list of 151 of them on Wikipedia. I won't name them, but like all those thorny things that lead us astray, you know, the behavioral economists and social psychologists study, those live in the gut. And so the advice is, you know, when you are looking to check your gut, you have to do that thing. A lot of, a lot of your listeners have probably read Thinking Fast and Slow. System 1, System 2, right? I have it here right under my laptop, actually, holding out my laptop screen.

Starting point is 00:30:32 That's so appropriate, Lenny. So the secret is not that sexy. It's System 2, right? So you engage that slow methodical process in which you do analytic thinking as a means of checking your gut. Slow in the grand scheme of things, right? Slow meaning not a foot second decision, not like months of analysis. That's not what I mean. The other thing is. The other thing is, thing you can do, and there's really great research on this, is you bring in the wisdom of the crowd, right? So the wisdom of the crowd is a phrase a lot of people are familiar with, and it works in a specific situation. The wisdom of the crowd works when the people involved with the decision are bringing diverse sources of information and judgment to the table. Obviously, if

Starting point is 00:31:16 everybody has the same sources of information, then it doesn't matter how many people are out there. So if you want to check your gut, get a bunch of different guts together. Get a bunch of different guts together. get a bunch of different people in the room who can bring evidence and intuition to bear and have an open, like, direct and kind conversation in which we might disagree. You know who's great at that? Researchers. Leading those discussions essentially and getting a bunch of people's opinions. Yeah, this is the structural solution I'm talking about, Lenny, is like, I never asked for research teams to have their own separate OKRs. I said two things. Number one, what's the

Starting point is 00:31:53 teams, okay, like shouldn't the PMs, the engineers, the designers, and the research, everybody should have the same set of metrics for success, right? Because either we're doing it together or we're not. And then I said, my metric for success is when they won't have that meeting without you. That's my metric for success. If they cannot have that decision making meeting without the researcher there, that means you've developed influence, strong trusting relationships. You're an active participant in the process, not just sort of somebody, who provides input into someone else's process. And that is when researchers can have huge impact.

Starting point is 00:32:29 I think of the PM role in a similar way, even though people won't have these meetings without PMs because they're often at the center of a lot of the stuff. But you want to be a PM that people want on their team. There's a lot of teams that are like, we don't want any PMs, we don't any product managers. They're just getting away. And I find that that's only the case when the product manager is not great

Starting point is 00:32:47 and not really good at their job, because most great PMs just make everyone's life easier. They do. The grease. I love it. You mentioned also that before we start recording that the biggest challenge for these researchers is the relationship with their product manager. Can you speak to that and what you've seen there? I'm wary of overgeneralizing, but I can tell you that from my experience and from what

Starting point is 00:33:09 I hear, the product research or product insights relationship is one of the most challenged. And I think it comes from the fact that fundamentally many researchers are just not included in the process that that PMs are running. And then there, actually I kind of thought, I did some asking around before, before this podcast. And so I thought,

Starting point is 00:33:35 you know, there are some tropes that researchers have about PMs that are worth PMs knowing, right? Like, just like four or five of them, like the things that researchers know PM say, which drive us nuts because they're not true. So the first one is that research just slows us down.

Starting point is 00:33:56 Research is too slow. This is bullshit. A great research team can do research in a day, a week, or a month. It just depends on what you want to get out of it. You know, like if you, you know, how much detail do you need? How many people do we need to talk to? What is the depth or breadth? Do we need to go to seven different countries to talk about our constituencies in Latin America?

Starting point is 00:34:18 Well, that's not going to happen overnight. But we don't often need that. The other way to look at that is that is it slower to get it wrong and fix it than to take a hot second to do the work to get it right the first time? You know, so that's BS. Like, good research doesn't slow us down and speeds us up. And also just along those lines, like a big part of your premises, you don't need to do as much research as people are doing,

Starting point is 00:34:42 like this middle research that a lot of the time is put into. Yeah. Research can go super fast. I think especially, so the macro level research, I hope what it is is tied to things like annual planning processes. We did a thing at Airbnb several years that we called the, it was like Insights 2019, Insights 2020. They were concept car projects and we spent quite a long time synthesizing the entire

Starting point is 00:35:06 year's worth of insights from every place we could get them and then developing with designers and engineers like a concept car for five years in the future. So that's a long process. But the micro level, there's so much business value to be derived there. so much business value and it can go so fast, Lenny, right? It can go so fast. You can have, you know, you can have results in 48 hours on these things. Like, we did a thing at Airbnb. There's a famous story, which I'll only tell in the abstract because I don't want to out anything, but it's like the, we call it the multimillion dollar button. And basically,

Starting point is 00:35:40 we did research which revealed that people weren't going down the purchase funnel because they were afraid, the calls to actions on the button was making them afraid that it would initiate a purchase when really it was just taking the next step. We changed the text on the button with help from our amazing content design or UX writing team. We basically changed seven characters and made Airbnb millions of dollars, right? Because what we found out, it was really simple. It was just like, hey, this button feels scary. The CTA on the button feels scary. So that's a great example of how micro, and that happened in like 48 hours. We would discover that insight or 20, overnight, basically. And we were like, maybe we should test some other

Starting point is 00:36:26 CTAs. We did the conversion, like, we added like 1%, which is really, really hard to do. So that's a quick example of how that type of quick research can drive a huge amount of business value. This episode is brought to you by ATREFs. Many of you already know ATREFs as one of the top tools for search engine optimization. It's used by thousands of SEO. and companies like IBM, Adidas, and eBay. What you may not know is that there's a free version that was made with small website owners in mind. It's called H.RF's Webmaster Tools.

Starting point is 00:36:58 It's free, and it can help you bring more traffic to your website. H.R.F. Webmaster Tools will show you keywords that you rank for and backlinks that you can get. It also performs automated site audits to find what issues prevent your website from ranking higher on Google. Every detected issue comes with a detailed explanation and advice on how to fix it. visit hrefs.com slash awt set up a free account connect your website and start improving it that's a hrrFS.com

Starting point is 00:37:26 slash a w t so just to make this even clearer I think this middle research zone is the stuff that does slow people down I imagine it's like what are the challenges host have with payments on Airbnb like what you're basically saying is spend your time doing the micro stuff like usability research and then the bigger stuff, that's part of overall planning. That's part of the planning cycle. It's not like every project you're working on. You need to have like a whole research project on. Exactly. The micro research should be much more common. A lot of researchers think that they're,

Starting point is 00:38:01 that that's sort of scutwork, you know, that like usability is something junior researchers do. I completely disagree. Like I think we need to get back there as an industry and be like, when you make a product easier to use, when you, like, discover problems with functionality, business metrics we care about will go up. I've seen it happen, right?

Starting point is 00:38:20 But that's not just work for interns and new grads, that's for sure. And then the planning process, like, absolutely, if we're integrated from beginning to end, we can help. And, you know, the thing about that middle range, I think you're right. Like, that's the stuff that makes the sort of stereotype that research is slow. And a lot of times it's also because it's just not pointed enough. The researcher can also say in that moment, I have studied the business plan. I know exactly where, like, I've seen.

Starting point is 00:38:46 seen the metrics trend. I have an idea about exactly where that's going to go. We still need to do that middle range research. The question is valuable, but it's now very pointed and that the time is worth it. Amazing. Okay. I want to cure the rest of these strokes. Okay. So research is too slow as the first one. The second one, I can do my own research. Why do I need researchers? And like, that's true as product people, I hope. I hope, you know, you are engaging with customers and listening well. Well, but, no offense, garbage in, garbage out. You know, like the thing is anyone can talk to a user. That does not constitute research or insights work, you know,

Starting point is 00:39:27 because one user can be powerful, but one user can be idiosyncratic. And a researcher knows how to get to the heart of that really quick. They know how to take that conversation and understand and situate it in a way, which means, like, sure, like democratize research. That's happening. There are tools out there that will let anybody get customer feedback, voice a customer type stuff. But a researcher is there to help you turn garbage into something that's not garbage. And avoid the bias that can come from, like, you're just reaching out to your cousin's family

Starting point is 00:40:02 and then doing whatever they thought you should do to the product. So that's the second trope. The third one is A-B-Test everything, right? And like AB tests are great, but one of my most painful things to do is to sit in a room full of PMs and data scientists who have just seen the results of an experiment that like flipped to stat sig. And then they're like, cool, I was significantly down over this course of time for these users. And then they just start speculating about why that is. Because the AB test rarely tells you why it changed in the way it did. And then this endless flywheel of AB testing goes.

Starting point is 00:40:42 And I'm like, hey, you don't have to guess. I know somebody who can get you an answer or at least evidence that addresses the question of why did we see the test result we did in a very short amount of time. Or you could use your customers as guinea pigs and throw more experiments at them over and over and spend a long time on it and come to the same place in the end. I think a similar critique that PMs often have is ABB. B testing is conclusive, scientifically, statistically. User research is just talking a bunch of people.

Starting point is 00:41:14 Why would I trust that? What is your best way to help PMS realize that this is actually very valuable data? And you should listen to it. It's not just a story here and there. Yeah. No, I think they're both right. A.B. testing is as close as we can get to making causal claims about products. Research is usually not oriented towards making causal claims.

Starting point is 00:41:37 or it should not be. But those causal claims rarely tell you how and why things happen. And if you want to develop, if you want to not make that mistake again in the future, you need to know how and why. If you want to build a better product in a way that doesn't just answer this narrow question that an A-B test answered, you need to know how and why. And so you kind of need both. Like beautiful partnerships between data scientists and research and insights people

Starting point is 00:42:02 are I think what we're going to see in that next evolution. And if you set that up, that's sort of very very, virtuous cycle up, if you set up the engagement where those people are involved from the beginning, you don't make those mistakes. You get the causal relationship, which is valuable for one reason, and the hows and wise, which are valuable for other reasons. Awesome. Okay. I think there's two more tropes you had. So one of them is like a simple one, which is like everyone loves to quote that, that it turns out a totally apocryphal Henry Ford quote, you know, about if I'd ask my users, it turns out for the best of our knowledge, he did not say that. And he, yeah, I know, isn't that

Starting point is 00:42:36 sad. I know. Sorry to burst your bubble, Lenny. Oh, wow. Does anyone say anything? I feel like every quote is... Is it powerful now? I know.

Starting point is 00:42:46 What is reality? Geez, can we? Well, let's just... Okay, maybe he said that. Like, he certainly believed that. That's what the historians say. But the reason that makes researchers so angry is because that's not a research. That's not what researchers do.

Starting point is 00:43:00 A researcher who's going to ask customers what they want is a bad researcher. You need a different researcher. Like, that's not, I've never done that in my career. No one on my team has ever run a study that's like that. You know, so that just makes researchers mad. And then the last one is about post hoc bias. It's, we knew this already. That was obvious.

Starting point is 00:43:23 And I think a lot about this book, which I would recommend to your listeners. So the author is a sociologist at UPenn named Duncan Watts. And the title is, everything is obvious if you already know answer. And it's about hindsight bias. He kind of like makes the argument that we rely too much on intuition, heuristics and pattern matching in a way that sort of is inappropriate to our experience. And it's like it leads us astray. It's kind of like a form of self gaslighting. And it happens because we end up sort of selectively remembering things and then constructing narratives around them in a way which makes us feel like we already knew that when we infected not. And he talks about

Starting point is 00:44:05 this other, one of those cognitive biases called the narrative fallacy, which is, is the idea that people love to make convenient, simple stories about the past. Like, if I asked you about your career, Lenny, and how you got to be this amazing podcast host, you'd be like, well, let me tell you about the series of events. And like, we do that. You know, it's like part of how we make sense of our lives and the information around us, but it would probably be a lie, you know, like you, you know, in the sense that like we all kind of twist the evidence we have to fit the narrative. We want to be true, because it's simple and lovely, it makes us happy. This is going to sound self-serving, but I find I'm the opposite.

Starting point is 00:44:38 I'm like, I have no idea how this all came about. Here's some things that kind of happened and somehow I ended up here. But maybe I'm being very modest and tried to not give myself any credit. That's beautiful. Thank you for these tropes, by the way. This is fun. I didn't know you were going to do that. So that's a fun little collection we've got here.

Starting point is 00:44:55 I wanted to ask about, there's this tweet by Patrick Hollison that I've brought up a couple times on this podcast that I think is really interesting. and his tweet is this. In my opinion, the best product will stem from a very strong mental model of the domain and user. User research can help you get such a model and validate it along the way, but it's important to view the syllogism of UXR's model as of user research to improving your mental model of the user to what product you should build versus user research tells you what product to build. Does that resonate in any way? thoughts on that way of thinking about user research.

Starting point is 00:45:34 Yeah, I mean, there's a double-edged sword we talk about a lot in the research community, which is about making recommendations, recommendations for design, right? So the best research doesn't, like, leave it at that, right? It tells you, and it's like the what, the so what, and then the then what. But the problem with that is some researchers go too far in the other direction, where they're like, we ran the study, it yielded these insights, and therefore, this is what we should build. And everyone else on the team is like, whoa, whoa, whoa, right? Glad to hear your thoughts on the matter, but like, there's a lot going on here.

Starting point is 00:46:15 Maybe we should talk about it. And that makes perfect sense. That's like a failure of communication. And I think that it like speaks to the thing that Patrick is saying is like, good research can sometimes tell us exactly what the problem is and exactly how to fix it. An example of that is the multi-million dollar button I told you about. But in a lot of the bigger picture questions, especially the macro ones and maybe also the really pointed middle range ones, the point isn't really, this is exactly what we should do and this is exactly what we should build.

Starting point is 00:46:44 It is let us develop a framework, which is based on actual evidence, and then together as a team figure out how we want to experiment our way to a successful product. To close the loop on this specific thread, what is your advice? to teams, researchers to help move out of this kind of reckoning and to move forward and help the field, both from a user researcher perspective and also from just like a company that maybe laid off a bunch of user researchers or is trying to decide what to do with the researchers. Thank you for asking. I think I said to you earlier, like, you know, and I feel some pressure as maybe the first conversation that you've had specifically about research on this podcast.

Starting point is 00:47:23 Yeah, I think so. And I want to help. I believe so much in this discipline. of research and insights. And I think there's the, you know, when I said the user, the UX research discipline of the last 15 years is dying, I didn't mean that I think research is dying. Far from it.

Starting point is 00:47:40 I think that there's a version of it, which we're now moving past and into a new version. We're going through an evolution as many do. And so the question for me is like, how can researchers and the companies and the other people with whom they work create a new version,

Starting point is 00:47:54 a different version in evolution, which is hugely impactful for the business. And so the advice I'd give, to researchers about that is develop diverse research skills, remembering like the five or five and a half tool list that I mentioned earlier, really go deep on that business knowledge. So speaking the language of product and business and metrics and understanding exactly how to use your insights like a scalpel,

Starting point is 00:48:20 building those strong relationships, which is not a thing that researchers can do by themselves. It requires two-way engagements. and also in a way which allows researchers to do fewer things better. So most researchers that I know are working on teams where they're like, I'm the only researcher and there's like, I have 7 PMs and 20 designers and like I'm trying to do 10 projects. And no one's going to do a good job that way.

Starting point is 00:48:43 So researchers have to learn with their partners about how to say no and focus on the most important things. But that's like only half of it, right? That's the research side. I have like two thoughts about about what companies should be doing. The first one is, it's a little bit of an aside, but not really. Like, one thing I learned through the responses to the article was everybody came out of the woodworks from the variety of insights disciplines that are out there.

Starting point is 00:49:08 Because I come from a tradition of user experience research or user research, but there are many insights disciplines in many industries. And they all wanted to claim one type of research or another and say like, oh, well, we over here in consumer insights or market research have been doing that well for years. And, you know, there are many insights disciplines. And generally, I think creating silos is stupid. Actually, I'm curious what you think, because here's the number one thing I heard when I joined Airbnb and you were there is I did a like a quick listening tour where I talked to a bunch of product people and they all kind of said the same thing. They were like, listen,

Starting point is 00:49:45 we have all these different people throwing insights over the transom. And it's great. We want to hear from the data scientists, from the product specialist, from the customer service people and the voice of the customer, whatever, all that stuff, but they're all coming over the side and we don't know what to make of it, right?

Starting point is 00:50:01 It's like too much. And that as much as anything is an argument for companies to stop siloing research disciplines. So when I joined Airbnb, I set out to create an integrated insights function where it's like, let's do UX research,

Starting point is 00:50:14 let's talk about the market and competitors when we have to. Let's integrate smartly with data science functions. Let's integrate all the stuff we're getting from customer service feedback, you know, we brought over what was then the NPS program and sort of said like, hey, like if we're getting customer feedback there, let's all just like use it all to fuel this one Insights machine.

Starting point is 00:50:35 So that's the first piece of advice like your companies. And the second one without being a broken record is to think differently about the broken cycle. So integrate researchers into a unified lean process. So if the researcher is not there from beginning to end, if there are not strong relationships between product people and design people at every level, engineering people at every level, and somebody who's their partner, their insights partner, we're going to fall back into this problem

Starting point is 00:51:03 where we're just a service discipline. We're not extracting the maximum value. It comes too late. We don't know what questions to ask. We're ignorant about what research can do. And so creating that integrated lean process where a researcher is arm in arm from the beginning is the most important advice I'd give.

Starting point is 00:51:19 That last piece may be the answer to this next question. But the question is, how can product managers be better partners to user researchers slash take more, get more leverage out of user researchers? I think that is in many ways the answer, making sure that they are creating a process for the product, for their products that integrates user researchers and insights from beginning to end. Also, being willing to partner with the research on the ruthless prioritization, I used to say that a full plate for a researcher was probably

Starting point is 00:51:51 three things. Two big projects and a small project, right? Like a side project. More than that, your researcher is probably not doing a very good job. And a project may take 48 hours. That's okay. But so they need your help to prioritize. They need you to participate. Great PMs will take the time to be with researchers to go into the field, even to travel. Did you ever do that, Lenny? I did. I went with Luis, who introduced to, like, came up with this, basically told me to chat with you about this topic. Thanks, Louise. Thanks, Louise. She, we did a whole tour to Paris. Our whole team, or the whole leads of our team went to Paris to do a bunch of focus groups and a bunch of research behind like actual mirrors. I've never done that before. Yeah.

Starting point is 00:52:36 Trip and it was amazing. Yeah. Can I tell you a quick story about behind the mirror? So this is back from when I was at Facebook. And it was the high times there was like 2012, 12, 13, and newsfeed is like really taking off. Ads are going into newsfeed. And I was working, I was the leader of a team that was working, among other things, on how to address post quality, right? Like, how do we think about what's a good post and how do we get feedback about it? And so there was kind of a team of engineers that thought,

Starting point is 00:53:06 one thing that you can do on Facebook is hide a post, right? So they were like, this is easy. Let's look at the posts that are hidden the most and use that as the signal. of what's a good post on on Facebook. Seems kind of reasonable, right? And like something tripped me on this one. And so I was like, so I did two things. So the first thing I did is I looked at the distribution of hiding by user

Starting point is 00:53:32 and found out that it's power law distributed, like everything on the internet. There are a few people on Facebook who hide a ton and then most people don't hide at all. And so then what we did was we said, let's find, we called these super hiders, we called them super hiders. And so we said, let's find super hiders around the office and we'll get a super hider in, and we'll do like a really traditional user interview. I just want, we just want to see. So literally the first person who walked in, I remember because this is a person who had those finger nails that are so long, you don't know

Starting point is 00:54:06 how they can do cup touch screens. But they did. They were amazing at it. And it was one of those rooms with the glass. And I insisted that the end, directors, the product people, and they were willing, whatever. So everybody's behind the glass, and I'm there with them. And the excellent researchers in the room, and they come in, and we're just doing a traditional think-aloud study. And so they, hey, can you open up your Facebook app? We would just love to see you, you know, what your experience is like. And so they, they opened up Facebook and we're looking, and they look at the first story, and they hide it. And they go to the second story, and they hide it. And this went on for a while. And like, she's definitely using Facebook,

Starting point is 00:54:45 but every time she'd finish with a story, she'd hide it. And the people in the back room were starting to chatter and they're like, wait, what? What is happening right now? And like good research that the good researcher that this person was, like they let it continue and they're like, oh, can you tell me what you're thinking right now? Come to find out that she was like, well, I hid that story because I'd seen it already. The model she was going for was inbox zero, which was like sad because it was infinitely scrolling. never get there. And the reason I like that story is because the people in the back room had their

Starting point is 00:55:23 minds blown. It was not that we assumed that was common behavior. Like, this person could have been unique. But it was enough because those people were there experiencing the research, the end of one allowed them to burst their own bubble and realize, okay, we can't think so naively about hides as a signal anymore. And we came up with a better solution. That is an awesome story. And such a good example of you don't need statistical significance to get massive insights. Like one example just gives you a, wow, this might be exactly what's happening. Let's go validate that versus like we are confident 100% this is what happened. I love that. It reminds me actually in the mirror in the mirror study that I was talking about in

Starting point is 00:56:08 Paris. There's a Facebook element to it too. We're trying to convince hosts how to feel more comfortable accepting guests who are booking instantly. And one of our theories was if they were connected on Facebook, they would be more comfortable letting someone book instantly. And we're just like, hey, what if you were to connect Facebook and see if they're friends? And everybody in Paris was very afraid of connecting and giving Facebook any data way ahead of what the U.S. house were feeling.

Starting point is 00:56:34 Yeah. So it just made it very clear. Nobody wants to actually give Facebook any data. So it was very anti-Facebook at that point. Yeah. That's so interesting. Germany and France were always are like bellwethers for what the rest of the world would be thinking with private data privacy concerns. Oh, man. Okay. A couple more things. So a lot of this started with a lot of layoffs within user research. And I think between the lines, there's a sense of teams don't need as many researchers as they hired during the ZERP era. I think a question in everyone's mind is just like, how many researchers do we need? What is a good ratio? I imagine there's not a simple answer here.

Starting point is 00:57:11 but just what's your general advice to companies of how many researchers is right? So this is the thing I've thought a lot about, especially in my role as the head of the design studio. That was like my fundamental question. It's like you have all these writers, designers, researchers, like how do you structure them, how many, and where and who works on what? And the organizing principle for me was always relationships. You know you have enough when the people who need to have a constant research partner have them. And I would much rather create pain in that situation than spread someone too thinly. So my, advice was always like, don't try to create a researcher to cover this entire product space.

Starting point is 00:57:46 Pair a researcher up with, you know, somebody who's going to involve them in a consistent engaged process and let them go to work and see the impact they're going to have, but protect their time. And then other people are like, wait a second, that person's doing great work. I want some of that. And creating that pain for them, like, because it's a pain of loss, like, is the number one way to grow headcount. That's how I always approached getting more headcount, was not.

Starting point is 00:58:11 arguing abstractly for why research is important, but by asking partners who wish they had it to do the arguing for me. And so there isn't a, you're right, there isn't a clean answer for like, hey, this is how this is the right ratio? Because it really depends on the nature of the product. Like is it a, is it a early stage product? Is it a late stage project? Are we talking about a startup or a late stage company? But, you know, I would argue there's always room for a researcher. Lenny, I'll tell you, and I use this in a keynote talk I gave lately, you published recently a list of, I think it was about 20 B-to-B companies and their first 10 employees. Do you remember doing that?

Starting point is 00:58:51 Absolutely. Do you remember how many researchers are anywhere on that list? I'll give you a hint. It's between zero and two. It's one. There's one researcher on that list anywhere. Anywhere. And that's messed up to me.

Starting point is 00:59:06 Now, look, it's just these 20 companies and each is in their own space. So I'm not going to overgeneralize, but there is an in a researcher can drive incredible value no matter what stage a company is at because a good researcher makes you go faster, not slower. And they drive impact because they answer questions which are impossible to answer in any other way. That's true if you're a startup. It's true if you're a late stage company. Now, if it's your first 10 employees, you know, like one one researcher is going to go a long way. You know, as you grow, making sure that you're matching up researchers so that they have strong partners. and the key parts of the business is the best way to figure out if you have enough.

Starting point is 00:59:44 Interesting. So your advice is as you're starting a company, your pitch is that you will have a lot more leverage and move faster hiring a researcher versus generally in engineers, but you'd be trading off. Essentially, that's what most of the hires end up being. I'm reluctant to overgeneralize, but I would say, and I would say I know in many founders who are in startup mode

Starting point is 01:00:06 are like, I know what I need to build. the problem is that I need people who can help me execute. And I think that's right. And so everything's a trade-off. But remember, imagine that you could have that Swiss Army knife at your disposal. As you, maybe you've got like an MVP out the door and you're looking to make your first major iteration or like many startups you need to pivot. Like this is where, like, it's like, hey, you don't have to do that alone. Like we, we deify startup founders who pivot appropriately.

Starting point is 01:00:36 But I think that is what we might call moral. luck where like we deify the ones who got it right and even though they made exactly the same decisions as the one who got them wrong. And the fact the matter is, like, if you have an insights person with you who has that Swiss Army knife of tools, like, you're not in it alone. You don't have to guess. Like, ultimately it will still come down to a tough decision that you and founders have to make, but you can have evidence that bears on that decision, which you wouldn't be able to get any other way. To close out on this and have just a couple more questions on this thread. I think one of your big messages to researchers is you can be empowered.

Starting point is 01:01:15 It's up to you to do the right sort of research and to move your career in the right direction, not become a researcher people than you. And there's this quote that you have at the top of your post where a lot of their reaction, or I guess the way you put it is, I know what you're thinking. They just don't get it. We're so misunderstood. Our plight is to deliver insights that users used to drive business value while we're forgotten. Never driving the roadmap. No seat at the table consistently miscast only to be laid off in the end. And what I'm hearing from you is, like, you can change that. Like, you can push back on doing research that isn't actually contributing. But let me ask you, what's your kind of lasting, I don't know, advice you would leave researchers

Starting point is 01:01:53 with to be successful? Yeah, it's tough to be operating in a broken system. And so I feel that response where you're like you feel kind of powerless. But I think that's not likely to lead us past this moment to the next evolution of research. So that's kind of where it's like, I don't blame any researcher at all for being in the spot they're in. It's been a tough go, right? However, crying about our lot, you know, is like not going to get us anywhere. So like, I think the point of the article for me,

Starting point is 01:02:24 and this is advice I give companies all the time when I do consulting with them, it's like, hey, we can set this up in a different way, which responds to the current environment in a way which will drive a huge, huge amount of impact. Now, that takes companies making the right choices. It also takes researchers owning up and developing skills, right, pushing back, understanding what research can have the most value, developing the skills and the knowledge and language around the business, like, becoming more influential, being excellent communicators. Like, it's one of the things I would evaluate the most in hiring, especially research leaders, because I needed them to show and

Starting point is 01:03:00 teach by example, is like, can you, if, like, it isn't just rigorous, research, it's like if a tree fell in the forest and no one was there to hear it. You know, you need to communicate it effectively and you need to do it in a way that's appropriate to the audience. Because if I'm talking to you, Lenny, it's different than I'm talking to Brian Chesky, you know, at Airbnb. And so I got to be able to give that presentation effectively and get right to the heart of it and speak the right language. And so if you're a researcher, it's not hopeless. Actually, the discipline, the future is so bright and we can help it along by continuing to develop these different skills

Starting point is 01:03:34 as companies build a model that's more inclusive. Awesome. Okay, I have one just random, tangential question about NPS. You have strong opinions about NPS, and I just wanted to hear your perspective on the value of NPS, your experience with NPS. Yeah, I do have a strong opinion about NPS. I like to say NPS is the best example

Starting point is 01:03:57 of the marketing industry marketing itself. And I think I'm going to... The problem is this, threatens many people's livelihoods because there's an entire industry of consultants and software providers that want you to believe NPS is a useful and accurate metric. The problem is, so the consensus in the survey science community is that NPS makes all the mistakes. So it's a garbage in, garbage out problem. So the likelihood to recommend question is bad for a whole variety of reasons. So it's bad because it's a zero to 11 scale. It's bad because it's usually

Starting point is 01:04:30 unlabeled. So we label the polls, but that's not the gold standard for research. It's bad because it's 11 items, right? And there's a couple of problems with that. Number one, we find that precision goes down after five items on average, maybe seven. Number two, especially on mobile, if you're taking this survey, what percentage of those options are below the fold? Right? We are not going to get accurate survey data, right? And so from a survey perspective, it's really bad. There's also this intuition, which is like, are you a person who will let, you know, like, are you, how likely are you, how likely are you to recommend Windows 11 to your friends and family? And not a person who goes around recommending operating systems. The question is fundamentally flawed. The argument is that's the,

Starting point is 01:05:12 that that question is a good indicator of loyalty, but there's a really simple solution. Any customer satisfaction, a simple C-Sat metric is better. It has better data properties. It is more precise. It is more correlated to business outcomes. I wanted to prove this. This is something that survey scientists know and marketers don't want you to know. And so we did the work with Mike Murakami, who led survey science at Airbnb and he's still there. Great researcher. And we basically redid all that work to find out if all that stuff was true just for Airbnb. And it is. It's simple. Don't ask NPS. Ask customer satisfaction. And the customer satisfaction question, what's the actual question for people Overall, how satisfied are you with your experience with Airbnb?

Starting point is 01:05:57 Or it could be some version of that, which is like, overall, how satisfied are you with your experience with customer service when you had a problem? So there could be a more specific version of that question. But those questions have better properties. And, you know, a lot of people say, well, hey, everybody's using NPS, right? So at least it gives me a benchmark because I can compare my NPS to industry NPS. The problem with that is the research shows that NPS is idiosyncratic. So it goes up in dows in ways that we don't understand, and there's a lot of inconsistency in how it's asked that creates variations in the data, which means it's not apples to apples.

Starting point is 01:06:32 So you can't even compare your MPS meaningfully to somebody else's. I love these hot takes. I'm curious to see who comes out of the woodwork too. People are going to be so mad, Lenny. I love that. I think, yeah, people, like, I've heard this many times and people don't talk about it. Okay. Is there anything else you want to share or leave people with before we get to our very,

Starting point is 01:06:52 exciting lightning round. Can I, yeah, I want to, I want to add one thing if I could, because this has come up on your podcast a few times recently, which is about the idea of people doing their own product walk throughs. Like, so should a PM just rely on their own dog fooding of the product and their own walkthrough to figure out how to fix it? And a couple of times recently this has come up like this, And I think the consensus seems to be sort of yes, this is a good thing. And I have a contrarian opinion there too, which is that I think it is really important for everyone to dog food their own products. The problem is related to relying on your intuition about those products, which is the thing

Starting point is 01:07:36 most PMs have trouble with is realizing you are nothing like the user. You are nothing like them in ways that will bias the way you think about what's good and bad in your product in ways that you can't. necessarily recognize. Like some things with a product, some problems with a product, you need a pulse to recognize. And like most good PMs that I know have a pulse and so cool. But a lot of them require like context of use, priorities, constraints that you just don't have. And you can't imagine purely on the basis of your own usage. So what I think that means is that you should definitely dog food your own product.

Starting point is 01:08:19 Doing product walkthroughs to identify lists of potential issues is a great thing to do. Prioritizing that list, figuring out which ones are more or less a problem and for whom is an area where you should be extremely wary of relying on your own sort of opinion, expertise, or intuition when you are dog fooding your own product. Thank you for sharing that. It's definitely come up a bunch on this podcast, so I think that's an important lesson for people to take away. Anything else before we get to a very exciting lightning room?

Starting point is 01:08:48 I appreciate you, Lenny. Thanks for having me on. I appreciate you, Jed. Well, with that, we've reached our very exciting lightning round. Are you ready? I am ready. What are two or three books that you recommended most to other people? I recently read a book by a business book by Barbara Kellerman called Bad Leadership. And what I love about it is that we spend a lot of time talking about good leaders. And she really dives into like the worst leaders and what makes them bad leaders in a way that I think is really valuable for everybody. I'd also recommend, I read a lot of fiction. So two recommendations there.

Starting point is 01:09:22 One, recent Pulitzer Prize winner, Demon Copperhead by a Barbo Kingselver. It's like an outstanding read that also is like really sad and moving and illustrative, especially when you're, if you want to understand rural poverty. And then completely other side of the fiction spectrum, if you're interested in science fiction, which I am, read the murder bot diaries. It's like about a sarcastic robot, a sarcastic killer robot and who doesn't love them. I love these fiction recommendations. I feel like we need more of these on the podcast.

Starting point is 01:09:51 Yeah, everybody goes to business books. Yeah, absolutely. What is the favorite recent movie or TV show that you really enjoyed? We recently watched The Last of Us and it blew our mind. I watched it after I played the video game after long last. If you were a person who plays video games and you haven't played The Last of Us, if you don't know, the show is based on the video game, the other way around. Do you have a favorite interview question you like to ask candidates that you're

Starting point is 01:10:17 interviewing? Think of a topic that you had to explain lately that was the most complex and then explain it to me like I'm five. And there are a lot of ways to vary that question. But the reason I like it is because I think, and I've asked this question to, you know, VP and C-suite candidates in like multiple disciplines. And sometimes it's related to a conversation. Like I might ask them to explain something complicated about quantum computing or music theory or it could be a complex business decision. But I want to see if somebody can break a complex problem down in a really simple way and give me an intuition for it in a short amount of time. I think that is a differentiator between good and great for many people. Do you have a favorite product you recently discovered that

Starting point is 01:11:04 you really like? Yeah, this is a really weird one, but my whole family started indoor rock climbing recently. And there's a challenge you have when you toprope, which is that you're looking up all the time. So they make these glasses, which are called belay glasses, and they have an angled mirror embedded in the lens so that you can look straight ahead. And the view you see is up towards the person who you're belaying. And I just thought that product is like so perfect for that. Like that, that's a niche problem and there isn't a better way to solve. Do you have a favorite motto that you often come back to, that you share with friends, either work or in life, they find useful?

Starting point is 01:11:44 Yeah. So this is going to seem like pandering, Lenny, but I don't know if you remember a conversation that you and I had. It must have been eight years ago. I remember where we were sitting, and it was about stoicism. Do you remember this? Anyway, we had this conversation. I don't, but I was into stoicism for a while.

Starting point is 01:12:01 I know you were because we talked about it. And so the motto is, comes from stoicism, which is basically, focus on the things you can control and ignore the rest. And a lot of people think of this as a lot of people think of this as the serenity prayer or the serenity saying. That was a 20th century invention, but Epictetus was writing about this, you know, BC. And I think about it all the time. So much of the stress and pain and worry that we have in life comes from things we can't control.

Starting point is 01:12:31 So I try to let those things go. Amazing. I learned that lesson from seven habits of highly effective people. and just the importance of thinking about these circles, if you can control, you can influence, and you have no control over. There's no reason to think about those other things. Absolutely.

Starting point is 01:12:47 Jud, this is everything I hoped it would be. We got into some really good stuff. I'm excited to hear how people react. Two final questions. Where can folks find you if they want to learn about what you're up to you? Actually, share what you're up to these days and how people can find you. And then also, how can listeners be useful to you? Yeah.

Starting point is 01:13:02 Thanks for asking those questions. So people can find me at judd-antin.com. That's the best way to find out what I'm up to. These days, I'm a consultant. I help people with Ux Strategy, org design, and crisis management. Somehow, I love dealing with other people's dumpster fires, and I've found that I'm constitutionally good at it somehow. So juddanton.com is the place to find out.

Starting point is 01:13:23 I also write a medium post that you can find at One Big Thought.com, and you'll find a lot of the topics we talked about today, including the original post that started this at One Big Thought. If there's one thing I could ask your listeners to do is to get next to your researcher. You know, like I just, I just think if you build those relationships and involve a researcher and insights person early and often, beautiful things will happen for you and for the business. So that's my, that's the thing everyone can do for me. I love that.

Starting point is 01:13:56 I've always done that. I loved my researchers that I've worked with many of them reporting to you. And so beautiful takeaway. Judd, thank you so much for being here. Lenny, thank you. It's been a pleasure. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.

Starting point is 01:14:18 Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny's Podcast.com. See you in the next episode.

Lenny's Podcast: Product | Career | Growth - The UX research reckoning is here | Judd Antin (Airbnb, Meta)

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.