Lenny's Podcast: Product | Career | Growth - Unpacking Amazon’s unique ways of working | Bill Carr (author of Working Backwards)

Starting point is 00:00:00 As Jeff would say, we took it as an article of faith. If we served customers well, if we prioritized customers and delivered for them, things like sales, things like revenue and active customers and things like the share price and free cash flow would follow. So therefore, when we're making a decision, thinking about a problem, we're going to start with what's best for the customer and then come backward from there. That informs like what's the work you have to do to then create this new solution for customers.

Starting point is 00:00:33 Today, my guest is Bill Carr. Bill is the co-author of the book Working Backwards, which is a synthesis of the biggest lessons that Bill and his co-author learned from their many years at Amazon. Bill joined Amazon just five years after it was founded, stayed there for 15 years where he worked on the book's business, and then as VP of Digital Media, launched and managed the company's global digital music and video businesses, including Amazon Music, Prime Video, and Amazon Studios. After Amazon, Bill was an executive in residence at Maveron, an early stage VC firm, then chief operating officer at Offerup, and these days Bill runs a consulting firm called Working

Starting point is 00:01:11 Backwards LLC, where he and his co-author, Colin Breyer, helped growth stage and public companies implement the many practices developed at Amazon. In our conversation, we go many levels deep on how to actually implement a number of the practices and ways of working that helped Amazon become the success that it is today, including the process of how to actually work backwards, how to organize your team with a single-threaded leader, how to divide up your metrics into input and output metrics, how to practice disagreeing and committing, how to implement the barraiser program in your hiring process, and so much more. Huge thank you to Ethan Evans for making this episode possible and introducing me to Bill. With that, I bring you Bill Carr after a short word

Starting point is 00:01:55 from our sponsors. Today's episode is brought to you by Assembly AI. If you're looking to build AI-powered features in your audio and video products, then you need to know about Assembly AI, which makes it easy to transcribe and understand speech at scale. What I love about Assembly AI is you can use their simple API to access the latest AI breakthroughs from top-tier research labs. Product teams that startups and enterprises are using Assembly AI

Starting point is 00:02:21 to automatically transcribe and summarize phone calls and virtual meetings, detect topics in podcasts, pinpoint when sensitive content is spoken, and lots more. All of Assembly AI's models which are accessed through their API are production-ready. So many PMs I know are considering or already building with AI, and Assembly AI is the fastest way to build with AI for audio use cases. Now is the time to check out Assembly AI, which makes it easy to bring the highest accuracy transcription plus valuable insights to your customers, just like Spotify, Callrail, and Writer do for theirs.

Starting point is 00:02:55 Visit assemblyaI.com slash Lenny to try their API for free and start testing their models with their no-code playground. That's assemblyaI.com slash Lenny. This episode is brought to you by Coda. You've heard me talk about how Coda is the doc that brings it all together, and how it can help your team run smoother and be more efficient. I know this firsthand because Coda does that for me. I use Cota every day to wrangle my newsletter content calendar, my interview notes for podcasts, and to coordinate my sponsors.

Starting point is 00:03:26 More recently, I actually wrote a whole post on how Kota's product team operates, and within that post they shared a dozen templates that they use internally to run their product team, including managing the roadmap, their OK-hour process, getting internal feedback, and essentially their whole product development process is done within Kota. If your team's work gets spread out across different documents and spreadsheets and a stack of workflow tools, that's why you need Kota. CODA puts data in one centralized location, regardless of format, eliminating roadblocks that can slow your team down.

Starting point is 00:03:58 Cota allows your team to operate on the same information and collaborate in one place. Take advantage of this special limit of time offer just for startups. Line up today at coda.io slash Lenny and get a $1,000 startup credit on your first statement. That's coda.io slash Lenny to sign up and get a startup credit of $1,000. $1.0.0.Lenny. Bill, thank you so much for being here and welcome to the podcast. Thanks, Lenny. Thanks so much for having me. Pleasure to be here.

Starting point is 00:04:32 It's my pleasure. So I was reading your book and something that I recognized as I was going through this is just how many new ways of working Amazon contributed to the way tech and business runs. And I made this little list. And I'm curious if there's anything I'm forgetting with obvious. So obviously, the idea of working backwards, the idea of one way and two-way door decisions, the concept of disagreeing and committing, input and output metrics, using memos versus decks, there's just the idea of two pizza teams, and then I know

Starting point is 00:05:02 that evolved into single-threaded leaders. Is there anything else that just like an obvious core thing that's maybe almost too obvious that I don't even think about that Amazon contributed? The one that's sort of non-obvious and is really the way in which Amazon created a set of leadership principles that were very real and the way in which Amazon created a set of processes to reinforce them. And this is, I think, I certainly haven't encountered anything quite like that. It was very intentional. So that is also a distinctive element of it that we try to point out in our book. Awesome. Okay. So maybe we'll come back to that because that is also a really powerful mechanism. So the question I wanted to ask about this is there are companies that are

Starting point is 00:05:50 bigger than Amazon that are more successful than Amazon that have been around longer than Amazon, but I don't think any other company has contributed so many unique new ways of working and also been able to coin them into such shareable ways. What would you say it is about Amazon that enables this sort of way of working and also just making things so just proliferate through the culture? That's actually one of the reasons why Colin and I set out to write our book because everyone knows about Amazon as an innovative product company, at least certainly during the time I was there, which was from 1999 through the end of 2014. The company rolled out all kinds of innovative products. The Kindle, AWS, Alexa, Echo, Prime, the Prime subscription itself is innovative.

Starting point is 00:06:39 and all those things, by the way. Yes. A lot of people found the world to use all those things. And obviously, Jeff was a huge driver of those things. But what people don't realize is that Amazon was actually, to some degree, equally focused on process innovation. In many cases, by the way, we stood on other people's shoulders. We cannot take credit for having, for most of these.

Starting point is 00:07:04 There were other inspirations or we built on work that other people's. others had done, which by the way, was what I think all great companies should do. And again, that's also why we wrote the book was because we would like to allow people to stand on Amazon's shoulders to learn what we learned and then take all or part of these things and, you know, build from there. But to sort of more directly answer your question, how or why did this happen? So this period of both product and process innovation actually occurred in this one narrow window of 2003 to 2007. During that window of time, all the products I just mentioned

Starting point is 00:07:47 and all of the processes except for one were all developed in this one four-year period. Wow. And this is the period actually where we were going from hypergrowth stage, you know, zero to one company to what I would call, you know, one to whatever, a thousand infinity, you know, that next step that companies have to make where what happens is you become, things become very complex.

Starting point is 00:08:15 We're no longer just a bookstore. We sell a lot of things. We actually branched out beyond just a retail business. We had a third-party marketplace business. We were experimenting in those days with providing, you know, running websites for third-party retailers in those days, too. We were developing new things. We were in many countries around the world.

Starting point is 00:08:36 So we've become, you know, very complex. And what happens to that point is that then, you know, you reach this point where the CEO can no longer be in every important meeting, can no longer be involved with hiring every person. And you need a system, a method to run the company effectively. And Jeff Bezos is, you know, fundamentally he's sort of a very scientific and analytical thinker. you know, his undergraduate degree was in computer science, I'm pretty sure, although I think he actually started off wanting to get a physics degree. He ended up moving over to computer science. He spent his early days at D.E. Shaw as a, you know, a quant on Wall Street, very quantitative mind. So he applied this kind of when he thought about this problem, he said, well, I need to be scientific about this.

Starting point is 00:09:29 There needs to be some system or some approach, some mechanism for me to be able to manage. such a company. So I'm going to experiment, like a scientist would, with, you know, different ideas, different hypotheses, implement them and see what works and iteratively improve. So that was kind of the mindset, which we took, which by the way, applied both to process innovation, but also, you know, product innovation. Awesome. I had Eric Rieson, and he also happened. I thought about this at the same time. He contributed a lot of core concepts to the way tech work. And he actually brought up a couple concepts that were in the cutting room floor, basically things that he thought would be like things people to adopt everywhere. And I'm curious, is there an example that at Amazon

Starting point is 00:10:13 where you built the process and had this clever term for it and just never spread or never actually worked at Amazon, anything come to mind? You know, the dev team, the design team, the product team, they're all in one group. And they'll go operate autonomously, but not completely autonomously because we, the senior leadership team, Jeff and the S team, want to know that they're on the right track. So we're going to create something called a fitness function, which was, let's figure out what are the four or five or six metrics that matter most for your particular area. Let's give a weighting to all of them. And then let's create an index for those. And we'll measure that index up and down. And that's the fitness function. That is a very nerdy way of organizing teams.

Starting point is 00:10:54 I love it. Yeah, super nerdy. But we realized, um, after, you know, I don't know how long, several months or a year of doing this, that the fitness function was not a good idea. This is what I would describe as a compound metric, where you try to take several important metrics and munge them into one. The problem is it actually becomes totally meaningless. When you're measuring things, you're trying to understand, like, what actions or reactions are creating, you know, the good outputs that you want,

Starting point is 00:11:25 revenue, customer growth. but by putting them all together, you basically obfuscate that. And what really we realized is we did is just break each one of these out individually and manage them each in its own way. So today I discourage teams and companies from creating any sort of compound metrics. I've done that once, and it was a terrible idea as well, where we had six different metrics. And every quarter, we were going to move a different metric that contributed to a higher metric.

Starting point is 00:11:52 And what we realized is we just never learn how to get good at one thing. and then it turns out there's always one thing that actually impacts the bigger goal most, so you just end up working on that thing anyway. Let's actually go deeper into the single-threaded leader piece since you mentioned it. It's actually come up a lot on this podcast of people working this way, where they have a single-threaded leader. And so clearly it's worked. And I guess let's just help people understand.

Starting point is 00:12:17 What does a single-threaded leader actually mean? And then why is it such an effective way of working? So the concept of single-threaded leadership was first born, you know, was born from this time of complexity at Amazon and where, again, large, you know, once you get to a certain scale, you get to a point of where there are competing departments, competing interests, and they are competing for some centralized pool of resources. For all of you who are working for a tech company, this is this pool of engineering resources or today, data science and AI resources. There may be other constrained resources often design as a constrained resource. But the point is now all these teams want that pool of resources to go build stuff for them, but they're competing in competition with each other. So most companies solve this by having an intense, centralized, highly collaborative process.

Starting point is 00:13:12 We decided to go in the other direction, for the reasons I mentioned, which were that we're just fine, spending all our time in these meetings planning, and a lot of the work we were doing, the artifacts we'd create, the documents, the projections were actually not very useful either. We're kind of bureaucratic time wasters, largely because a lot of the assumptions built into them were deeply flawed. So you're debating numbers in these documents that are based on flawed assumptions, which is a waste of time.

Starting point is 00:13:43 So what we realized instead was how do we get, you know, the three things we really wanted were ownership, speed, and agility. And so we experimented with that and said, let's create teams that can stand alone, where there's a single leader and the cross-functional resources that they need are all, you know, either directly report to them or are dedicated to them. So they don't necessarily have to be a straight-line direct report. In Amazon's case, for the most part, it was. There were some dotted line, but it could be all straight line.

Starting point is 00:14:15 It could be all dotted line. It could be a mix of the two. But fundamentally, we've moved from what we called a project orientation to a program orientation. So a project orientation means, oh, we're going to do this project to change our search result page and algorithm. And the project is defined in this way and it's going to take six months. The resources will come and swarm on that. And then they'll move off to some other thing in some other part of the company. The program-based orientation says, let's stick with the search example.

Starting point is 00:14:46 there's a team that works on search and they always work on search. And instead of thinking about things on a project-by-project basis, they think holistically about what they need to do to improve search. They have a set of metrics by which they're looking to drive those metrics, largely ones that they can control, things like what percent of the time is a customer clicking on one of the top three results in my search page? Or how many milliseconds does it take for the page load time

Starting point is 00:15:15 in this browser type on this device type, et cetera, et cetera. And they then are running their own roadmap. They are deciding what are the most important things for us to go work on and having a prioritized list of those things and be able to sort of start at the top of the list and work their way down with the pool of resources that they have. Sometimes and most times they may want more resources to be able to tackle more, but they spend less time sort of in resource contention, resource fighting, and instead focus on

Starting point is 00:15:51 building what they can build with the resources that they've got. And so, you know, the benefit of this is if, you know, their success or failures, you know, they're really dependent on themselves now. The only thing they could maybe argue about how they could do better is if they had more resources, which they can petition management for. But this way it also solves a big management problem, which is instead of management, senior management, refereeing every item on a roadmap, they're refereeing which teams have how many resources, which is kind of more of like a once or twice or three times a year decision,

Starting point is 00:16:27 versus refereeing everything on the product roadmap and then all the resource contention issues. That's like a daily issue. And so it frees teams up then to actually go and sprint ahead. There's a lot of work you have to do to get ready for this. So, for example, in a software environment, when we first started and we had, you know, kind of a monolithic code base that was, you know, not pretty, we weren't ready to do this because you have all those interdependencies.

Starting point is 00:16:55 Once we move to, you know, a service-based architecture and then, you know, teams could own their code with, you know, defined endpoints, you know, APIs that other teams could, you know, understand that are well documented, then we could sort of move in that direction. And the other thing is we had to create what I would call countermeasures because there's no free lunch in org structures. Any org structure, you're trading off one thing for another thing. In this case, you're trading off potentially functional excellence. So in other words, if you no longer have every single engineer or every single marketing person or every business dev person reporting into a C-level leader of that particular function, And instead, they're spread out in small teams across the company reporting into some generalists

Starting point is 00:17:43 who is probably not going to have functional expertise in several of the functions that they're leading. You risk the problem of then the people in those teams not gaining functional competency. That's the downside. And we can talk more about this, but we created a lot of countermeasures to still enable us to have functional excellence while creating these single-threaded teams. To drill into this a little bit further, is the kind of the origin of this, this kind of recognition in Amazon

Starting point is 00:18:12 that the best stuff comes from one person's vision and just like one person driving and one person's ass being on the line versus the often decision by committee approach? It's less about that. I do want to be clear. It's one leader and their team who are accountable and responsible.

Starting point is 00:18:31 So with respect to, you know, what are we going to go, Bill? how are we going to go measure success, all those things? This team and that leader are responsible for documenting that, writing their plan. Now, they don't just get to go off and do that. There was an intense review process at Amazon where either at some level, whether it be the vice president, senior vice president, or all the way up to the Jeff level, and his direct report is called the S-Team, this plan would be reviewed and scrutinized deeply as well. And there'd be a discussion. and interchange and basically getting alignment between the senior leadership team and each one of these single-threaded teams on that plan before the team could go off and run.

Starting point is 00:19:17 The beauty of that, though, is that once we had those discussions, those interchanges, then the teams were free to sprint hard after their plan. They didn't have to worry about whether, am I aligned with my CEO? Am I aligned with my senior vice president? could know that they were. But yes, this creates then clear, you know, if they're going to deliver it or not, it's up to that owner and that team. Whereas when you have this highly like cross-functional approach, and there's not, you know, one clear person who's responsible for this one project that's on this roadmap, I've seen many as CEO pull their hair out saying like, I don't, I have no ownership and accountability here. How do I have that? They're pushing on a string because they can't because

Starting point is 00:20:04 that they're different people and leaders are kind of part-owning, half-owning, you know, a long list of things instead of fully owning a short list of things. I like that. I like that metaphor of pushing on a string. Is this approach similar to just the GM model, or is there a big difference when someone's thinking about going GM model versus the single-threaded leader approach? Yeah, I mean, obviously there are probably different definitions of what people consider the GM model, but I would consider that being this person is a P&L owner.

Starting point is 00:20:36 And you can, of course, create, you know, many P&Ls within a P&L. Like, for example, in the book business, we could have, and I don't know, most time we didn't do this, but we could have created a P&L owner just for fiction books or just for professional and technical books, which is a very large category with big differences between the others. And then you say, great, that team, they have their own, you know, dedicated team, they're fully responsible for the revenue numbers and other numbers. But you have to be thoughtful about how you do this because one of the three questions you have to ask when you establish one of these teams is, does the team have the resources,

Starting point is 00:21:15 you know, within their control to effectively manage this part of this department, this product, this P&L? And sometimes then you, if you narrow things down too much in some cases, then the answer is no. In other cases, the answer can be yes. very easily. A great example, this was in Prime Video, one of the businesses that I managed. You know, we could create a single-threaded team who just were working on applications for TV sets like Samsung, Sony. We could create another team that's working on game consoles and another team that's working on mobile phones and tablets. And then within each one of those, we could further

Starting point is 00:21:52 break it down. We could have one team working on Xbox and another one on PlayStation, another one just on iOS. In those cases, then, it's very clear how you can break the teams down and they can have very clear ownership. Awesome. Let's go back to the countermeasures topic. And then even just a little more broadly, you talked about one thing that was important to put in place before you move to the single-threaded leader model, which is creating APIs and basically breaking apart this monolith. What are some other things that you think you need to put in place to be successful in trying to shift to this model? The other thing was these functional countermeasures. Let's just stick with the engineering for an example.

Starting point is 00:22:32 So in 2004-2005-ish, I started managing a single-threaded team, actually managed two different ones, one for music and one for video, which are now, you know, Amazon music and prime video. They weren't called that in those days. But I started managing a small team of software engineers at that point. Well, I have never, well, I have written lines of code, but that would be back in high school. we're talking about like Basic and Pascal. I have a master's in business. I'm, a background in sort of marketing. I'm a generalist. Okay. So I'm not equipped to coach. I couldn't possibly conduct a code review. I couldn't possibly conduct an architectural review.

Starting point is 00:23:17 I couldn't possibly coach or mentor an engineer on how to improve their craft. But I was one of many of these examples. And there could be reverse examples. where instead of me being a business leader, I was purely an engineer. And now I'm managing a team that does marketing and business development. I wouldn't know anything about those things, if that had been my background. So what we did in the, I'll stick with the engineering example, is what we, we came up with various countermeasures. Like one example was that we still had, you know, a sea level leader of engineering in, you know, Rick Dalzell and most of the core infrastructure, and core services still reported into Rick.

Starting point is 00:24:03 So it's things like payments or infrastructure, search. And Rick still could be a technical leader for the whole company. And he and his team could create things like, what are the standard ways that we're going to do code reviews? What are the standard ways across the company that we will interview and screen engineers? What does the promotion process look like? What are the defined steps from getting from an SD1 to an SD2, SD3? How do we document and describe what are the requirements?

Starting point is 00:24:38 There were many things like this. And effectively, what it also meant is that anyone who is an engineering vice president or in many cases as a director, they would often have something else beyond their day job of some sort of subject matter expertise area where they would also contribute to the company. A good example of this would be that they might sit on a panel for promotion from a certain level to another level in the engineering world. Or they might be available to do code review outside of their organization for another organization. So people had other jobs in addition to their day job to build and maintain functional excellence. And there are a lot of examples like this across the company.

Starting point is 00:25:22 Let's go in a different direction and talk about one of the one of my favorite principles. of Amazon, which is disagree and commit. And I think in the way I even describe it, I know is wrong. And I think people hear this term and they often use this principle incorrectly. For example, it actually starts with half backbone and then disagree and commit. So I'd love to just hear how you've seen this actually implemented well and what people should do and think about when they're trying to implement something like this at their company. So when I was at Amazon, there were 10 leadership principles and they've since expanded them. But of those 10, this was all, always the least well understood when I was at Amazon to, and partly because it is actually the

Starting point is 00:26:02 most nuanced and difficult to actually use. So here's what it means. What it means is that have backbone and disagree, meaning when we are in making any kind of a decision, important decision, if you are part of that team, part of that unit, it is your obligation to voice your point of view if you disagree with the approach it's been taking. And the point of that disagreement, by the way, is to provide usually additional information or a new point of view that people have not considered. So I like to geek out a bit on the process of decision making and have read more and more about this. And I think that Peter Drucker probably has the best writing on this topic. as he would describe it, you know, a decision is good decisions are made by first understanding

Starting point is 00:27:01 all the different, you know, points of view and pros and cons to the potential issue at hand or the potential direction. And that great leaders, what they do is they solicit these different points of views. They have a team that they work with to debate and discuss things. So another way to think about this, a king and their court. In an ideal world, you know, if you assume that there's no political motivations, the court is there to advise the king and help them think through different problems and provide different and opposing points of view to allow the king to arrive at the right decision. And this is sort of no different than that, which is it's the disagree part is about bringing forth, you know, new information, new data, new point of view that would be contrary to the current direction. So that's the disagree part, and you're obligated to do it all, you know, as we would describe sort of all the way up the chain if necessary, if it's an important issue. And people are not hearing or understanding your point of view.

Starting point is 00:28:06 Now, the important point is, first of all, about hearing and understanding your point of view. What often happen, I can tell you, if someone in a leadership role, someone would come to me with a disagreement. And many times I'd appreciate it, by the way, because they'd bring some point of view that was useful. but sometimes they bring the disagreement and cite the reasoning behind it, and I already knew that reasoning. We'd already thought of that reasoning. We'd already thought of that. In which case, I would say, I hear your disagreement. We have already considered that factor.

Starting point is 00:28:36 But even though that factor is there, you know, here are these other factors that, like, outweigh that. Now, that is the point at which, as long as the disagreeer is hearing back from the leader, that they understand their point of view, understand like why they are pushing back and seem to fully understand it. And they've taken that into consideration. That is the point for them to commit because the point is you provided your information. They've processed that information and they've decided to go this way with the knowledge of that. Where people get confused about it, they don't maybe understand like when they're supposed to

Starting point is 00:29:15 stop disagreeing is one thing. And so hopefully that explanation. and made people clear like, this is when you're supposed to stop. And then the commit part done well means that it's not just like, I'm going to commit. I don't really agree with what we're going to do, but I'm going to get behind this. It's a, ideally it's, oh, now I've heard the argument. I've actually now thought about the argument. And hopefully that person has now understood.

Starting point is 00:29:50 why we're taking that direction. And so their commitment is based on that understanding because then they can reflect that understanding back to their organization too. Because the worst thing to do is to say, yeah, we're committed to this. I don't really agree and I still think it's wrong, but I'm committed to it. That's not actually commitment. So it's really about, this is really about decision making and understanding the facts or an information that people are going to use to make a decision and then be able

Starting point is 00:30:20 reflect that back. Imagine there are many times I've gone through this where I still don't agree. What's your advice to a manager or to a report of just like, okay, when you actually still don't agree, do you still, do you just behave like, yes, I agree with this and don't really voice your concerns or something else? I work with Jeff on all kinds of different new ideas. And Jeff has, Jeff doesn't think like a normal person. You know, he's his level of sort of creativity and the way he thinks, the timescale, which he thinks.

Starting point is 00:30:52 There's many ways of it the way he thinks that, you know, there was no one else Amazon that thought that way. And so there'd be times when even after we've had that discussion, I maybe still disagree. But then what I would do is I'd focus on, okay, well, what is the, what is the, the kernel or the core of like why Jeff thinks that we should. should do this. And I would focus on that kernel. I got great advice actually from one of my managers at one point, Steve Kessel, who said, you have to look for what that is. And then your job is to then take that kernel and try to run with it and expand it and try to see how I can take that, that, you know, that idea, that concept, and then make it into something viable. And it doesn't always work. But it's, it's about then having that understanding of what it is, not just sort of like going through

Starting point is 00:31:49 the motion of like stomp, stomped, stomps, stomped through it. Like that's not going to work. And it's also, I've seen people who try that and, you know, their career doesn't go very far. You have to have some degree of faith that, you know, there's something there and I'm going to try to do the best I can to, you know, make that part. How would I, how would I productize that idea? how would I make that viable from a business point of view or whatever the different constraints are?

Starting point is 00:32:17 Awesome. So the advice there is focus on the parts you agree with and think about how you can find out if it's actually right or not. Agree with or even just like, you may not even agree, but like what is, like, what is, what is the core of like what, what that person is thinking is the big benefit or good guy or what is the or thinking vector that they're on that's causing them to want to go in this direction? Thinking vector. Love that term. Along the same lines, another principle that I love is

Starting point is 00:32:47 leaders are right a lot. And I feel like this is a term that it almost goes unsaid. Like, it almost can say this in a lot of companies. Yeah. And I'm curious just the origin of why that became an important principle and then how it's implemented at Amazon. Yeah. So going back to this last discussion, so one, you know, one fallacy we should all acknowledge is that when you're making these decisions, you know, you can kind of take, and you're trying to use data to make decisions, like you can make the data kind of look however you want it to look to sort of try to meet your decision. You know, if I'm looking at some issue, I've got some big data set, I can come up with, you know,

Starting point is 00:33:27 ways of looking at a data set to support this idea and ways of looking at that data set to, you know, not support it. So the data doesn't, you know, rarely makes the decision for you. what is happening is then a lot of judgment and interpretation of the data weighing that weighing various factors to then come to a decision. And that is sort of the right-a-lot part. The right-a-lot part comes from having experience, you know, having what we call sort of sound judgment, which generally come, you know, some people maybe are born with this, not a lot of them. Mostly they get it through experience. A lot of experience is actually about being wrong, by the way, about making

Starting point is 00:34:11 mistakes. And by having looked at a lot of problems, made decisions, or observed others making decisions, being a student of that, and then using that to understand then how to wait different information when making a decision. So write a lot is that you're good at that and that then it proves, and that generally speaking, people want to follow someone who ends up by and large, like, you know, going in the right direction, right? You're the leader of a team. The team is, you know, petitioning you on multiple sides. And if you keep kind of going off in some direction where most of the team is scratching

Starting point is 00:34:51 their head saying, like, I don't think that was the right decision, you're probably not going to go very, they're not going to want to follow you very far, and you're probably not going to go very far. So this is something that you develop through, you know, experience. and I'd say from having the opportunities to, you know, observe and work for others that are good at this. I love that it's a lot. I like that it's not just leaders are right. It's like right a lot. Yeah. There's still be, you know, no one is right every time. That's totally unrealistic. Yeah. Let's talk about the titular concept of your book. And that's a word I've never used, but I think it's appropriate, which is working backwards.

Starting point is 00:35:32 First of all, just what does it actually mean to work backwards versus working forwards? The title of the book comes from two things. One is, you know, one of the leadership principles, which is that, you know, customer obsession and the principal states something along the lines of that, you know, great leaders, start, you know, start with the customer's needs and work backwards from there to sort of, you know, meet those needs or solve them. And then also because we created a process in this window was talking about earlier, the 2004-the-2007 window, we created this process for new product innovation called the working backwards PRF-A-Q process. And they both refer to the same idea, which is that as you're

Starting point is 00:36:21 guiding star or the point from which you're going to start, part is what are the customer's problems or what are the customers needs and then figure out, okay, well, what would be the solution to that? What are potential solutions to that? And to do those things to starting with without the constraints of my financial constraints, my resource constraints, my legal constraints, my engineering constraints, whatever all those constraints may be. Because the problem is what most of us do is we start with those constraints and work forward from there. Or we start with things like, I got to increase revenue. How do I increase revenue? I need to increase active customers. How do I increase active customers? For customer-oriented

Starting point is 00:37:16 behavior, we tend to start with those things, which may often lead you in the wrong direction. Whereas we had, as Jeff would say, we took it as an article of faith. If we served customers well, if we prioritized customers and delivered for them, we took it as an article of faith that then things like sales, things like revenue, and active customers, and things like the share price and free cash flow would follow. And so this is important because there's no, I still can't give you objective proof that that is true. I don't know who could. And so it was saying, this is an article of faith that if we do that, we think those other things,

Starting point is 00:37:56 will work out. So therefore, when we're making a decision, thinking about a problem, we're going to start with what's best for the customer and then come backward from there. And then in that coming backward process, we're going to have to figure out, well, to do that, gee, I'm going to have to solve this engineering problem or I'm going to have to figure out how to make this thing cost less or make this thing faster or solve some one or more problems. And, And that's the backwards product. That informs like what's the work you have to do to then create this new solution for customers.

Starting point is 00:38:35 Awesome. So just to summarize, you start with what are the customers needs and problems? And I think a big part of Amazon's approach is what are like the lasting problems they'll always have, which is I think it's like lower prices, faster shipping and all those things. And then think with no constraints. When you work with companies to implement this idea of working backwards, is it always what is the customer problem in need versus like revenue or growth or something like that? Or is there other examples of where you work backwards from at different sorts of companies?

Starting point is 00:39:07 Well, the working backwards part is strictly about the customer's needs. Yeah, we don't want to work backwards from revenue. I guess there is, we don't really, we didn't really use this term for sort of other things like cost structure. Cost structure was actually a part of working backwards from the customer that if we had a low cost structure, we could afford to give customers lower prices. Therefore, let's figure out how to have a low cost structure. Because in itself, driving out costs, you know, doing things more efficiently, doesn't inherently benefit customers because you could just choose to take more profit. It only does if you decide that in doing so, I'm going to lower my prices to customers or provide some other benefit. it. So, no, we used it in this method of like I'm starting from the customer, and then very

Starting point is 00:40:02 specifically we used it in this method of, you know, new products and features that I'm going to go build on behalf of customers. Awesome. This episode is brought to you by Wix Studio. Your agency has just landed a dream client and you already have big ideas for the website, but do you have the tools to bring your ambitious vision to life? Let me tell you about Wix Studio, the new platform that lets agencies deliver exceptional client sites with maximum efficiency. How? First, let's talk about advanced design capabilities. With Wix Studio, you can build unique layouts with a revolutionary grid experience and watches elements scale proportionally by default. No-code animations adds sparks of delight while adding custom CSS gives total design control.

Starting point is 00:40:45 Bring ambitious client projects to life with any industry with a fully integrated suite of business solutions, from e-commerce to events, bookings, and more, and extend the capabilities even further with hundreds of APIs and integrations. You know what else? The workflows just makes sense. There's the built-in AI tools, the on-canvas collaborating, a centralized workspace,

Starting point is 00:41:06 the reuse of assets across sites, the seamless client handover, and that's not all. Find out more at wicks.com slash studio. Okay, so then when you go work with a company to implement this idea of working backwards, what are the very tactical things that you do to help them here? I know PRFAQ is a part of that.

Starting point is 00:41:25 chat about how to implement that. What are the steps to shift to working backwards? Yeah. So the first shift is to take this, you know, so that's just a concept, right, working backwards. Well, how do I turn that concept into a scalable, repeatable process? And that's exactly where Jeff's mind went. And eventually, without getting to the origin story, we came up with this process called the PR FAQ process. So what it means is that whenever we're devising a new product or feature, we're going to start by writing a press release describing the feature and describing it in a way that speaks to the customer and to some degree, you know, the external press and world where the idea is in my description of this, it better jump off the page or something like,

Starting point is 00:42:11 wow, you know, I really, you know, as a customer, I will really need this. And so what I work first is to say, okay, for your product development process, let's, start by using this method to, as the method to decide, what am I going to go build? And, oh, by the way, to use it as a method to sort between a lot of different choices of what you might build. In summary, the way that process works is that you're in the PR, you're going to describe very carefully and clearly, like, who's the customer, what's their problem and what's the solution that you're planning to build?

Starting point is 00:42:50 That sounds really simple and easy. It's actually very hard to do, to do that well to crisply and clearly define those. The first two things are the things that are hardest to define. Like, who's the customer? Like anyone says, like, all restaurants are my customer. Okay, well, that's a mistake now. I mean, like, which kinds of restaurants are your customers? In what kinds of cities?

Starting point is 00:43:14 In what kinds of formats, et cetera, et cetera. And then what is the specific problem you are solving? And ideally, you would some way have like quantified that problem or there's some data or customer insights that have led you to understand that problem to know that it is a meaningful and big problem. Ideally, a problem that people would, you know, pay money if you could, if you could solve that problem for it, because you can just look at the economics of that problem. And, you know, if instead they use your solution, this would be beneficial to them. So I work to have them first implement this PRFAQ process is the first step. And then the next step really is to go from there to say, okay, writing PRFAQ is one thing. Well, how do I actually use them?

Starting point is 00:44:02 How do you actually develop them? Because there's this iterative nature to writing PRFAQs where it's sort of a concentric circle review. Like you start off small like with one author and with low fidelity writing these things. and then you start to share them with a small group and get feedback and improve it. A wider group, get feedback and improve it and onward and onward until, you know, depending on the size and scale of your company, you get up to the CEO as a way to strengthen and improve and really codify this idea and determine whether it's a great idea or not.

Starting point is 00:44:39 So I help them understand like, how does that work? How do you do this iterative process? And then once you've done that, then what do I do with these PRFAQs, like once I've got them? How do I then think about that with respect to my roadmap? Awesome. Okay, that was an awesome overview. I'm going to fire off a couple questions around the first part. Do you still suggest people do it as a press release?

Starting point is 00:45:01 It feels like press releases aren't a thing anymore. Do you ever suggest people do it as a tweet or as TikTok video or a blog post? Good question. So the first thing is it's not a real press release. Okay, we could change the nature of it. And if instead we wanted to call it the customer problem solution statement, right? We could just change it to that because there really are, you know, three money paragraphs in this. First of all, yeah, it's not meant to be a real press release.

Starting point is 00:45:34 So don't use the language you would use if you were sending an actual press release. This is like an internal document. Okay, so that's the first thing. The second thing is the heart. of it really is that like first paragraph that's a short description, that's second paragraph that's the problem statement, and that third paragraph that's a solution statement. If you wanted to ditch the rest of it and the artifacts of the press release, you could. I think there are other benefits to it. Like the headline, is this headline like long and drawn out and like, I can't

Starting point is 00:46:03 even tell what the heck this thing is from this reading this headline? That's generally a bad, if you used a tweet, that wouldn't work very well. The date is also a meaningful thing. When you write the press release, the date is meant to be a hypothetical timing on which you're envisioning, you know, launching this thing, which tells the reader something, are you thinking that this is something that's so simple and easy we're going to launch it next month or so complex that we're going to launch it in a year from now? So there are some other directional cues within it. Like I said, with everything, these are tools that people can use and maybe, and, you know, I'm sure that companies will find other ways to improve upon these tools. But if you don't use those parts

Starting point is 00:46:45 of them correctly, you're kind of missing out on like, what's the main benefit of you're getting out of this. And do you try to write it in a way that would be announced like a press release feel or is it mostly just like, who is the customer? What is, like, do you try to pitch it as a part of this experience? So you try to write it in, in that way, but you're, but the one thing is you don't want to use like hyperbole. And it would be like, very factual, like, with like, you know, numbers, like data rich document, too. So, so again, not like a real press release. A lot of like internal confidential data would be like in this fresh release. So it's, it's a tool that's, you know, has a very specific, you know,

Starting point is 00:47:32 use to it. Is there a template that we can point people to in the show notes to help them craft this, I think there's a version in your book maybe, but is there some online that we can point people to? Yeah. So we have a website related to the book, which is www. working backwards.com. And there's a resources section within there, and you'll find a template. Amazing. Okay. And then the concentric circle piece. So the idea there is basically get feedback from an increasingly larger swath of the company. And it sounds like a big part of that is also good buy-in as you go along the way. Yes and no. So first of all, there are some things where you may write it and you, the author, will, you know, if we were in the old world,

Starting point is 00:48:15 would like take the piece of paper crumpled up and throw in the trash can, which is in your own, you've realized, now that I put this down on paper and read it, this is not actually that good of an idea. I'm going to try something else. By the same token, you may then have written one and you think this is a pretty good idea and you show your up here or your manager and they give you feedback that makes you want to then ball it up and throw it into the trash can. So part of this concentric circle thing is not just that everyone you write lives on and gets all the way to the CEO.

Starting point is 00:48:46 Like, there are no like stats on this. But let's just say in some virtual world, some imaginary world where, yes, all these things, you're a product manager and you've got, you know, a director of product manager report to who reports to some senior vice president division reports to a CEO. Well, if you truly run this out and you write 100 PRFAQs in a year, maybe 20 of those make it their way to the CEO, right? The point is not every single one of them is destined to go that far. They're, you know, the numbers, you know, get narrower. And this sort of leads me down the concept of what you're really trying to create is a product funnel, not a product tunnel. And with a funnel, meaning, you know, lots of things at the top, fewer things at the bottom.

Starting point is 00:49:35 the tunnel means that everything that comes in is also going to come out the other side. And the problem with that method is that means you have no consider, you're not actually having a method of like consideration and comparing it against other things that you might build or how you deploy what are, frankly, most companies, your most precious resources, which is your engineering team. You should be looking at various choices. You should think of yourself honestly as like a venture capitalist. They don't fund every company that they meet with.

Starting point is 00:50:04 They actually found a very, very low percentage of them. And at Amazon, you know, we had lots and lots of PRFQs that were a great idea, but we didn't ship them because we had other ones that were just a better idea, which had, you know, a bigger potential impact. So you want that, you want to create this like corpus of ideas that are well thought out and select the best ones. It feels like a lot of these processes are basically just ways to stop stupid shit from happening. I think the narrative is a good example where you have to, like,

Starting point is 00:50:34 expose your thinking deeply. This is a great example of that. Yeah. And it's also, I would say, an example where this is a process to prevent the other process, which is the product development process from becoming the thing. Right. Right. Where you just get locked in on what are we doing in this sprint, you know,

Starting point is 00:50:56 what are we trying to get done and like focused on shipping stuff. What I recommend is you try to break into two different processes. One is the process of deciding what you should go build, and that's what the PRFAQ is designed for. And then once you've decided that, then yes, by all means, use all that good thinking. Now, how can I ship it efficiently and effectively with, you know, few to know bugs? I was just reading this Harvard Business Review article. I think that's called the thinking to doing gap, where a lot of companies just spent a lot of time talking about ideas and solutions and not actually doing anything. and so I'm curious how you try to avoid that at Amazon,

Starting point is 00:51:35 considering there's this period of just like, let's explore, explore, explore, we're fine. There's a couple ways, and of course, I'm somewhat having to imagine, you know, what are the problems in such companies where that's going on. So one such version of this problem is what I'd call the big idea that's not fleshed out problem. So I'm sure that every single person listening to this podcast

Starting point is 00:52:00 has either themselves done this or have witnessed others in their company who come up with a concept of like, oh, I think if we built this, boy, that would really, you know, solve things or that would really work well or that would really grow things.

Starting point is 00:52:16 And the, and it may sound good to everyone. It may sound good, you know, to you to everyone and then maybe you start then working on building it. But the reality is that actually once you've spent some time, looking at that idea more deeply, you then start to identify several roadblocks or maybe a fatal flaw with this idea. And in fact, no, you shouldn't waste any of your time going to building

Starting point is 00:52:44 that thing because it has a fatal flaw. So one problem is that companies get stuck, I think, where they never actually go do that documentation. And so it's a debate and discussion about concepts that aren't really well fleshed out. And so people's ability to actually evaluate them in any realistic way is they don't have a good way. And so in those situations, you know, what gets done is probably more of a function of, you know, politics or will or, or, you know, a culture of, you know, completely like top down. I think the, you know, the other way is where they're debating and discussing things, that they just don't have, like, good methods where then they can take things and then go build them, meaning they probably don't have the right

Starting point is 00:53:33 org structure or processes in place to then go take the good idea, assign it to someone who will own it, go look at it, and after they have owned it and gone and looked at it, if it works, then they and their team, you know, can go actually build it. What I was found as, you know, once as I became more senior in the company, my, my role became bigger and bigger is that when something came up, some idea that didn't neatly fit within my org structure, I couldn't necessarily delegate it to someone that this, there were only two things I could possibly do, which is just sort of set it aside altogether because otherwise just be a real distraction to people. or I had to decide this was a compelling enough idea that we were going to take a resource. Could be one person, could be a whole team, depending on the idea. And we have to assign that resource to actually go look at this and work at this. Otherwise, it will never happen.

Starting point is 00:54:31 I've been through those many times. Okay, so there's two more concepts I want to try to touch on before we wrap up. The next one is the idea of input and output metrics. There's something that Airbnb we super implemented. it became a very de facto way of thinking. And actually, there's a lot of Amazonians that ended up at Airbnb, a lot of leadership. So there's a lot of the stuff that we ended up doing like the memos. And so on the input out metrics, to just describe what that is and why that's so important,

Starting point is 00:54:59 why people kind of think about metrics in the wrong way often. Yeah. So the origin of this one really was, again, kind of in our early years at Amazon, 99, 2009, 2000, 2001. We would, you know, we're a public company then. We were growing. But then, you know, growth started to like, it wasn't just all like up into the right and like, woohoo. You know, every company is going to hit a wall eventually.

Starting point is 00:55:25 And it's not going to be, you know, if you're so lucky to have even been at a company where it's just like going up into the right with no gravity, good for you because many people don't, never experience that. What most people experience is the reality is there's a lot of gravity pulling against your your revenue numbers and you've put a plan out there and you wanted to grow 15% or 20% or 75, whatever it was. And now you look like you're not going to hit that number this quarter. And so what ensues then is they, we're not going to hit our number.

Starting point is 00:55:55 What should we do about that to hit our number? And this often happens with, well, you know, there's like a month or a month and a half left in the quarter. And then we would run around like chickens with our heads cut off and come up with a bunch of ideas that tended to be like promotional in nature, it tended to be like price reduction in nature or like we'll send this extra email or extra ad or whatever it might be. Right. And the reality is we did that, you know, we went through that enough time, several quarters. And we started to realize, huh, these fire drills don't really work. We're not, we didn't really like get meaningful progress

Starting point is 00:56:33 against the number with these like last minute things we decided to go do. And oh, by the way, they were kind of a big distraction. Probably if they did work at all, they pulled revenue that might have just gotten in the next month or next quarter into this one. So it wasn't really kind of a zero-sum game there. And we realized, you know, we're not really actually working on things that matter

Starting point is 00:56:53 to customers that are going to, like, move the needle over the long term. And this is about the same time when Jeff and the S-Team were reading the book Good to Great. And I would, you know, you have to ask Jeff what it is. But if you ask me, I think that this was the single most influential and effective management book for our company. Because what it caused Jeff to do, and I won't describe what, you know, most of you probably know what it is. If you don't know what it is, go read to good to great. It is, in my opinion, it is the best, you know, most important management book you'll ever read.

Starting point is 00:57:31 because what it did is to help us codify our growth flywheel, meaning what are the inputs that if we improve these things, which in our case was, you know, how do we have broad selection, how do we have a great customer experience, where great customers' experiences in retail things like how easy was it to find what you wanted to buy, how easy was it to buy it and how fast did it get to you, were the prices low? Do we have lots of merchants on our platform?

Starting point is 00:58:04 And by the way, could we drive out costs? So we identified these things on our flywheel. And this identification of these things was such a critical moment for the company because then it realized, okay, well, what we need to do is spend our time focusing on like, how do I measure each one of those things? And then how do I improve each one of those things? So it shifted our focus away from like this short-term thinking of pushing the revenue number up to this longer-term thinking that if we just improve these things,

Starting point is 00:58:33 whether it's, there's no day that people will wake up 10 years, 20 years, 30 years from now and say all else equal, I'd rather shop at a store with fewer items than more items or a store with higher prices than low prices or a store where things get to me more slowly versus, you know, more quickly. So if we can just improve these things, you know, this is our path to winning. So those are all inputs, inputs to the customer experience. And so we then figured out ways to measure them creating a set of input metrics. And so then when we would develop our operating plans and review our business each week and set our goals, we were hyper-focused on those inputs and the input metrics. As a simple example, there was one tool that Jeff and the leadership team,

Starting point is 00:59:21 the S-team used, called S-team goals, which are effectively a list of what they would harvest, be like, here are the most important goals for the company that I've harvested from all of our operating plans. And I can't remember exactly what year, something around like 2007, 2008. They looked at that list, which is about 500 items long, by the way. And they counted it up. And of that list, only 10 of them actually had a financial metric in it, like revenue or free cash flow or like gross profit. These other things were generally speaking all, you know, one of those inputs, like I mentioned to you about low prices and selection and speed of the customer experience. So, yes, the point was, again, it's this other article.

Starting point is 01:00:07 So we took it as an article of faith that if we can just improve these inputs, the outputs will take care of themselves. The inputs are the things that drive the outputs, which are revenue, customer activity, free cash flow. And so one of Amazon's, you know, it's not really a secret, but one of Amazon's great strengths was that would focus on those things and make, you know, just continuous process, continuous improvement on each one of them and measure them, you know, rigorously. The flywheel, you reminded me, that was another, it feels like that's another concept,

Starting point is 01:00:40 Amazon proliferated through all of companies is everyone's trying to create their own little flywheel. And I imagine everyone has that image of the Amazon flywheel in their head with a little orange circle in the center and the black arrows. On the topic of input metrics, just briefly, what is an example of a good? good input metric because I imagine people are listening like, oh, shit, I got to think about my metrics as input and output now. What's a sign that's a good input metric? Sign that's a good input metric is, first of all, map your end-to-end customer experience.

Starting point is 01:01:07 Like, I never worked at Airbnb, but okay, step one is that they clicked on some ad somewhere and showed up in the website or the app. Now, you're in the app. Now, you're looking at this first screen. Well, the first thing, what they're doing is they're browsing and or they're searching. Okay, how are we measuring the speed, quality, and ease of that browsing and searching? Now they've got onto a different, a detail page for an individual property. How are we measuring the speed, ease, and quality of the different actions they may take, like, you know, reserve. Forgive me if I get any of my terminology wrong, I'm not an Airbnb.

Starting point is 01:01:46 You are, but it doesn't matter. It's close enough. And then, you know, so then you've reserved. Now you have interactions with a property owner. How do I measure the quality of those? Am I going to, you know, how many messages go back and forth? Is a lot of messages a good thing? Is that a bad thing?

Starting point is 01:02:03 At first, you may not know the answer to that question. Same thing. Every step of the way, then there's the actual rental experience. How do I instrument and measure every part of the customer experience? So you know it's an input metric if it is measuring something, with respect to the customer experience. Which ones are the right metrics? Which ones are the most causal to the outputs?

Starting point is 01:02:30 I couldn't begin to tell you. This is actually where you're getting paid for. If you ordered Airbnb is to figure that out. And basically through an iterative process of measuring, observing, improving, and looking at what the effect is on your outputs. So, again, we didn't really create this concept. This is a concept from 6 Sigma, which is using what's, you know, Damaic, which is I have a process. There's an output of this process, but the inputs are a black box to me.

Starting point is 01:03:04 So how do I understand those inputs? Well, Damaic stands for define, oh boy, define, measure. The A is going to come back to me in a minute, improve and control. and I'm going to have to, oh gosh, the A is lost, I've lost it for a second here. But I'm looking at we can define, measure, analyze, improve. Analyze, thank you. Yeah, duh, analyze. So, you know, we just use that process, which is, and by the way, the way we think about it

Starting point is 01:03:38 first is like, well, you need to throw a lot of things at the wall. You don't really know which of these things are going to be the most causal. So you know you're doing input metrics. if it is, do you control it? Meaning, can you apply resources to make this thing better or worse? Does it touch customers? It doesn't always have to touch customers, but if it is affecting the customer experience,

Starting point is 01:04:02 it's almost certainly as an input. And then which ways you're going to measure that input, you need to try more than one way. Because, like, again, we tell a story in the book about one of our most important input metrics, which was how much selection do we have? and we were actually not measuring that right for several years. We had to refine that measurement.

Starting point is 01:04:23 So I don't know if you saw this, but I asked on Twitter what questions I should ask you and told people you were coming on. And something that came up a bunch is with working backwards, obviously some products Amazon has launched have not worked out, say, the fire phone is a classic example. What have you learned from that process of just like, okay, here's signs, maybe this won't work out, also knowing many things are not going to work out.

Starting point is 01:04:45 There's no way to really know. Yes. So the one important thing to share is that all these tools that are describing this book that Amazon is using, whether it's using documents and meetings with a PRFAQ process or input metrics, is that none of these things give you the answer. They are tools to help you make decisions. So sometimes you're going to make the wrong decision. Firephone is a great example that comes up often people ask, well, you know, if you've got this great PRFAQ process, like, how did you get Fire Phone? So I was sort of tangential to the Fire Phone team, and I work on it closely. And, you know, different people have different opinions. So I'll just share my opinion, which is that if

Starting point is 01:05:30 you think about, again, how does the PRFAQ process work? Well, there's a customer problem. Well, what was the problem that the Fire Phone was seeking to solve for customers? I would argue this is a case where we made the mistake of what we had a technology solution in mind, which was 3D effects. Then we took that solution and were then in search of a problem. I don't think it solved any meaningful problems for customers. In Canada, I was, you know, we had to build a version of the music application and the prime video application for this phone. and I couldn't figure out how this 3D part would make it better for the customers to discover, watch, or, you know, play back any of these media. Maybe there were games that could have been, you know, great solution. I don't know.

Starting point is 01:06:22 But I think, you know, the simplest place to go when you see a failed product is to ask yourself what problem did you solve. And I could get into all kinds of other examples outside of Amazon too. But nine times out of 10, I think that's where if it wasn't poor execution, if the product was executed correctly, what was wrong with the concept of the product? I imagine there was a lot of disagreeing and committing on that concentric circle process. Is there anything that you found of just like the number of disagreement and commits in this process of PRFAQ, filtering out? I don't know. That tells you like maybe this is not a good idea. Not necessarily. So I'll tell you partly also why the fire phone happened was, you know, from my point of view,

Starting point is 01:07:09 I think, you know, I think that we had had a number of successful products where in some cases there were a lot of people who doubted whether it worked, a lot of people doubted whether the Kindle inside Amazon doubted that the Kindle was going to be a good idea. I remember contentious, you know, board meetings on this topic. So even, you know, even within a company that was considered innovative, you would have a lot of people, you know, who would doubt things. I can tell you that for years, is working on prime video, I would tell people about, you know, what our ambition was, you know, watching our TV set and we're going to have our own motion, you know, studio, they'll make our own movies and TV shows.

Starting point is 01:07:45 And they would laugh at me. That was crazy. So that's not necessarily the sign of whether, you know, the product is right or wrong. And so that, that's, a problem, actually. That makes it harder to know. And I think something Amazon's incredibly good at is being okay with a lot of failures. And I think that's part of the reason. There's been so much innovation. Is that true? I'd say it's partially true. I mean, again, it's hard for me to do it compare and contrast with other companies because, you know, but I can tell you, did we have a lot of things that we launched that failed? Yes. Some of them are, some of them are, you know, very, you know, public and

Starting point is 01:08:27 obvious. I'll give you one that people don't really realize it's something called we had a feature in the early 2000s called slots. And what it was was basically third parties could, you know, bid on different search terms and put like a little ad in there. Well, it didn't work. Obviously, that works now on Amazon, but it didn't work then. It didn't work then because we simply didn't have the scale and that we had, that Amazon has today. So a lot of times a product idea, It's a perfectly good idea. You just have the wrong time or the technology isn't there. I mean, Jeff wrote about a product that was a puck that said in your kitchen that you would talk to and ask it for things and could shop from it.

Starting point is 01:09:11 He wrote about that in 2004. Well, the technology wasn't there to be able to create that little puck, which one day would become Echo was sort of a decade away. But we had a lot of things we launched that failed. we were not afraid to take what we considered a well-calculated risk. I think many, many companies are less willing to do so, less committed to product innovation. And really do not want that fear of failure. They do fear of failure and they're really focused on sort of their near-term financial goals.

Starting point is 01:09:46 It's sort of not their fault. It's kind of the way a public company and Wall Street sort of interact. with each other sort of creates this dynamic. Just to pull on that thread a little bit more, it feels like a lot of companies talk about we're okay failing, we're okay launching things that don't work. But then in practice, their performance review is impacted. Teams get shut down, budgets get pulled. Is there something that you recommend to companies that want to actually improve in this?

Starting point is 01:10:13 Like, what could they actually change and actually do this well? Yeah, I just spoke with actually a senior executive at a well-known Silicon Valley the company about this topic the day and said, well, what is it, what is it we had structurally at Amazon, especially from a people point of view that would enable or encourage people to take these risks? Because, yes, in a lot of companies, if you go work on the project that fails, like then your career, you know, is in the garbage can and or your compensation system, like you're going to lose out on that bonus. So there were two things. One was our compensation system. So there were no performance bonuses. So if I was running the book business and I had a killer year from a financial

Starting point is 01:10:58 point of view, there was no like extra kicker for me. And if I ran the book business and it had a bad year, there was no financial penalty for me either because our compensation was based on the stock price. So we all had an incentive to do was right for the company, frankly, over a long term because, you know, trying to win off of short term fluctuations off Wall Street is kind of a losing proposition, which meant that therefore, if I am, you know, so I mean, because I had that situation, I moved off of working on our largest P&L and then at the book business and music and video business, now I'm going to go work on digital media. There is zero business there. This might not work. Well, my, it didn't, my compensation didn't change as a result of that.

Starting point is 01:11:43 Didn't change, you know, one way or the other. We tended to also have our performance management system that then would change compensation based on evaluating what did you actually deliver kind of more in an input method? We cared about the outputs too, but there are plenty people that could be in a business that's like up and to the right, but has nothing to do with them. And so we tried to focus more on, well, what did you actually build and contribute, you know, ways you improve selection or lowered prices or whatever that might be? those two things about the compensation, you know, mattered a lot. And then the second thing was having, you know, a CEO who was really committed to it. And it wasn't something that they delegated to someone

Starting point is 01:12:30 else. So Safi Bacall, I think, writes about this in his book, Loon Shots, where part of the conditions that are necessary for innovation to occur are that you actually create different structures of decision making, of approvals, of all kinds of kinds of things if you create some, you know, team that's going to go build something new and innovative because most of the structures inside a big company are kind of designed to crush and impede a small, innovative team that's trying to go build something new. They need speed, but, you know, some approval here, approval there. It's going to like get in their way. And one of the ways we solved that two ways. One was when we went to go build, you know, digital media in AWS,

Starting point is 01:13:14 we put like two of our smartest, you know, leaders in the company. company on those things, Steve Kessel and Andy Jassy. And number two, they were meeting with Jeff regularly. Jeff was deeply engaged with them, reviewing like, what are we going to go build? Part of the decision to decide we're going to go build. And so he could then also, between their seniority and, of course, him being the CEO, they could run interference on these sorts of things, too. So even if you want to have innovation, even if you really, you know, you really do crave it, you're willing to take the risk. If you don't set up the organization, in the right way, like, you're just not going to get it.

Starting point is 01:13:52 Amazing. I'm glad we got into that. I wasn't planning to talk about that, and I'm glad we did. Final topic. This concept of bar raisers, it feels like it's been such a core way of allowing Amazon to scale successfully. And I think there's something a lot of people can implement. It's like a very one-off thing you could just implement at your company.

Starting point is 01:14:11 You just talk about what this idea of a barraiser is in the hiring process, and then what people can do if they wanted to have this. to their hiring process? So the barraiser hiring process is a process, is actually one of the first ones that was established the book. It was pretty early in the company's history back in 1999. And we created it for a simple reason to quote one senior leader at Amazon, we had new people hiring new people, hiring new people.

Starting point is 01:14:35 We were in our hypergrowth phase, okay? You know, we were, the company was only, you know, what, three, four years old. And we were growing like a weed at that point. So what this started off actually in our tech org and what, you know, our senior leaders in tech realize is, my gosh, we, you know, we hire some new engineering leader. And the next thing is that their job is to go hire like, you know, the senior managers and they'll go hire like managers. And all these people have been here for, you know, like a week.

Starting point is 01:15:07 So we don't really even know our company yet. They don't know our culture yet. They don't know our standards yet. So what information are they using to make these hires? And, you know, what information they were using is obviously they were just using their own personal judgment. And their personal judgment combined with whatever criteria they used at prior companies that they work for. So let's say if they came over from Microsoft, you know, if Microsoft had some methodology or criteria, they probably would just apply that. Well, is that methodology or criteria relevant to our company?

Starting point is 01:15:42 Because every company has a different culture. And, you know, I'm here to tell you. that if someone's been a super successful vice president at Microsoft does not mean they can be a super successful at Amazon or at Google or Facebook. Sometimes they can, but these companies are very different. They all do work very differently. The way leadership happens and decisions are made are very different. So how do we fix this problem other than letting it run rampant and basically our hire a bunch

Starting point is 01:16:10 of people who are, we don't know if they fit our culture and we don't know if they fit our high standards we have for what we expect. of, you know, engineering leaders or engineers. So they created this barraiser process, which, by the way, they borrowed from Microsoft, which had a process called As Appropriate. And the concept was that on every interview loop, there's one person who is not the hiring manager, who doesn't report to the hiring manager, who's not the recruiting manager, they're, you know, a line, they're in the business, like they're a software developer

Starting point is 01:16:43 manager or they're a marketing manager and they are on the interview loop and they're a barraiser which means when we get to the debrief meeting they will run that meeting not the hiring manager not the recruiter they will run the meeting and it also means that they technically have veto power over the hiring manager which by the way a good barraiser never uses or I never saw barraiser use I was a barraiser and in my 15 years at Amazon I never used it never saw it. used. And then finally, which actually was not true in 1999, but later became true, was once we established our leadership principles, we created a set of objective criteria that would be used and an interview methodology would be used in every interview, which was the objective criteria

Starting point is 01:17:28 would be our leadership principles and the methodology would be behavioral-based interviewing. So this barraiser was basically, would be a subject matter expert on how this process worked. They'd conduct the debrief to make sure that we were actually adhering to the process, that people were sticking to the objective criteria, other than saying, I don't think we should hire this person because, I don't know, they don't seem to want to work here enough. That might, maybe that's a valid reason, but it's actually not part of our objective criteria. And so the, the barraiser was there to act as a balance also on the urgency bias that every hiring manager has, which is like, I got to fill these roles. And, but rather than filling them with the next warm body,

Starting point is 01:18:12 they find, make sure they fill them with people who actually meet our standards, fit our culture, and meet our standards for, you know, functional excellence too. Such a cool process. Two questions along these lines. One is, who has the final decision in hiring? Is it the hiring manager? And this is just advice from the borriser? Yeah, so this often gets confused.

Starting point is 01:18:33 The decision maker is the hiring manager. The whole interview loop and the barraiser are actually just there to help the hiring manager make the right decision. Now, oftentimes the hiring manager could, you know, feel like this is actually a bureaucratic process and a group of people that I have to sell and they're just in my way between me and hiring this person, which is kind of a natural feeling to have. But one of the feedback I would always give managers who are new to this is like, no, no, no, that's not the way to think about. I think about these people are helping you because the amount of time you're going to put into the hiring process may seem like a lot. But if you hire the wrong person, boy, that would be

Starting point is 01:19:09 amount of time, you're going to have to deal with managing that person, that's going to be a lot more. The impact on the team, impact on you. So making a great decision here is important. They're here to help you. So yes, the final decision was with the hiring manager. Technically speaking, the Farriser could block them from a decision to hire someone. But they would allow, well done, they would help the hiring manager see the reasons not to hire the person through kind of a Socratic method and how they would guide the discussion. And then when you're choosing a barraiser, is there any suggestions you have of who to choose and how often you pull them into these things?

Starting point is 01:19:46 Because it could also be a huge time suck. It is a huge time suck. And it sometimes could be up to 10 hours of my week spent actually as a barraiser. The selection process is, you know, you start with as a company, I would recommend, if you wanted to do this, you pick a department to pilot it with, pick some, pick people who are, A, care a lot about your hiring process, B, appear to be good interviewers, and C, seem to have high standards. It's also a great role for people who are earlier in their career by giving them this additional sort of leadership opportunity. It's a great way to grow and

Starting point is 01:20:23 develop leaders, by the way, because this added responsibility is a great way for them to start testing out leadership. And you have to train them properly and you have to have dedication to the process, but I generally would try to pilot it within one group at first. One last question before we get to our very exciting lightning round. Many people are listening to this. They're considering implementing some of these things, trying to figure out how to actually make these real. If someone were trying to move along the path of becoming more Amazonian, which of these

Starting point is 01:20:57 elements and processes do you think often has the most impact and or is there something fundamental that needs a change to allow for some change like this? to happen at a company in your experience. Yeah, good question. And the first thing I'd say is one thing to be careful of is, a lot of times when I'm talking to a company about these processes, they say, well, does this mean we need to turn into Amazon? And the first thing I tell them is, well, first of all,

Starting point is 01:21:23 I couldn't turn you into Amazon if I wanted to because you have your own culture. And secondly, no, that's not the idea is for you to try to become Amazon. The purpose is to sort of look at these processes and best practice they have. and consider adopting parts or all of them into your organization to improve these same. Every company of a certain scale has these same processes. So this is just a different way to do them. So you should have scalable, repeatable processes for each one of these. You know, pick one.

Starting point is 01:21:54 Here's one choice of ways to do these things. The other piece of advice I give is that a lot of these changes are relatively profound. They really require buy-in all the way. to the CEO because if you're really going to change the way you do product development or if you're really going to change the way you do hiring, that probably requires buy-in of the CEO. And so I would, you know, seek to get that probably before I would move too fast. Some of these things, though, can be piloted in your own little group, like you're one little product development group. You want to decide you want to start writing PRFAQs. You probably can decide to do that. But again,

Starting point is 01:22:30 you know, try to check with your leadership. The other thing I would just tell you is that for any of these processes, these in our book or any book, implementing a new process is not, you know, easy. And if you kind of go into it lightly and kind of dip your toes into it and sort of try it out, it's probably not going to work for you because it'll be hard at first. And it requires some level of commitment to actually work through that hard part and say, like, no, I'm really committed to doing this. And it will take, you know, a few months for you to get good at it. So you have to have commitment and discipline.

Starting point is 01:23:04 to get through it. It's, you know, anyone can really do these things. It just requires commitment and discipline. And in our chat, we've basically just scratched the surface of a lot of these things. If people want to dig deeper, there's obviously your book working backwards, which we'll link to in the show notes. I know you also work with companies to implement a lot of these practices. Could just talk about what it is you can help folks with and then how to potentially engage if they're interested. Sure. Great. Yeah. Colin and I, one of the reasons we wrote this book was to pass on what we learned to the next generation of business leaders sort of at scale with a book, but also because we had a passion to work with companies, you know, directly one-to-one.

Starting point is 01:23:42 And so we are advisors, consultants, call it what you will, but non-traditional, like we don't have a team of people working for us. We, each of us just work directly with the companies who engage us. And generally speaking, what we do is we are, the right kind of company for us to work with, first of all, has to achieve a certain scale. Companies that are sort of in the product market fit phase, like they need to focus on getting product market fit. They probably don't really need to focus much on like how they put in scalable, durable processes. Like, sure, some of these could definitely be helpful to you, even if you're in that phase. But really these are designed for like, my companies kind of become complex now.

Starting point is 01:24:23 I've got multiple product lines. It's, you know, well over 100 million annual run rate, growing fast, complex. So most of our clients are either, you know, large, you know, well-passed C-C private companies or their public companies. And in most cases, you know, a sea level leader or the CEO themselves has, you know, read our book and recognizes that they have a lot of the same problems that we had at Amazon and looks at these as, you know, useful solutions and wants us to help us to help them implement them.

Starting point is 01:24:56 So we tend to usually first actually go in and do an assessment of how they do things today, because to help people move from one place to another, we have to understand where they are. And then we come up with a prioritized list along with that, the CEO and C-level leaders of like, what are the things that would be most useful? What are the symptoms and problems you're having and what are the sort of the root cause solutions

Starting point is 01:25:18 that could be found in these processes? And then we sort of prioritize those and come up with a plan to work within the organization to help them implement those. And what's also sort of different is that we're very hands-on working at all levels of the company. And we will, as we do it, we will be there in the meetings with the teams to help coach them and teach them along so that we make sure that it actually gets implemented properly

Starting point is 01:25:42 into spec and they get to the outcome they want. Sounds amazing. How would people engage with you if they wanted to explore this? Simple way is you can just send an email. I'm Bill at working backwards.com and Colin is Colin at working backwards.com. You can also just check out our website, www. at www. working backwards.com.

Starting point is 01:26:00 We have some information there. We have a contact us form. Those would be the best ways. Okay. Well, with that, we've reached our very exciting lightning round. I've got six questions for you. Are you ready? I'll try.

Starting point is 01:26:12 Interestingly, as I look through the list, many of them relate to using Amazon, which is pretty funny. The first is, what are two or three books that you've recommended most to other people? So I'd say in the management world, not surprising you'll hear good to great. I'd say Drucker on management or Drucker the effective executive.

Starting point is 01:26:32 And then the other one, I'd say that's a little bit different. I'd recommend the Steve Jobs biography. I've, you know, I never worked at Apple, but sort of looking at that arc, a lot of the way those things worked was not that different from what I experienced at Amazon. So it's sort of a good window into what it's like to be inside some company, tech company that goes through product innovation and, you know, big growth. on a personal basis, recent books would be Seven Eves by Neil Stevenson is a favorite

Starting point is 01:27:02 and gentleman in Moscow. Amazing, can get them all on Amazon. Yes. Another Amazon related question potentially is, do you have a favorite recent movie or TV show? Might be on Prime, might not be. Yeah, my favorite recent movie is the latest Dune movie and I can't wait for the new one to come out.

Starting point is 01:27:19 What is that coming out? It seems like I've been waiting a long time. I think it's supposed to come out next month. But I used to know this, you know, I used to have to know the answer to this question, but I don't anymore. But I anxiously await the next one. I thought that last one was awesome. I even liked the original Dune movie, so I'm probably unusual that way. And I just watched, along with my wife, we just enjoyed watching the TV series A Spy Among Friends, which was on MGM Plus.

Starting point is 01:27:49 MGM Plus. I have not even heard of that. I had not actually heard of it either. to subscribe to you. You can basically go on to Prime Video and you can find this show and you can kind of subscribe to MGM Plus through that. Thank you, Prime. What is a favorite product you've recently discovered they really like?

Starting point is 01:28:05 Maybe that you bought on Amazon. Maybe now. This one did not buy on Amazon and this one is, most of you may not understand this one, but I'm an avid cyclist and I got myself a new set of wheels for my road bike this year. Actually, my road bike and my gravel bike. It's the Zip 303 Firecrest. the latest model. And boy, these are just fantastic wheels. They're light. They're sturdy. They absorb all the bumps well. I can use them on a road bike. I can use them on a gravel bike.

Starting point is 01:28:34 Awesome wheels. Wow. That might be the most obscure random product that we've had yet. Recently we had a humidifier. So I like this collection of products we're building here. Create a wish list on Amazon maybe. Do you have a favorite interview question that you really like to ask? Yeah. It's actually quite. basic, it's, you know, tell me about their most significant professional accomplishment. And I always clarify this. By this, I mean not some award you won or some promotion you got. I mean something you built or some product, some process, some organization you built, something like that. And I could basically then, you know, once they get into that example, ask a lot of, you know,

Starting point is 01:29:21 probing and follow-up questions. I could fill an entire hour. interview, just sticking with this one example to really understand how they, using the star method, which is to try to understand everything from, you know, the situation to the result and everything they did in between, who they influenced, how they had influenced, what decisions they made, what roadblocks they encountered. If I just pull on that one string, I can learn a whole lot about a candidate. Next question, what's a favorite life motto that you often find yourself coming back to, sharing with friends that you find useful? Well, one that I end up coming to a lot professionally and somewhat personally is this

Starting point is 01:30:02 one called Slow is Smooth and Smooth is Fast. This is a, I believe the origins of this one are actually from the Marine Corps for the scout snipers. So I'm not trying to promote, you know, that particular craft. But the point of it is that actually, and we did this a lot of Amazon, it really, oftentimes, to really to go fast, you actually need to kind of go slow first and to be very clear on what you're doing and where you want to go. And most people confuse, you know, speed with velocity.

Starting point is 01:30:39 And the difference between the two is that, you know, speed, velocity has both speed and a vector to it, meaning you're some specific destination. And so I see a lot of people who were going very, very fast. but the destination isn't very clear. They haven't really sort of thought that out well. So slow is smooth, smooth is fast. There's a similar quote that I've always thought of, of you've got to go slow to go fast. And I always thought it was Stephen Covey, but I just Googled as you were chatting and someone

Starting point is 01:31:06 named Peter Singh in the book Fifth Discipline. It took me a while. I always wanted to go fast first and not go slow first. So I'd say a lot of my personal development and growth. and that's a big thing that I learned from Jeff at Amazon. Final question. I don't know if you'll have an answer to this, but is there a pro tip that you could suggest for using Amazon,

Starting point is 01:31:30 something that people may not know about how to get the most out of using Amazon.com? Sorry, I have no secret insights. There is not like something like if you go on Monday mornings at this time, the prices are lower or something. In stock, it's better. No, I know of no such thing. Which I think is great because it's built exactly as it should be for customers. I guess so.

Starting point is 01:31:56 But it used to be that you, you know, maybe in the days of the slower internet, that I would tell you to go at, you know, non-peak hours, but this isn't really an issue anymore. I'd be wild if that was still an issue. Bill, this was everything I hoped it would be. Thank you so much for being here. You already shared where folks can find you online, so I'll skip that question. So final question is, how can listeners be useful to you?

Starting point is 01:32:18 You know, we're always looking for, you know, feedback. You can post a review on Amazon for our book. That's probably the best way. You can fill out the contact us form or send us an email telling us what you found most useful in the book or what you actually found that is missing. What would you like to learn more about? If we were to, say, write another book or write more, what would you like us to tell you about? Amazing. And that's Bill at Working Backwards.com.

Starting point is 01:32:48 if they have that feedback. Go by the book Working Backwards on Amazon and other places and Working Backwards.com to learn more. Bill, thank you again so much for being here. Thanks so much, Flaney. Really enjoyed it. Me too. Bye, everyone.

Starting point is 01:33:03 Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes. or learn more about the show at lenniespodcast.com. See you in the next episode.

Lenny's Podcast: Product | Career | Growth - Unpacking Amazon’s unique ways of working | Bill Carr (author of Working Backwards)

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.