Software at Scale - Software at Scale 8 - Farhan Thawar: VP Engineering, Shopify

Episode Date: February 1, 2021

Farhan Thawar is VP Engineering for Channels and Mobile at Shopify. Previously he was the CTO and Co-Founder at Helpful, CTO Mobile at Pivotal and VP, Engineering at Pivotal Labs. He was named one of Toronto's 25 most powerful people.Apple Podcasts | Spotify | Google PodcastsWe discussed frameworks for making life decisions, the structure of engineering organizations, setting up successful hiring pipelines and healthy organizations, committing to large engineering decisions (like React Native for Shopify mobile apps), pair programming, growing engineers and leaders, and more.Highlights1:28 - A framework for making long term decisions. “Work with smart people on hard problems”.9:00 - The varying experiences of being an engineering leader at both small and large companies.12:30 - The Roles and Responsibilities of a VP of Engineering16:30 - Diving into the decision of going with React Native for Shopify mobile applications. What’s the process to make that decision?26:12 - Should large scale decisions like these be bottom up or top down? What’s the tradeoff. 31:00 - How much should an engineering organization invest into platform, vs. customer facing work? Why Farhan thinks they might have over-engineered their systems at Helpful.35:30 - Measuring developer productivity. A well reasoned, first principles argument against using commit count40:00 - Managing 120 direct reports at Pivotal, and how to keep an open mind about the actual scaling limits of engineering leaders. The value of scheduled vs. unscheduled 1:1s.49:30 - Pair programming, especially in a remote setting52:00 - Hiring 2021 engineers in 2021 - managing that hiring pipeline64:00 - High leverage activities to drive high quality hiring 67:00 - How to think about attrition in engineering organizations? Is there a magical percentage to keep track of? Segmentation is key.72:00 - The best way to grow engineers and engineering leaders at a company This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.softwareatscale.dev

Welcome to Software at Scale, a podcast where we discuss the technical stories behind large software applications. I'm your host, Utsav Shah, and thank you for listening. Hey everyone, and welcome to another edition of the Software at Scale podcast. I have with me Farhan Thawar, who is... Farhan Thawar, I think I've pronounced that correctly. Yeah, Farhan Thawar. Yeah, and Farhan is currently a VP of engineering at Shopify. He came to Shopify via the acquisition of Helpful.com, where he was the co-founder and the CTO. Previously, he was the CTO of mobile at Pivotal and VP of engineering at Pivotal Labs via the acquisition of Xtreme Labs. He was named one of Toronto's 25 most powerful people.

And prior to Xtreme, Farhan held a bunch of technical positions at Achievers, Microsoft, Celestica and Trilogy. And sometime in the middle, I think he also launched Xtreme Venture Partners, which is an early stage incubator in Canada, which is doing extremely well from the free crunch based data I have. It's had 13 successful exits. So I just want to start with that's a lot of very, very inspiring or amazing experience. What's your secret? Is there any secret sauce about, yeah, just your philosophy or something?

Yeah, sure. No, you're not the first person to talk, to like talk to me and ask me questions about like how I chose these roles or how I ended up in these situations. I think the way that I like to go about making these decisions is I'm optimizing for something. And I never particularly knew what I was optimizing for until I sat down and tried to write down like, what is it that I'm optimizing for? And, you know, this, it, it actually started the, the, the discovery just started when, um, it was 2015. And, um, you know, I was thinking about leaving Pivotal to do something. I didn't know what to do. And, um, the, my, you know, ended up being my co-founder, Daniel came to my house and was pitching me on like starting this company. We didn't have a name. We didn't have really an idea. He just was, you know, he's an interesting character and he was pretty assertive and coming over and, you know, there's a whole story about how that happened,

but he came over and said, Hey, we should start a company together. And it was my wife. Yeah, that was helpful. Yeah. Yeah. The longer version of the story was I tried to cancel because my son was throwing up and he came to the doctor's office to meet me there anyway. So anyway, there's a longer story about what it means to be an entrepreneur and be tenacious because Daniel is definitely that. And he pitched this whole idea and he said, we should start a company. And then after he left, my wife said to me, you have to start a company with this guy. And I was like, what? Like she didn't know who he was. She just, you know, she saw his interactions with me. And she asked me this question. I

reworded it, but she basically asked me this question. She said, will the things you learn by doing helpful make you more or less valuable to the market? Meaning like, were you learning something there that would just be valuable because you're learning it? Even if, and she added this to the end, even if it fails after three years. So even if it, even if this, you do this whole venture, you spend all this time, energy and money, right. And it fails in three years. Is it still going to be worth it? And it's what a great question. Right? And I said, yes, she goes, then you have to do it. And so that caused me to actually sit down and write down like, what is it that I'm looking for when I choose these opportunities? So whether it was,

you know, I was, I launched, it was called Extreme University at the time in the in Extreme Ventures, or why I quit Microsoft to start to go to Extreme Labs, or why did I leave Pivotal and then start Helpful? And then why did we sell Helpful to Shopify? Like all these things. And it turns out, I just really want to focus my time on being around smart people and hard problems. Like that's literally the easy answer. And so it's not like I have some plan to say okay let me start a company and then get acquired and then become a VP or let me raise money work on this problem because it's like cool or like I had some previous ambition to start a company actually I didn't even have a previous ambition to start a company what I said was how do I learn at the

fastest rate possible and the only way for me to do that in 2015 was to work with Daniel DeBow because he's one of the smartest people I know. And so the only way to work with that guy is he's starting a company. Like there's no other choice because he was already doing that. So it's not like I have a master plan, but I do have a plan in that I try to focus on these things

to the exclusion of anything else. Actually, if my framework becomes false, if I'm not working on the hardest thing I can be working on with the smartest people I can find, I must resign and do that thing immediately. Like that's how my brain is wired and I get extremely frustrated,

bored if I'm not doing that thing. Yeah. And the converse is, yeah, as soon as you get bored or you get dispassionate of a role, like you're like, I have to just start something new. Because yeah but be careful there not start something new or do something smart people yes working on hard problems because but i'll tell you they're they're people who have a different framework right so daniel his framework was i need to start another

company because i think because for him he sees the future and in order to invent that future you have to start something, right? That wasn't my inclination. My inclination was, how do I be around smart people and work on hard problems? And those two games ended up colliding in us starting a company together. But he has, I'm assuming, a different framework than I do. And yeah, what you get out of that is as soon as you're not super passionate about a job, your performance could also suffer. And you get to skip all of that with this framework.

Right. And for me, passion comes from learning, making mistakes, looking dumb, because the only way to make mistakes, like do things new, right? As you try it, it's your first time doing it, you're going to look dumb. And I enjoy that to the exclusion of, hey, do you want to work on this other thing, which is probably less risky from like, what the world would say, right? Like, my parents did not like that I quit Microsoft, for example, right? They for them, immigrants to a new country, and then seeing their kid quit, will be considered, I guess, a less risky, like a, you know, very stable, less risky job, right? Well-paid and start and go to a startup like that didn't make sense to them, right? But for me, it is extremely risky to do that because I'm not pushing myself and learning

as fast as I could be learning. That's really inspiring for someone, like, especially for me, who's like early on in our career and trying to think, oh, what should I be doing next and all that? Yeah, but it's funny. So I think what's interesting about that is it's not that I had that plan, but I was always doing this, right?

So in school, I always took the hardest courses I could find because maybe I already pre-optimized for the idea that if the course is I maybe I already pre optimized for the idea that if the course is hard, I'll probably learn more. If the course is hard, probably sharper people are in the class, like all of those things I think I inherently knew, but it wasn't until I sat down and wrote them down. And I'm like, Oh, I never understood how I my mind worked. And I wrote it down. And I'm like, this is going to be doing. And there's a lot of value in writing stuff down because all of your scatterbrain thoughts

can get simplified. I agree. And I think there's one more advantage. It's that you won't get distracted by things that are not part of your framework, for example. So in writing down the things I care about, if some other opportunity came my way, that was confusing, right? And by confusing, I mean, it had it had factors that were not relevant to my framework. So imagine high paying job, great company brand, prestigious title, interesting location and perks.

Like all those things are not on my framework, but if it's not written down, they can confuse you. And it's definitely okay to value like a lot of those things. A hundred percent. No, no. I give the example actually that I had somebody working with me and their brother in the U S was in the hospital and they had to pay for the hospital bills. And so he came to me and said, I've written down my framework on what I need. And the number one was cash, not equity, not bonus cash, because he's like, I have to pay a

monthly hospital bill. And I'm like, great, let's work to figure out how to optimize this framework. It's not a judgment call. It is a, that's his framework. And guess what? He became a very high paid consultant for like a year on a contract so that he could pay his brother's hospital bills. It all worked out great. And now he ended up going to a startup that actually recently got acquired, but it's all good. But I mean, but that was the right thing for him. And I'm not making a judgment call. I'm saying, what is the right thing for you? And so I give people the framework as a way to figure out their own framework. That makes a lot of sense. I want to talk about an experience which I think is pretty unique that you have,

which is being a CD or like an engineering leader, like a VP of engineering at a really small company and at really large, multiple different companies, right? So you might have a CTO of like one large company, but then they've basically only been there. But you've seen a bunch of different companies and how engineering runs. So what is something that is like unique or unexpected about being in that role? It's a really good question. And until you asked it, I never thought about myself as somebody who I guess has been at those different scales, but you're right. The, you know, the thing that's surprising to me is that there is no single model that can work, meaning set a different, different way. There are many different ways to write software

and all of them can work. So there isn't a, a right way and a wrong way. And I think as I talk to many engineering leaders, sometimes you run into folks who have a preferred methodology or a preferred way of building things and that's their way and they're the expert in it. Amazing. And then you run into folks who have tried different things in different companies and the thing they tried at the last company didn't work at the current company. And so they have to figure out a new way. And I think that's more of my experience in that I've seen so many different things. And I would not try to take all the things that worked in the last place to the new place, even if the scale was similar. Right. So one, you know, one example was like Pivotal was about 2000 people when I was there. Shopify was about 4000, I think when I started.

So it's kind of similar scale, right? I mean, there's a doubling, but it's still similar. I wouldn't take many of the lessons and transpose them to either company because they're just completely different cultures and different ways of building. So for example, Pivotal uses a very religious form of agile, pair programming, nine to six,

sustainable pace, TDD, CI, CD, small teams. There's a completely very strict following of how to build software there, microservices. And then you go to Shopify and it's monoliths, Ruby on Rails backend. They have a less agile process in terms of like delivery. We still focus on a lot on automated tests. So there's some similarities. We have some pair programming, but not nine to six. So it's different. Both companies build software in an amazing way.

Both companies are successful. So I think that there's, that's probably the most surprising thing. And I probably, if you took me back, you know, 10 or 15 years in my career, I probably was searching for like the one way, like what's the right way to run an engineering team? And what's the right way to build the architecture of software? Like, I think that I was probably thinking there was one way. And over time and experience, I realized that, oh, there's lots of different ways that can work. And there are different cultures you have to take into account and different ways of building. And they can both be right.

And I don't think, I think the way to open your mind to that is to really try to understand how different companies run. And I also think that that means there's nothing, there's no wrong way. I think that's something that in our industry I do see where people say microservices is the right way or microservices is the wrong way. And I think that there's no wrong way. I think that there's no wrong way. I think that there's many right ways. So if you can't take what you've, you can't at least directly apply what you've learned at previous companies, right? And that makes a lot of sense. So could you walk me through like, what are the roles and responsibilities of like a VP of engineering? Like let's say that I start as VP of engine at Shopify tomorrow.

How do I become useful to the org? How do I know that I'm being productive and actually helping out in some way? Yeah, you're asking a very esoteric question that probably people grapple with in these types of roles. So there's a few things that I think are useful to think about in that framework. One is that at Shopify, what we try to do is make sure that there are specific initiatives and projects that the leadership actually owns end to end. So instead of just saying, hey, you oversee a bunch of these teams, there are projects and specific deliverables and deep dive spikes that I do that only I'm responsible for. So I can't just say, oh, that was that team or it was that person. Like I actually own them myself.

And you're accountable for the success of that. Yeah. So I have a beef with the word accountable. I like the word responsible. Okay. And the difference to me is there's actually a great set of interesting like research you can do on the Finnish school system and there's this guy named Pasi Salzberg I think I'm probably saying that wrong or Salzberg who revamped the Finnish school system and he has this great line where he says accountability is what's left once all responsibility is removed and what that means is that you want to be the person who is doing the thing and then be responsible for the thing, not just like a sign off person.

And sometimes in some companies, you can be accountable by just being the person who like signs the thing at all levels, you should be able to figure out the unique parts of the initiative that you can own that makes the whole better than it would be without you there. And so at Shopify, we really do focus on leaders having responsibility for some aspect of the final product. And I think that's really helped us make those spikes. And it's why you'll see, you know, one of the incredible things about Shopify is the senior leadership team, specifically Toby, the CEO and JML, the CTO commenting on PRs, right? Like writing codes still. And not to say that that's a, like, a, a, like they're micromanaging some aspect of something. It's more about understanding things to one level, level,

level deeper than you normally would. And it's something that I don't think I would be able to learn anywhere else, but Shopify. So that's one part of it. The other part of it is of course, being surrounded with great people in the company that you hope you can unblock, right? I don't think I have any special sauce that being around me makes those people better, but if I can listen to them and understand what the issues are and unblock them, remove barriers, remove administration, help bring other smart people to the table, show them something like, oh, I know you're focused on this, but you know, over here, we're doing this other thing and it might be useful to put together with this. I know you talked about this in your podcast with Bharat around when to rewrite something. Like if you can

bring in an opinion and come to those discussions with an open mind and help somebody see something maybe they didn't see before, I think that's part of making that team better. Got it. So as a leader, you're focused on a few key projects and unblocking the rest of the organization. That is a good summary. I would take it, I mean, since this is an engineering podcast, let's take it a few levels deeper.

If I can help engineers get leverage, right? So for example, you know, I wrote a blog post about a year ago called React Native is the future of mobile at Shopify, right? That was a way for us to gain engineering leverage because we've now made a decision around a technology framework. We're investing into the framework. We're hoping that by making a choice like that, that all of the mobile initiatives at Shopify will become enhanced because everyone's working on the same stack. We can have foundations teams building out core capability for the whole company. We can work on open source. That's a way for then engineers who are working on the thing. And I'm not like in the project, but because of that decision, now a bunch of things can get

unblocked and people can move faster. That's another example of helping people get leverage. So I would say unblocking is part of it and then gaining leverage is the other part. How do we make people feel like they can be more productive because of the work we did versus just that they have to build everything from scratch every time? Okay, so that brings me to one more question then. So I'm guessing that there's above a thousand engineers working on mobile at Shopify, right? I don't know how to count that because now that we've moved to React Native,

you can imagine many, many more engineers on the front end who are React could contribute to mobile. Yeah, but it's basically a fairly large number, right? Yeah, it's a large number. Yeah. So how would, what kind of decision process do you have to go ahead and say, this is what we're going to do? Like, how do you feel confident when you do something like that?

In, you know, I know you're fairly young in your career, but as you get more experience, what you realize is that you're never going to get a hundred percent information of the information that you need. You're going to get inklings of information. You're going to, in our case, we're going to prototype and build things from a spike perspective. It spikes into different areas to say, Hey, this could work. And then you're going to look at curves, like where is the world going? And so what we saw, I mean, here's a good example, right? Outside of Shopify in 2007, a couple of my friends started Extreme Labs after the iPhone came out, right? They saw the iPhone

and they were like, this is going to be big. And to be honest, even though I built lots and lots of mobile now, I didn't see it like they saw it. I had a Windows mobile phone and I was into like computing and mobile, but I never thought it would be like it is now. But they saw it and they were betting on the curve. They saw, hey, if you can, you know, the iPhone, of course, BlackBerry was out and everything else. But what they saw on the iPhone was that Apple somehow managed to put an entire computer, like an operating system, like a full OS into the phone and put it in your pocket. And then from a UX perspective and UI perspective, you're able to navigate, like navigate through the desktop version of the web. And they were like, this is going to change everything. So they're betting on the curve. So similarly, similarly for us, we look at some of these core technologies,

and we say, where do we think it's going? Where's the world going? Clearly, much of UI and UX was going, you know, UI engineering front-end development was moving towards React, and it was a better way to write UI. And we started seeing mobile moving in a similar direction with a similar engineering stack. And we felt we could make a similar bet to say, you know what, actually, when we have to do something complex and deep and close to the metal, we can probably write

that native, but the UI should write React Native. And then we saw prototypes being built, especially on hack days when we have three days to build whatever you want. People who had never written a mobile app were writing mobile apps in React Native like in two or three days. So it was, we were betting on the curve that it would just get better.

And then one last thing that we do at Shopify, which I haven't seen as much at other companies is that if we think it's risky, we run directly at it, which means whether it's Ruby or Ruby on Rails, React, React Native, we don't just keep it at arm's length and say, okay, it's risky. We're going to bet on it, but it's over there in the corner and we'll just stay up to date with it. Instead, we build key relationships. We become core contributors. We go to the standards committee

we run the working group we like you know i have full-time people working on the open source like we we go all in and that allows us to make the bet with eyes wide open because now every technology has skeletons in the closet but now we know what they are we know what the limitations are and we can either work around them because we know them so well, or we can fix them because we're part of the core team that works on it. Got it. So basically a hundred percent commitment once you've made a decision, a particular way in terms of resources. Yeah. Yeah. In terms of resources, in terms of how we think about it now, there's a, you know, I have this in my calendar, like once a year, I kind of look at the decision we made and say, is it still this, like there's, you should revisit, but not every

week, not every month. Right. Maybe once a year is a good time. I actually, when we made the decision in 2020, I still, you know, people go, what about Flutter? Flutter is amazing. Like, I think Flutter is amazing for the Shopify stack. React Native makes more sense, but Flutter is also amazing. But I do have things on my calendar. I'd be like, look at, take a deep dive in Flutter. What's changed? What's going on there? Are there reasons to, you know, reevaluate? Should we do a prototype? That is always there, but not weekly. Yeah, I love the idea of just like a yearly check in on some of the key decisions and what the consequences were. You should have one for your framework. Check every year. Am I working with the smartest people I can find? Am I learning as fast as I could? Sorry, that's my framework. But whatever somebody's framework is, is the thing that they should be looking at. Got it. And I want to go again one level deeper on that. So you made that kind of decision, which is we're going to go with React

for mobile. I think with all sorts of different kinds of org structures, you can get into like different traps. Like if there's no key decision maker or there's nobody responsible, as you said, people get into group think they don't want to take like a hard decision. How do you like structure your organizations or your teams in a way where you can make these like, so these hard decisions and without maybe getting full consensus? Because if you had to wait for everyone to sign off on this, you're probably not going to get that.

Yeah. So it's similar to the information, like the perfection of the information. Like you're never going to get to 100% perfection of information. You're also never going to get to 100% commitment from a team on a decision like this or many large company decisions. So all you can do is try to get as much information as possible

and spend as much time with people getting all the context. But I would say the thing that really can help you here is building some of the components, whether it's a spiky prototype or launching something, right? Like we made the decision to move to React Native in many ways after we started building like apps and launching them into production on react native and we're like wait a sec we're seeing um very good velocity on what we're building we can also build cross-platform we're seeing high code we were seeing high code share which actually turns out to be not the goal and or maybe a bad thing if you have like 99 code share maybe you have to be a little bit lower because you want to take advantage

of the native capabilities um we saw very encouraging crash rates, like very low crash rates. And then we're like, okay, we're feeling good now because we're getting data. We've been collecting it, but we're now in production. And so you're right. You're never going to get 100% commitment. But again, we looked at the curve. We talked to lots of folks. I did lots of internal AMAs and talks and Slack discussions and documents and Google

Docs and presentations. And I met with Facebook once a month and like came back with my findings. I talked to other companies. Like we did all the things you could do knowing where we want to end up was make a decision. And then when we got all the data, we said, this looks like the right decision. And then you make a bet. And the way to mitigate that bet I talked to you about, which is that we made investments.

We sponsored a company to do open source. I hired some of the core contributors. Like we said, we're making the bet, but we're mitigating this way. So you're going to have people who don't agree. And so, you know, we gave people pathways, right? We said, hey, you're in mobile, cool. We're still going to need native,

but you should still probably learn React Native, but we will, you still, we're still going to need some native components. Please stay close to the code on the native side because we're going to need that. You could potentially use this knowledge to move into a front-end role. You can move into a back-end role, which is outside of the mobile stack.

You might want to embrace this. We'll give you training. We'll pair program with you. So we kind of opened up the options, but we said the reason we're doing it is for leverage at Shopify. It may not be the right decision for other companies. They might want to use native. They might want to use Flutter. They might want to do something else, but for us, it made sense. And it did follow a similar trajectory to how we

made decisions in other areas, right? Like how did we decide to close our data centers and move to the cloud? How did we decide to continue to stay on Ruby on Rails, right? Like from the beginning. So we have a history of making these leverage style bets and it's like a campaign, right? You have to talk to people, you have to get their feedback and slowly with data and moving, putting things in production, right? Like we have this line, code talks, bullshit walks. That allows us to mitigate the risk on making these kinds of decisions. And then, like I said, once a year or so, do a deep dive to see if there is something that, some new information that has caused us to make a change. That makes sense.

Do you think an engineer like bottom up can drive a decision as big as this? And if they could, how do they go about doing something like that? Oh, you're so young yet so many good questions. Thank you. So that's a good question. I would say that there are two types of these initiatives in a company. Some come top down because the person who is closer to, who's looking across to make this decision is actually seeing things from multiple parts of the organization, maybe even multiple segments of, in our case, merchants. And so it would be hard for someone at the high context level to be able to see that because they're not looking across such a wide scope. So those are some kind of decisions. And then other kinds of decisions are bottom up. Right. So for example, we've had like, I talked about hack days, right? Three to four times a year, we have, we take three days and everyone's allowed to build whatever they want all the way up and down the company. And they, those things, the goal is to ship something. So you can,

it could be a prototype it could be um you know a new process lots of things can be built there and some of these hack days projects turn into full company-wide initiatives because of the things that they uncovered and so there are many examples of that um at shopify well, where it starts off that somebody goes, hey, have you seen this? And because it was a hack days, we have like voting and people start seeing it more. The leadership starts seeing it like, wow, this is I never thought this was possible. Let's put let's let's let the hack days team continue on it. So let's keep those people off of their projects, continue on this and let's see if it goes somewhere. And there's really good examples of this turning into company-wide initiatives that then get like sponsorship and budget and hiring that can turn into a full-fledged initiative. So your question was like, how do you do it? I think the, you know, I'll go back to that same line, which is like build a prototype and show why this prototype is so much different than the way we've been building things and where it could go. Now, one of the things, you know, I'm running the CTO office now at Shopify is in my mandate is to encourage more of these like deep dives into some of these areas where either someone has a bottom up idea or we have an idea.

We want to fund for somebody to go spend some time in some area to figure out these exact things. So it's like an internal accelerator in a sense. You have a few people with a potentially promising idea and they need some more time. And you just say, here, work on that instead of your day job. Right. And so, yeah, I don't like the word accelerator so much because it sounds like you have like some Y combinator inside your company. But it's more about specific deep dives. And I'll likely focus on like technological deep dives versus like product deep dives.

But I can see that also being something we do where we want to test something quickly, get something to the market and see the response or learn, right? Again, it's all about learning. So what can you build, launch, and then learn from? It doesn't matter if it's successful as long as you've learned something from that experience.

Got it. So one thing that struck out was like a bottom-up presentation could be something like a presentation that could get traction and then leadership has to make sure that good ideas are surfaced and actually funded. And that's how both of them work together. Not presentation, prototype.

A prototype, yeah. Yeah, so you have to be careful because you want to make sure it's as close to the fidelity of the thing you're building as possible. So like a high fidelity experiment is one in which the code is written to exercise all of the hops you have to do

to understand it. Because usually when these things happen, it's, it's a combination of things coming together. And in order for those things, combination of things to come together, whether they're disparate services across the company, whether it's integrations with third parties, you have, you want to wire them up in code, not in a Google doc. Right. That makes sense. Yeah. And so, and then you want to show like, Hey,

this thing works. And I think that's more of it. But yes, it's 100%. It's 100% possible. And like I mentioned, hack days is one way, right? And this happens all the time in any company. But I've definitely seen, you know, at Shopify as an example, someone will ask for something. And six months later, the thing is not built. And somebody builds it in hack hack days in three days. Right? Like it's, it's, it's not happens to any company. It's amazing to see because just what focus and time of only a few days can do. And the motivation.

A hundred percent. They want to do it because, and sometimes it's because there was their own passion or sometimes it was because somebody said they it's useful for the company. And they are like, I know how to do that. So in a lot of engineering organizations, or I should say most engineering organizations that are bigger than like three people or like five people, there is like some percentage of engineers or people just working on shipping products directly to users, right? Like working on new product features. And then there's the other part, which is like platform in a sense, where they're working on making the other people in the organization more productive. So this could be, you know, like internal test teams or like CI, CD deployment systems.

How do you think about investment? So it's clear that you think a lot about investment until like once you made a decision to react, we have to make sure that we have a few people on the standards committee and all that. How do you know that we're investing enough or we're not investing enough? It's a good question. I don't think you ever know, right? Meaning you're never going to know if you're investing at the right amount. And that could mean over-investment or under-investment.

Like I'll give you an example actually from my startup, Helpful. We all came from Pivotal. And so when we built it, we built Helpful, it was an asynchronous video messenger, for example. That was one of our products when we built it, we built Helpful. It was an asynchronous video messenger, for example, that was one of our products. We built it using a microservices architecture. We had like an orchestration layer. We had built it in quite a robust way. And when we had abandoned the product and our team was, we grew to about 25 people, so it wasn't that

small. When we decided to abandon the product and it wasn't that small when we decided to abandon the product and move to a different product we left we left like the helpful video messenger like running like we didn't turn it off we just like let the services run and then we worked on to a new um a new project i think six months or eight months later when we were um you know i think maybe i don't know what i was doing looking at the bill or something for our cloud hosting or something i was like wait a, like this helpful message is still running. And not only was it running, there were companies using it and those companies, their usage was growing. And what was, what was my first response was like, somebody looked at me

and said, wow, we really built it like, well, like that's insane. Cause like, you know, usually you think that as a startup, like it should die, right? Like something should break. It should like go down. Like a service should like not restart. Like something should happen, run out, even just run out of this space for a log, like nothing happened. And it was totally running. And what I said was, Oh my God, we wasted so much time. And the engineer looked at me and said, what do you mean? I go, we overbuilt it. Like we spent too much time architecting it correctly because we were trying to learn something. And we spent too much time on our testing framework and microservices architecture and orchestration that like we built like a real product, which is like it is amazing. And you have to be proud of it. But at the same time, you have to look back and go, could we have shaved off like three months and learn the same thing

and like then moved on to the new product earlier? Like that was my takeaway. And so it's rare that that happens because usually you're moving quickly and everything's falling over as it's like scaling. But in our case, like in this example, we overbuilt it. So it is hard to know if you've got the right amount of people.

Now at Shopify, we do have like a developer acceleration team and that team does try to ensure that the developers are as productive as possible, removing roadblocks, making their CI quick, making their development environment quick, right? We have a tool called dev, like dev up and it just sets your whole environment up. it's amazing there are there's a whole team that focuses on that i would say any time you can add resources that to that team along some curve and it improves the productivity of everybody else you should do that especially at the size of shopify and so we do invest robustly into that i remember um chatting with kevin scott he's the cto of mic. And when he was at LinkedIn, he was telling me something like, I think it was 45 to 55% of the engineers were working on my platform.

Like keeping it up and platform. That's a surprising, but still awesome number to hear. Yeah, it's an insane number. And I understand what was happening, like for that to be the case. I don't have off the top of my head, like a percentage, but we have a significant amount of people working on making the engineering experience of Shopify. And then don't forget, we have a platform. So all the outside developers building on top of Shopify, amazing. And I don't think anyone would say, could you add more people to that team? Yes. Are we adding more people to that team? Yes.

We can continue to do that. But there's a, there is a an investment that we have to put into that. And then there's investment we have to put into building things for merchants directly and then an investment into the you know, into all parts of that, into that ecosystem. I don't think there's like a ratio that like, I don't, I def, I definitely don't have like a, a ratio that I'm like, Oh, I've got to make sure I add like one engineer there for every 10 over here. Like, I don't have like a ratio that I'm like, oh, got to make sure

I add like one engineer there for every 10 over here. Like, I don't have that in my head. Maybe somebody does. Yeah. Is there a way you can like measure developer productivity? Everybody talks about measuring and how hard it is, how impossible it is. Is there something that you've come up with or you've given up saying, oh, it's just impossible. We just have to go by like surveys or something. Yeah. So I've definitely not given up because I spend lots of time thinking about this and reading as much as I can about the state of the art of people trying to figure out developer productivity. I don't think there's a good answer.

So here's the state of the art. The state of the art is there are a bunch of things you can do to give you insight into places you should look. But there isn't much more than that. Like there isn't a tool you could run on a GitHub repo connected to JIRA, connected to, I don't know, Google Doc, tech designs, connected to like rescue time, which shows you what you're using on your computer. Like there isn't like some magical formula. It's like, oh, my God, we should all work like this person. But there likely is interesting data from some of these, from, you know, there's a bunch of new companies that are trying to do this that I think might give you insight into how developers are working. I agree

with you on surveys. I'm a big fan of like send out a survey and try to fix the things that are time wasters in a company but I don't think that there's um like here's a good example I remember hiring um a salesperson and when they came to the interview they brought their like in Canada it's a t4 but in the U.S. it's a w2 like their tax slip and they're like oh here's my tax slip for my last company I'm like I'm like what are you their tax slip. And they're like, oh, here's my tax slip for my last company. I'm like, what are you showing me? They're like, well, I want to show you I'm a good, this is the level of salesperson I am.

Like that was their way to show that they're like, I know how to close deals. Here's my commission check, basically, right? And there's probably lots of things outside of a commission check that help make a great salesperson. But in many ways, that's a closer metric to how good they are than something we can find on the engineering side. But I'm a big fan of, like we talked about prototyping, I'm a big fan of removing proxies

from some of these measures. So for example, I don't think like number of check-ins, lines of code, like any of those actual, like what you could pull from a GitHub repo metrics are useful. What I do like to see, for example, is demos, like prototypes. Show me something that takes our infrastructure end to end

and puts it in front of a merchant. And then let's see what we can learn from that. And if you show a demo and I can see the merchant value, then you're kind of engaging from a developer perspective, all of the pieces that makes that thing valuable for a merchant. So for example, you've written this code being written, there's design and UX and UI being thought through there's product prioritization being thought about there's data being discovered around, like, what are we tracking and how are we going to measure if this is working? So product prioritization being thought about. There's data being discovered around, like what are we tracking and how are we going to measure if this is working?

So it's kind of the end-to-end putting it together. And great developers will be able to work with sharp folks in all those different areas, put it together, and then show you like a spike through those layers. That's valuable. And then if you look at it week to week or every two weeks or whatever it is, you kind of can start figuring out,

is this like, is a team performing well or not based on the feedback that they're getting and how it's going on those like bi-weekly sprints, for example. So I'm a big fan of the visceral, what is the highest fidelity thing I can see? I mean, and just to show you how important it is, if you change one thing,

if you change the demo to a screenshot, you will lose a lot of fidelity, right? You'll lose everything to do with animation and transitions. You'll lose everything to do with animation and transitions you'll lose everything to do with performance you'll lose every like you'll lose so much context just by like moving one direction away which is why if you continue to move directions away like look at github check-ins or look at a status report like you end up continually losing fidelity and um to the point of like maybe missing a very key insight. Yeah, that makes a lot of sense. Like GitHub commits are so far away from

what did we end up shipping to the user? That's exactly right. And one person can have 100 commits in a day. Another person can have one, and there's no way to know which one is doing better than the other, including lines of code. One person can ship one line of code, but it's the right line. The other person can ship a thousand lines. Like there's no way to know. And that's why I'd rather try to figure out from the value that we're delivering to the user. Now, the other side of that is what about all the platform work api design what about um all of the non-functional requirements maintainability flexibility so it that's why that's why it becomes a hairy problem

yeah there's there's no clear answer to this you just have to try like hiring the best people i think and and speaking of proxies like this goes back to like a podcast of yours or an interview of yours i was listening to sometime back where you had a debate with someone about like flat versus hierarchical organizations and you mentioned that at one time like you had 120 direct reports and that's mainly for you that packet loss right skipping that packet loss getting information directly but how do you manage something like that? Because whenever you see people saying, oh, I have too many direct reports, I have 10, I need to scale myself. What are people talking about? And like, how did you scale yourself in that scenario?

Yeah. So I'll take you back to that time. Like, you know, there's every company is different. And what we were building there was a system to, and this is like Streamlabs, right? the right number of direct reports for a leader? They're like, oh, yeah, maximum 10. And then you should scale and hire a manager. And then maybe the manager can have like, you know, 10. But then you can have maybe five managers to a director. So they have a scope of 50. And so I started researching this and I was like, well, let's just try it. As we were hiring people, let's just like not hire people we don't think are necessary to the process at the time of the scaling of the company, right? And the interesting thing about Atrium Labs at that time period was mobile was new, right? The company started in 07 and the

iPhone came out in 07. The App Store came out in 08 along with Android. So it was like early mobile. So you couldn't hire somebody with 10 years of mobile, for example. And the company was new. So we got to try all kinds of experiments. And so as we added people, we just didn't hire any managers. And the way I dealt with it was the following. One was, what was the goal of the manager, right? I tried to figure out what is the goal of the manager? And do we need somebody here? I'm not saying I was like, they reported to me, but I don't think I was their manager. They all just reported to me, you know, in the org. But I tried to figure out what is it that the manager would help this person with, right? Okay, so one was I'm blocking, right, to get stuck. One was what are they going to work on? Another was how career development,

how do they get better at what they're what they're what they want to get good at another was how do i make sure they feel a sense of purpose like um actually working on something that's bigger than themselves um and then you know if you think of like the dan dan pink drive book mastery autonomy and purpose autonomy how do i how to make sure that they have creativity in their role to you know do the right thing, given the context that they would have because they're much closer than I was. And what we just, what we designed was a system whereby we had a single priority queue of tasks using Pivotal Tracker with which a PM would help them with.

So they knew what to work on. They had autonomy because I had 120 direct reports. I couldn't tell anybody to do any specific one thing. They just kind of within there, within the framework, right? So we use the very similar model to Pivotal. We worked nine to six. We were doing pair programming. We were shipping every week to the client. We were getting, doing demos every week. So all these things together with a single queue of work allowed engineers to be quite productive, unblock a lot of their own things, get mastery because they're a pair programming purpose because these mobile apps were going to go on millions of phones.

And they didn't need a manager for that. Like in many instances, what I would do is I would do a one-on-one and I would sit down with somebody and ask them about like, well, how are things going? And like, we can use this time. And they would say, can I leave now? Like they were, they were ready to go back to work. Cause they're like, I'm pairing with somebody smart. So I'm getting better at my craft. I'm an economist. I have creativity there. And then I'm working on something bigger than myself. So they were like, what do I need you for? And so I was like, cool, like they don't need me. And so what I did was I didn't do any scheduled one-on-ones, but what I did was because I didn't have one-on-ones, I had much more time to be around. And so I was always

there in the office and I was there for the unscheduled one on ones. But when people came to me with crazy problems that I was available to take them. And so then I would say I would, so I did tons of unscheduled one-on-ones. I just didn't do any scheduled ones. And the other thing I think was paired was just, I just got lucky is I have a good memory. And so I could remember like, I don't know how I did it at the time. I was definitely younger, but I could remember obviously everybody's don't know how I did it at the time I was definitely younger but I could remember obviously everybody's name but like what they were working on their skill set what project they were on like I could you know even to the point of this is funny I could

I would read the I would do pay actually I ran finance and HR and recruit like a bunch of things in that company including engineering I could look through the Excel payroll sheet and like spot like salary mistakes. So interesting. Yeah. It was just, it was, it was a weird, uh, interesting time. Very, very intense time. And a lot of people will actually, here's the other, the last thing I would say, um, outside of that, like, don't try this at home. Like people should figure out what works for their company. But, um, many engineers messaged me after and say, that was the defining moment in their career. Like they didn't come back and go, oh my God, it was so horrible.

And like, they came back and they're like, I really loved working that way. Yeah, like the autonomy and lack of a manager. It sounds pretty much, but you also get the career development. I think that is the one thing which I think about like in my one-on-one. So like our schedule one-on-ones just overrated.

Do you think if an engineer has everything else worked out? I think that there's value. There's a value equation you have to figure out. So I'll give you an example. When I was at Microsoft, I had a scheduled one-on-one with my manager every week. And what I realized was we taught, I mean, and when I was there for three years, so let's say I did 150 one-on-ones. And then there were times when something would blow up and I would knock on his door and be like, I need to talk to you. And what I realized after leaving Microsoft, because I went, you know, went from Microsoft from the startup world and I was evaluating all these things was which was more valuable, the scheduled one-on-ones or the unscheduled one-on-ones and i realized then i was like at microsoft for me it was the unscheduled one-on-ones it was like i had like a fire burning and i wanted to talk to my manager

and the scheduled one-on-ones while they were nice like half an hour to one hour chats about like life i wasn't like super i didn't find them as effective or useful as the other one-on-ones and so that's why that's why i went down this route of like making the doing this experiment of saying, what if I made, what if I prioritize unscheduled one-on-ones? Now they can be super useful. So for example, if your scheduled one-on-ones are a place for you and your, your direct report to have a conversation about any, it's your direct reports agenda and they can bring up career development.

They could tell you about their projects, unblocking, hiring, like all sorts of things. It can be super useful. Just for me at that time, it wasn't useful. And I can tell you at Extreme Labs, people were like, did not want to be in one-on-ones because they were like, I know what to do. I have a person waiting for me to pair. I need to ship this to a client. I want to get feedback and I'm learning at a crazy. And I'm working on like Facebook or Twitter or Instagram.

Like, let me go. Yeah. Is there a difference here between like senior engineers and junior engineers? Or do you think that doesn't matter at all? Okay, so that's a broad question. So I think that, I think number one, there's a big difference between senior and junior. Like in this scenario, I guess. Oh, in this scenario.

So I think, you know, at some point you know i was at extreme labs and we got acquired right but i was there for a total of like six and a half years at some point you do have to put in like a career development framework so that people can realize their own ambitions like whether it's going up the people manager track whether it's going up the people manager track, whether it's going up the individual contributor track. And so we put those things in, we still had a very low, like number of just pure managers, because we felt that there was still a lot of value to having a system that allowed you to like not have a manager, for example. But I don't think it was the senior or junior managers who were the ones that split or those numbers of engineers didn't make the decision to bring in management.

I think it was that as we scaled, we were doubling every year. And so we got acquired when we were 350 people and there were still not as many managers as you, as you would imagine. But that wasn't the driver. Like the senior managers were learning as much or as more from that system than the juniors, because, you know, take two senior people and pair. I mean, there's a great article in the New Yorker, I think about Jeff Dean and Sanjay Gamawat pair programming. You're like, there may be two of the most respected engineers in our field. And guess what? They pair program whatever every Friday.

Like, so they're learning from each other just like any two engineers would learn pair programming. Yeah. Yeah, I haven't done as much pair programming as I should have maybe. Like, do people still like pair program well in like a remote environment in your experience yeah so remote pair programming has been a thing for many many years i think at least over 10 or

so years with all the tools that were available um recently some very good tools have come about like at shopify we use one called tuple which is a tool that allows you to not only pair program but also adds in the availability of like an observer. So somebody can actually be a third party observing the pair and it's got all the great audio streaming and you can see each other's screen. Like it's, it's actually very well done for a remote world. We were using it even before we went fully remote. But now it really shines as a way to like pair on something and work through something.

So there are and there are not it's not the only one. There's lots of tools that I think that's not that's not the blocker. The blocker is. Do you feel like you can put yourself in the environment to work with somebody synchronously through a problem? That's the biggest blocker in remote. As you think about async versus sync, pair programming synchronously will be the timing blocker for you. That's pretty interesting. With remote, there's all these different time zones and all of that. You have some sort of core working hours where maybe that's when you pair

program. Yeah. There's a couple of different ways so um we have time zone affinity so we have like a north american time zone european time zone aipac time zone and if you're gonna pair program you're likely pairing in one of those zones the other thing we have is something called pairing sidekicks where we have like a bot in slack that automatically schedules you for i think one hour of pairing per week if you're in that Slack group. And for me, it's amazing because one hour a week is exactly the amount of, like, you know, it keeps me close enough to, like, see the fidelity of what somebody's working on and it gets people to know, like, me as well. And I'm a huge fan.

Are people, like, worried, oh, like, the VP is, like, observing me pair programming? Like, have you seen that? Probably not because, like, I'm not that smart programming? Probably not, because I'm not that smart. That's pretty casual. Yeah, I'm not that smart. So they're like, some random guy is going to ask me dumb questions for an hour. Like, cool. Yeah, that sounds good. I mean, talking about talent in general, Shopify's biggest announcement that I've seen pretty much everywhere is Shopify is hiring 2021 for 2021. I'm just like curious on how do you achieve that scale? Like how do you set up like a recruiting

pipeline that can actually get you like so many engineers? Like what's like the secret sauce or like what's something that you've learned so far in this process? Yeah, so we're in the first month, so it's still pretty early. But I mean, you're asking questions that are pretty generic to all of recruiting, not just Shopify, but like, I think that there are many things that people can do to make their place, like their company, a place that engineers want to work at. Right. So, and whether it's 2000 engineers or you want to hire 10 engineers, like, I think the things are very much the same, right. Can you create an environment where engineers are learning, growing in their field, feel like they're working around very sharp

peers, like all of those things are true. We're lucky at Shopify in that we're very mission aligned, meaning we really believe that there should be more entrepreneurship in the world. And so we fight every day to reduce the friction to becoming an entrepreneur. And that resonates with a lot of people, right? Not just like at Shopify, but outside of Shopify. Like when I, when I meet people inside Shopify, I ask them like, Oh, how long have you been at Shopify? They'll say, I've been on Shopify for four years, but I've been at Shopify for two years. I'm like, so they, they're merchants or the opposite. When people quit Shopify, many of them quit to become a customer of Shopify, like become a merchant on Shopify.

Like if that's, I haven't seen that before. Like it's very rare. You leave a company and you become a customer of the company, or you come in and you come to the company because you were already a customer, like, like a very adamant customer. And it allows you to like create your own light, like your own livelihood that you can, you don't have to work for anybody. You can just, you know, you can create something out of nothing, be an entrepreneur in the world and live according to your own terms because of that. So there's a mission alignment there. And I mean, to build a pipeline, there's a, there's a lot of different things. Like it's, you know, there, there's not like an overarching strategy, but, you know, we are well known for having great engineering. We have built, you know, I think our CTO, JML, tweeted this out in March, like when COVID hit, we were seeing Black Friday, Cyber Monday level traffic every single day. So it's showing you the engineering scaling challenges we have to go through. Like I was talking to a colleague the other day, not at Shopify, and they're like,

isn't all this scaling stuff solved? And I'm like, not when you're our scale, right? Like we are, the line I said to him was, we are on the edge of our seats. Like it's amazing what we're able to do to grow as fast as we have to grow to keep our merchants having a great experience. And it's like, it takes great

engineering and sharp people and people want to be part of that mission. So now on the tactical side, yeah. How do you meet and interview and see if all these people are a fit? Yeah. There's, there's a combination of like, how do we make the process as efficient as possible? How do you reduce the latency of a particular person in the pipe so that they, you know, it's, it's, it's only hopefully a few days between like the time that they meet with somebody and they figure out if the Shopify is a fit for them or not. How do you reduce bias in interviews when people are here? How do you make sure they're a fit? Because people might come in and then realize, oh, actually, you know, I just met somebody

today, actually a candidate who worked at Facebook for only five months. And I was like, why is that? He goes, well, when I got there, I realized it wasn't a fit. I'm like, okay. Like, so that happens. So how do you do that at Shopify? How do we make sure there's an alignment even after they come in? Because interviews are not the best predictor of performance, but real work is.

And so is there a way to get them up to speed quickly? Onboarding. We're spending a lot of time on, like, fundamental technologies we use. How do we get people up to speed how does Shopify work giving them context so they can get productive quickly so they can feel like they understand what we're about and what we can understand what they're about and see there's a mutual fit and if there's not cool like let's figure that out quickly and then if there is great how do we get them productive as quickly as possible? So I think that there's lots of like pieces in the funnel we can optimize. And then there's pieces on like,

how do we level up the existing Shopify leadership? Right. So if you're hiring 2000 engineers, guess what? It's all levels, but it also means there's an entire cohort of people here who are getting smarter every day, who are leveling up there's promotions, right? So somebody come in and doesn't necessarily have to be somebody's boss who's already here, could be the other way around. And we're spending a lot of time on that. Yeah, that makes sense. And that's like a number, like, is there just something like different you have to do at like 20 people or 2000? And you mentioned like there's mission and there's engineering brand and definitely Shopify has advantages but like what is something like unexpected you know that's like completely

different at that scale of recruiting so I mean here's an example that's something you probably wouldn't do at a startup you might that that bigger companies could try and I think we'll do some experiments here too is like automated candidate assessment okay right so if you're a small startup and you wanted to figure out somebody's coding chops, you might have them come in. They might submit some work. They might show you their GitHub repo. There's a bunch of things you could do.

You could even have them code in the interview. As you get to a larger scale, some companies, and again, we might experiment with this, have looked at automated assessments. Are they useful? Is there a bias? Is it actually a problem that's relevant to Shopify? Does it use the language we use, like a framework we use? Like, do we care if somebody understands how to code at this level? Or are we looking for something else? Does it help us with diversity? So that's one example of something you might use at a bigger scale. The other thing is, like I mentioned, diversity is, how do we make sure we find the best people? If I put a job posting on LinkedIn,

I will get some dimensions of diversity. Then if I put it on Twitter, I'll get different dimensions of diversity. I put it on Facebook. If I put it on Instagram, like, so we have to make sure all the pathways are open to get the best people. The amazing thing about Shopify is that we have over a million merchants and they are completely diverse, like all sorts of people. So shouldn't our employees be all sorts of people? And if you only advertise on like the Shopify website or one of those services that you're not going to get everybody. And so from a scale perspective, we try to do as many experiments as we can at many, many different places to ensure that the pathways are open for those folks. And then of course,

when they come in, are we making sure that our interviews aren't unconsciously biasing folks for whatever reason? I mean, this is a complete aside. At Extreme Labs, right, we hired like a thousand people and we famously almost didn't do any interviewing because we mostly just tried to look at people's work. And that really helped us on the diversity scale. So I think there's lots of things we're going to try this year. And we're excited about, you know, meeting and bringing in as many sharp folks as we can over the year. Yeah, I think that makes sense. Like, cause a very easy trap to fall into is like hiring only through your networks.

100%. Only get people who are mostly like you, right? Yeah, and it's a double-edged sword because if, you know, in many companies, referrals are the best source of candidates. But then at the same time, if the referrals come in and they're all of a particular diversity dimension you won't in our case we won't be building things that our merchants need

because we don't have all the different kinds of merchants right actually here's a good example when um when we were building helpful i mentioned we were building an asynchronous video messenger like a snapchat for work i was at a conference and i was talking about it and somebody put their hand up and said hey how do you make sure because it's video that you're testing for all sorts of races, because as you're doing like filters for Snapchat and our case, we have work filters. How do you make sure you're not making a mistake there? Because you don't have a, like, you may not have all the people inside your company that represent the different dimensions of diversity. And I was lucky to be able to say, actually, our team was super diverse. We had like more women than warm women engineers than men. We had people of all races and

backgrounds. Like, so our, our own team using the thing, like it was something I didn't even have to worry about. Like, it wasn't like something like that. Maybe another company would have to worry about because I was lucky that we had all kinds of folks in the company. Now we didn't have everybody, but like I said, we got up to 25 people, but it was diverse enough that we never ran into a situation that we had, we had not thought of something because of our lack of having a breadth of different types of folks. And that's something that I was, you know, and it was funny that the questioner said, good answer. I was like, it is a good answer. Like we just happen to have a diverse team. Yeah. There's a very famous example of like YouTube

not working well for left-handed people because just nobody on the mobile team or on the team that time was left-handed. So the video would like upload upside down. Oh, geez. Yeah, and very similar. I heard a similar thing about, I think, Facebook Android. Like everyone on the Facebook mobile team is iOS or something.

Like, or maybe it was at some point and now maybe they force people to use Android. Okay, yeah. So that covers the recruiting center. Put your open positions on as many different platforms like LinkedIn and Facebook and Twitter so that people of all different kinds, like we're looking at everything, all of these different platforms,

all of them apply instead of just optimizing for the Twitter people or the LinkedIn people. Yeah, I mean, I would go farther. I mean, it's not just about job postings, right? Because we know, actually, here's good data, right? We know that, for example, if you've got 10 requirements, if a male, a man sees that those requirements and three of them match, he'll likely apply. But if a woman sees those requirements and is missing three, she won't apply.

So job postings are one way. Events are another way. In today's world, they're all like virtual events, virtual career fairs, talks, podcasts like this one, writing articles, engaging. I try to engage as many folks. We have this thing with the, I forgot the name of the organization now, the women in computing. They have a card deck. They have a deck of cards. And every day like we sponsored them. And so every day we pull out a card and we talk about a famous woman in computing. Like there's so many different ways to get in front of folks. And I think that you have to use many, many techniques to find as many smart people as you can. It's not just like one thing. Like one of the, actually, here's one

of the funniest ones is that whenever somebody goes on Twitter and says they're leaving their job, like I'm famous for just like messaging and saying, DM me like all the time. And what's funny about that is not that I see those. And I reply with that is that people send them to me from work and they go, Hey, this person should be a shopper. I'm like, well, I don't know why I'm doing that. You can say that too, but I just, I just just say DM me. And then we have lots of conversations with folks. And sometimes they're like, I already have a job, but it's a good conversation.

So I think it's about all, I mean, here's the secret. If you're looking for the secret for your listeners, the secret is spend way more time on recruiting than you think you should be spending on recruiting. That's the secret. I think there was a good talk from Sam Altman of Y Combinator where he said,

you should spend either zero or 25 or 20% of your time on recruiting, like zero, meaning you don't need to recruit or 20, which is like a full day a week, which is insane. Because people think about, you know, eight hours a week, but that he's right. That's how much time he spent. Actually, when I was doing my startup i would message the team and say i'm not coming in this week because i'm going to be recruiting every day like i would be meeting people for coffee out at the career fair like i was like and and i don't know if it made sense to them until i started showing up with people right like literally i was like i hired this person i hired this engineer like like they were like okay now we know what you're doing when

you're not here yeah what do you think is like the highest leverage activity like i know there's like a lot right and you have to spend a lot of time but like what have you seen has been like the biggest impact like was it the engineering brand or is it something else is it going to all of these events i don't know where you get these questions from man these are good questions um okay what's the high so two different things what's what works the best and what's the highest leverage are different things. So what I spend a lot of my time on is I talk to the candidates directly all the time. Like every day I have, I'm meeting people, whether they're interested in Shopify or not, it doesn't matter.

Like I want to have a good conversation and see what they're thinking. That's I think the most effective. It's not high leverage because it takes time, but if every leader spent 20% of the time doing this, that's probably the right amount of time, then you can kind of hit whatever goal you set out yourself to get to. What's high leverage? High leverage is scalable things, right? Podcasts, talks, video, video writing writing code open sourcing like there's all kinds of things from an engineering brand perspective there are you know you can engage with the community do community events like we do events with like others like when we announced our um moving to react native we did an event with shopify twitter maybe it was lyft like some other company as well so like we like we we went out and like did these community events.

So it's not one thing. It's kind of like culture. It's like not one thing. It's like a thousand little things. But you have to kind of do all those things because you don't know what will be the most effective at the onset. Like I'll give you an example, like Streamlabs,

the number one thing we did for even full-times was have a great internship program. That for sure was what got us all the great people we got, even the full-times. And so maybe if we do this interview a year from now, I'll tell you what we did that worked. I'll know by then maybe, but right now we're doing everything we can to find as many sharp people as we can and open all those pathways. Yeah, I'll put a calendar invite for like a year. There you go, a year. Just like ask Farhan what worked. Yes.

And what you basically said was culture also scales, right? Like when you have a good internship program, people talk about that and people say this is a great company you should work for. They refer their smart friends. Ah, see, you've hit on another super interesting point that most people don't pick up on, which is the best recruiting

is to be an amazing company to work at. I've met, I've been talking to lots of companies. I mentor a bunch, like I'm an investor and advisor in some. And when you've got a recruiting funnel that comes in, but you've got a leaky bucket on your internal company because the attrition is so high, like that, it's not, I'm not even talking about like the scalability of like trying to fill a leaky bucket. I'm just thinking of the flywheel is not working because the internal company is not working and that the feel and the environment like that gets out as part of your recruiting process.

And so it'll make it harder and harder to recruit from. So I would rather fix the what's fixed, how your company is running because it's either whatever the reasons are not working well before you start doing a big massive recruiting push so you know i remember this from another company where somebody said to me hey i was i was vp engineering and they said hey you should run recruiting and i said only if i can run hr interesting and they're like they're like why is that the case and i said because they're connected like you have to have a great company and a great company means being a great place to work and being a great place to work means lots and lots of little things like how long it takes your expenses to get paid and

your vacation policy and your benefits and work environment and reducing admin and that will directly affect the recruiting yeah like a way to think about it is if you have a good product, it's easier to sell it. It might not sell automatically, but it's easier to sell it. Exactly the right idea. Exactly. So you have a good product and it's easier to sell and then your recruiting becomes that much easier. Yeah. So let's dive a little deeper on one thing that you mentioned, which is attrition, right? If I'm a VP of engineering or just like an engineering leader, how do I know, or like, how do I think about attrition? And like, how do I, I'm not looking again for like a number, like number like oh we should target at least x percent or no more than x percent a year

how do i know when it's bad or how do i know when it's okay especially in tech when people are moving around like all the time yeah i think that there's a few things you can do to try to figure out what the right number is for your company so location matters a lot like i think we know something like in the Bay area, tenure tends to be like 18 months, right? Which is maybe tied to like also stock option investing, right? But I think that there are other things. I mean, I'll say one thing, the best thing you can do is make sure you do an exit interview. Everybody who leaves to figure out like, what are the things that could be going better so that you can fix them? Potentially you might think that they're okay. Like you might go, Hey,

these are good pieces of data, but we're not going to work on them. But I'll say a couple of things. One, you can have extremely low attrition, meaning people stay for a long time and that could be bad. And you can have extremely high attrition and that could be good. So it depends on what you're trying to optimize for. So one example, you know, I'll give you is that at extreme labs, and that can be good. So it depends on what you're trying to optimize for. So one example, you know, I'll give you is that at extreme labs, we had extremely low attrition after 90 days because people stayed and like there were, there were a fit, but we had pretty high attrition in

the first 90 days. Like we had like by 10 to 15% attrition in the first 90 days, which is insane. But it was part of our process to help people feel like, is this the right place for them? And we would, you know, actively talk to people and some people would quit and some people we would let go in that first 90 days. And then after that, they would stay for a long time. And so I think that there's a way to kind of figure this out for your company. And the other thing to think about is, you know, over time people have their own frameworks and if they stay and it's not the best place for them to be, it's up to you to either help move them around the company, just to show them other opportunities or help them see things outside of the company. Cause maybe it's no longer the place for them. And, um, that doesn't have to be a bad thing.

So it is, it is kind of misleading to just talk about a number, but it can be scary to like hear extreme numbers and then try to figure out what's going on there. And then you can segment, right? Like, is it the first 90 days or the first year? Or maybe somebody's been at your company for 10 years and like it's time for them to like try something new. Yeah, it's like an observability problem. I think what you said is exactly right.

Like everything is, it's unique for the company you're in. And based on- But it's a an observability problem. I think what you said is exactly right. Like everything is it's unique for the company you're in. But it's a good thing to track. Yeah, it's a good thing to figure out why people are leaving. It's also a good thing to group things together. Like, I mean, here's an example. Somebody might say, one person might say they're not happy with their compensation. Another person might say they weren't happy that they couldn't get promoted. And they might be the same thing. It might be that they were trying to get promoted because they didn't think they were being compensated correctly and they thought that that was their avenue right like um and i don't think that they're they can be necessarily like um like you have to look at the each case individually and then you could group them to say okay it actually turns out maybe we need like a proper career laddering or maybe we're not up to date in our market salary grid. And that's why people are leaving. There's lots of different things to think about there. And sometimes people don't know, right? If you asked me, like when I quit Microsoft, if you asked me why I quit,

I would tell you I didn't get promoted in three years, right? I was a product, I was a actually program manager. I was a head of engineering for MSN. Like I was doing a bunch of stuff. I never got promoted in three years, but then looking back and I look at now, engineering for MSN. Like I was doing a bunch of stuff. I never got promoted in three years. But then looking back, and I look at now, I look at my framework. I'm like, oh no, it wasn't that. It was, I wasn't learning as fast as I need to be learning.

And nothing to do with it because when I quit, by the way, they like offered to like raise my salary and give me a promotion, right? But that wasn't the reason, but that's what I thought it was. So sometimes people know and sometimes, you know,

even I didn't know what I thought I was leaving for. Yeah, I think that gives like a pretty good framework, right? You have to basically analyze and try to see if there are buckets and all of that to actually understand. Yeah. And you want to fix it. You want to fix it, right? And sometimes if people say, oh, it's a very intense work environment. I'm like, okay, is it intense, but intense for 40 hours a week? That's probably good, right? Or is it, and maybe that was a good way for the person to figure out this is not a place

for them. Yeah. You want people to be like result oriented. You just don't want them to be burning out. Correct. And that's why, that's why I'm a big fan of like, I want people to work as, you know, being a high pace learning environment, 40 hours a week. Makes sense.

And maybe like a final question. So as an as like an engineering leader, what do you think is like the best way to grow like someone from being like a more junior engineer to like a senior engineer or someone who is the manager into a director? Like, how do you help? What's the best way to help someone? Yeah, two different questions. I think that for me, this is an obvious answer for anybody who knows me, the senior to junior, junior to senior is pair programming. Like the best way is to be in high fidelity, working with people across the company. It doesn't have to actually be like much more senior. It just has to be like

two engineers at many different levels will learn from each other. So I think that's probably the best way. One of the examples I always give is that when you're talking to people and let's say you're in a meeting and somebody says, hey, we're thinking about building this experiment. How long would it take to build? And a junior engineer will make an estimate. It's probably wrong. A senior engineer would make an estimate that's probably right. But the staff engineer, who's one level higher, will make the estimate that's probably close to right, just like the senior. But they'll say one more thing. They'll say, but, you know, if what you're trying to learn is this, we can make this one small tweak and we could probably try this in like one third of the time.

And I think it's that feedback loop that makes people more senior. Right. So a senior might say the junior might say it'll take you like a week, and it's probably like two months, right? The senior says two months, the staff says this is by two months, but if you wanted to test it quickly, if you made these changes, we could probably do it in like two weeks. So I think that's like, that's on the engineering side. Now on the manager side, I think, I mean, it's a good, it's a good question. I think the way to think about it is, what are the types of problems you're solving? And how do you make those scalable?

So typically what I see from manager to director is that the level of deliberate problem solving changes in like, hey, I'm trying to solve this problem for my team. And as a director, you're trying to solve problem for multiple teams. And as you look across those, you might say, oh, actually, it's not just that I want to solve this thing here. If I solve it this way, it'll, it'll make like 40 people's lives easier. And so the scope tends to increase as you start building those, some of those

scalability problem solving muscles, right? So, you know, one, here's one more concrete example, right? If you, if you're, if you're a manager and you fix something for your team, what I would hope that somebody more senior would do would be like fix the process for everybody in the company. Right. And I think that's something that you can, that makes you more scalable. And usually as you're going up the ladder, you're trying to figure out how do I build, how do I turn these things into systems versus just solving the point problem, right?

So one problem could be like, oh, the CI is slow. And so for your project, you might redo all of your tests in a way that allows the CI to go faster. But if you're, as you move up the chain, you might say, oh, actually, maybe as a company, we should benchmark all of CI. Maybe we need people working on CI full-time for three months. Maybe we should come up with a testing strategy for the

company that allows us to think about CI. Maybe we don't need CI, like who knows? Like there's a bunch of things, like, I think that's the kind of, you start moving up levels to figure out what is the right problem layer I should be solving this at. Okay. So like learning to think maybe in a broader scope and also executing probably like, so making sure that you actually get results in that broader scope. Yeah. I mean, it's, yeah, it's hard to know because I don't think there's like a checklist, but I do think over time it does like the more scalable you can

make yourself the, you know, actually, I mean, here's, here's another way to think about it. I have this conversation with folks as I say, if you can't be replaced, you can't be promoted. So if you're a manager, one concrete thing you can do is try to figure out who's, who's on your team. That's going to replace you. How do I work with that person to get them up to my level so that I can go work on something else? That's basically the idea between thinking and systems, because now it's counterintuitive. Cause I think people feel like, Oh, if I'm not the manager, like, what am I going to do? Like you get scared. Cause you're like, you know, and actually all my career, I've basically been focused on like, how do I replace myself? Like, how do I either create a system so that you don't

need me to do that thing anymore, or somebody to do that thing? Or how do I train somebody else to do what I'm doing so that I can go do something else? And it's because it comes with extreme learning when you do that, right? Like I'm learning how to make it a system. It puts somebody else there. And then I'm like, okay, I'm going over here to learn some other thing. And I never get, I get scared if I can't figure out that way to replace myself versus I think people get scared if they're like, what am I going to do now? And I'm always like, there's always things to do. Like you're over, there's always things. So I never, yeah, there's always problems to be solved. So that's one concrete way that I think people can take away is like, if you're a director, like who's the

next director on your team? If you're a VP, who's the next VP on your team? What are you doing for that person to get them to be a VP so that you can go do the next thing? Yeah, good engineers and good managers are basically automating themselves or replacing themselves. Exactly.

Yeah, cool. Well, thanks so much for being a guest on the show. I think this was a great conversation. You spoke for a pretty long time and yeah, thank you for being a guest again. Amazing. Thanks for inviting me. Thanks for all the good questions. Seriously, I've done a And yeah, thank you for being a guest again. Amazing. Thanks for inviting me. Thanks for all the good questions.

Seriously, I've done a lot of these and you had some really insightful questions. Thank you.