Y Combinator Startup Podcast - #1 - Hiring Engineers with Ammon Bartram

Episode Date: May 15, 2017

Ammon Bartram is the cofounder of Triplebyte (YC S15) and Socialcam (YC W12). Triplebyte (https://triplebyte.com) connects software engineers with companies that are hiring. Read the transcript here (http://blog.ycombinator.com/hiring-engineers-with-ammon-bartram).

Hey, this is Craig Cannon, and you're listening to Y Combinators podcast. In case you don't know about YC, here's what we do. Twice a year, we invest 120K in a large number of startups. The company's moved to Silicon Valley for three months to work with us and build their company. At the end of the three months, they demo to a room full of investors. Many of the companies have gone on to be very successful. Some of the ones you might know are Dropbox, Reddit, and Airbnb. I'm excited to get our podcast, and I'm excited to get our podcast, Aaron Harris and Katman Yalik for starting the original YC podcast, Startup School Radio. Our first episode is with Amin Bartram. Amin has co-founded two YC companies, Social Cam and TripleByte. TripleByte connects software engineers with companies that are hiring. And some of the most frequently asked questions at YC are around hiring, so I thought it would be good to have Amman in as he thinks about it all day. Our discussion mostly focuses on hiring engineers, though much of the advice he shares could be applied to hiring for other roles.

Okay, here we go. Hey guys, today we have Amin Bartram, co-founder of Social Cam, TripleByte, and he is here to talk to us about hiring. So could you just give us a quick intro about what you've worked on? Cool. So I joined Justin TV fresh out of school in 2009 when it was just 25 folks and kind of went through the roller coaster of the early days of Justin TV.

And there I worked mostly on the video system. And I think that was where my first sort of taste of hiring. We were, at the end of that, we were hiring pretty aggressively, and that was when I first realized how much noise there is in the hiring process. Okay, yes. And then I was part of the spinoff of Social Chem, and that was a sort of video sharing app. I did that for about three and a half years.

And we were acquired by Autodesk in 2012, and I worked there until 2014, and then took a bit of time off and started Trulbyte. Cool. And so triple bite just for context, for people? Can you explain? Sure. So we're a recruiting startup.

So we help startups hire engineers. And so engineers apply to us. And then we do sort of a full interview with them, you know, pass the engineers who are good and match them with companies where they have a high probability of doing well. Cool. So people ask us a million questions about hiring, recruiting, all of it. In general, let's assume that you're, you know, early stage startup, what should companies be looking for in an engineer?

There's not a crisp answer to that. I think the sort of the pithiest answer I can give to that is you have to decide what it is you want to look for, and then you have to effectively look for that. So sort of the status, actually, I think there's sort of an important truth here. Most companies think that they are trying to hire good engineers. That's what they say to themselves. And what they don't realize is that, you know, company A's definition of a good engineer is significantly different from company B's. And so what you have is a situation where everyone has this definition in mind, and they're all different.

And this is a big source of noise. So, for example, if one company thinks that engineers need to be, you know, very fast and productive and be able to show that in an interview, And a different company thinks that engineers need to be very intellectual and be able to deeply understand computer science topics and talk crisply about that. What happens is sort of all of the all of the awesomely productive engineers, very practical and not necessarily in the strong academics who applied to the second company, fail. And all of the very academic could totally solve all your hard problems that maybe aren't quite as flashy in a web environment, engineers who apply to the first company also fail. And so it's, you know, for companies at a bit of a larger stage, I think the obvious answer is you want to hire both those people. And so it's about building a process that can identify more broadly different types of skill. For smaller companies, I think you're more in a situation where you may well actually only want one of those folks.

You may well, you know, the thing that's holding your company back may well be productivity. And we need someone who's going to come in and be productive and sort of bang out code. And if that's the case, you know, you need to realize that it's not important that everyone you hire be strong in computer science. And if you are, you know, a company where you're facing security issues and, you know, really clean code, really precise code and solving hard problems is important to you, then, you know, at an early stage. And it probably makes sense to have a process that skews more in that direction. And so do you have general advice for people that come to you guys at TripleByte or just you as a friend advisor for diagnosing, like, what kind of engineer is? is a good engineer for your company? What do you tell people?

We don't. So it's funny. We're rarely in that situation, actually. I think most people have strong preconceptions. So we're more often in the situation of sort of broadening people's people's vision of what a skill engineer can be. But I think the obvious, yeah, let me circle back to the question. I think what happens a lot, this is a mistake that's easy to follow. into is when people are interviewing an engineer, they tend to ask about the things that they

are the best at. There's this overlap between the things that you're the best at and the things that you think are the most important. Every engineer thinks the things that they know are kind of the core of the discipline. And so you ask everyone who you interview those questions, you bias yourself toward hiring engineers who have those skills. They join your team. They ask people, they interview the same type of question. And so the whole organization and can grow in a direction that might not make sense. Okay. You know, it's very complicated because there are plenty of examples of companies with, you know,

with defined engineering cultures that have worked out very well. So, for example, Google, you know, has intentionally or intentionally grown very much in a computer science direction, and that's obviously worked out, you know, very well for them. Yeah. You know, okay, there are other companies in OIC that, that, that, I don't want to say names here. But there are companies that take a complete opposite, take a very human productivity-friendly approach, and they're also extremely successful, excellent companies. And so, you know, answering that question is not, basically, there are success cases on both sides.

But I think as, you know, when you're hiring your first employers, I think you need to just basically, you know, try to decide what's holding us back, you know. And so, yeah. And so say, like, say I'm trying to vet this pool of engineers and they all, like, fit the certain rubric that I'm, I've created, right? Yep. But maybe one of them did a boot camp and has some projects, and then one of them, you know, went to a great school, has a CS degree. So how do you, how should I think about, you know, credentials and experience?

Boot camps versus CS degree, I don't think are all that different. Now, okay, that's obviously a kind of a forceful statement. So I think experience matters more than where you got your education. So someone fresh out of a CS program who doesn't have internships. It's still essentially a junior engineer. And perhaps they may have a more academic slant to what they've studied than someone out of a boot camp. But both those people lack real world experience.

I think categorizing that differently than someone who has worked in the industry for five years and can own projects. So the skill that you can most easily measure in the skill that you can most easily measure in an interview is ability to think through rather these small problems quickly. That's really what interviews can evaluate.

The skill that you need in an employee is quite different. The ability to solve large sprawling problems well, but over a long period of time. And there's obviously a correlation there. It's not, we use interviews as a proxy of evaluating actual skill because there is a correlation there. But the correlation is not perfect.

And an interesting observation is that people who are fresh out of university and boot camps, actually in many cases, because they've been practicing, are better at the kind of problem like it's asked in an interview than your very senior, you know, 10 years of experience at large company engineer. What the senior engineer has, typically, is experience making our architecture decisions, owning projects, gathering requirements, carrying that whole process through. And so, okay, how do we evaluate for that? It's super hard. Basically, what ends up happening, and this is honestly unfair, is that experience is used as a proxy for that. This is something that we're focusing on a lot of trial bite. But what, you know, this is very strange.

Actually, if you are, if you have five years of experience, it's just flat out easier to pass an interview. You will get a job offer, you know, after a worse performance. People think maybe that senior engineers, you know, you expect their senior, they should perform better in the interview. And that's actually not generally true. The barred to getting a job offer goes down as you have sort of a present-looking resume. And this is not, you know, it's not irrational in the part of the companies.

It's just the reality. Sure. Okay. And so then when you guys are like pre-screening these people for triple bite, for example, like what are you looking at? What are you having them do? So we, the approach that we take is to evaluate as much as we can in isolation and be aware what we're evaluating.

So we explicitly evaluate. programming, just programming productivity, given a relatively, you know, a speck-out problem. So, for example, we like, describing an algorithm to solve a problem, it's not super math, it just who's the set of steps they have to do. Can the candidate take that and, you know, render it into well-working, well-structured code? And interestingly, junior folks actually often do better than senior folks at that sort of problem. We then separately do a evaluation of sort of academic computer science skills. You know, is the engineer knowledgeable about computer science and about that approach to problem solving?

We then separately, one thing that we took from Stripe actually is we do a debugging section. And so we give the candidate a large code base and that has some bugs. And we ask them to sort of take some time, dig into the code base and then try to find and fix these bugs. And I think that does a great job of solving some of those problems. basically because this is a skill that comes from experience that is often missed by more, more, more traditional new years. And then finally we do a system design section. So, you know, here's a talk, here's, you know, here's some requirements, design a production web system to satisfy these requirements. Okay, now we're going to change the requirements. You know,

how do you adapt your design? How do you talk about tradeoffs? And all of that's done remotely because the person's at home, right? Yes. We do this all. over Google Hangouts. Okay. And so what's a reasonable amount of time for someone to just, like, go through one of these exercises? Are they all widely different? To go through our exercises?

Our interviews are about two hours in length. And so we spend about 30 minutes on each section. Okay. Cool. And you find that's, like, a very strong data set in terms of correlating how successful they are? Yeah. Well, we've done about 2,000 of these interviews over the last year and a half.

And so we've been able to sort of drill in on the parts that are, that are most predictive. and cut time off and shorten it. I think if you're starting from scratch, you would probably need about twice the amount of time to get through all that stuff. Okay. And so having gone through all these interviews at this point, was there anything that you thought

was really important in the beginning or something that's very common in the valley that many people think is important? That isn't really important? Too much reliance on a single question. So the classic interview format in tech companies is a number of one hour, 45 minute to one hour sessions.

And often engineers pick the questions themselves. And they're usually, like, little nuggets of, like, they're sometimes pejoratively called brain teasers. I don't think, like, almost no one actually asked brain teasers. They're more like legitimate programming problems. But they are little nuggets of difficulty. Like, how do you, you know, given a string of parentheses, how do you turn up the well match?

Okay, given multiple types, how do you turn with them all match? If you don't, you know, that's a classic interview question. And it ends up, there's just a huge amount of noise. If you just take a bunch of candidates and in a controlled setting, have them all answer, you know, three or four of these questions, you'll see there's just, you know, there's some correlation, but there's way less correlation than you would think. And companies, honestly, I believe this in my previous, you know,

when I was, you have this question, you ask them on the question, and you see this variation, and you assume, oh, the people who answer this question well must be smart and good programmers and the people who get totally tripped up must not be. And then when you actually like inspect that you see that there's this huge amount of noise and like we have this pretty incredible

lens on this because we evaluate engineers pretty rigorously and then send out to multiple companies and see what happens, get detailed feedback. And so yeah, is that feedback from the actual interview process or then once they're placed you actually know as well how they're doing? Both. Okay.

But I'm kind of talking about the interview process. So we screen engineers, we send them to companies, and then they do the interview there, and we get feedback. And it's just pretty incredible how much is agreement there is. So a candidate who, you know, does really well at one company, and we get total, this is what the best people we've interviewed in months, this is a rock star, goes on to fail an interview at somewhere else. And a pretty interesting stat we dug up is I compared the rate of agreement between interviewers at companies. with a data set of users reviewing movies online. And the numbers were actually, basically, and the inter-rater agreement was equivalent.

So basically, knowing that an engineer did well at one company, gives you about as much information about whether that engineer is, you know, skilled as, you know, knowing whether, you know, the New York Times film critic, you know, rated 12 years of slave as excellent or terrible. So, okay. Maybe you don't have an answer to this, but say I'm really good at brain teasers. Where should I interview?

Larger companies. Probably larger companies. So, and this makes a certain amount of, okay, this is all very complicated. It makes a certain amount of sense. So bigger companies, so brain teasers always introduce noise, but we've found that bigger companies rely more on that type of interview. And they do that partly for some rational reasons. So bigger companies care more about measuring your innate ability and less about measuring whether you can jump into their particular code base

and be productive on day one. But it's way more likely that a smaller company is going to say we're using Ruby on Rails. We need a very productive Rails developer. Come take this interview and we're going to measure how well you can do maybe how well you can work on our actual code base. Okay. Whereas the big company,

You know, Facebook, Dropbox, Apple, Google are more likely to say we care about smart people. And within the confines of the noise, of the interview process, they're trying to identify intelligence rather than specific experience. I mean, they also have like capacity time to train people. Yeah, precisely. Whereas like a small company in no way. Okay. And then like prior, one of the questions, what about skills that people don't think is correlated that are strongly correlated to a successful engineer? relatively easy problem solving.

So we've found that asking pretty easy interview questions is often more generally predictive than asking harder interview questions. And so to break this down, there are two sources of signal from asking a question. You can get signal on whether the candidate comes up with the right answer, and you can get signal on whether they,

whether they struggle, how easy or harder is them to solve the problem. And so we score both these things for a bunch of questions and we've done this for thousands of candidates. And what we found is that if you go in and look at how much, you know, the individual score

on one question we're asking correlates with how the candidate does on the job. And what we found is that, as you would expect, getting a question right is correlated with being an engineer. And as you would expect, being able to answer a question

easily is correlated with being good engineer. But there's also, of course, you know, false negatives, right? So there are great engineers who fail into real questions and they're great engineers who struggle with questions. And if you do, if you look at like the actual predictive ability, you know, rather than just the correlation of getting on the right side, right? The sweet spot is actually far lower on the scale than most people intuitively think. And so, yeah, can you give you like a couple examples of what those easyer questions might be? Yeah, sure. Just like saying like, you know, we want you to create a checkers game. Absolutely no, you know, no logic, I think complicated, just a class that has a board, has a grid, it's got pieces, pieces move around.

This is really a pretty mundane, straightforward task. That actually ends up being, you know, how well candidates do that, ends up being a more stable predictor of engineering skill than sort of, you know, here's you know here's a you know here's the I'm going to give you a sentence that can consist of a list of words

all glammed together and you define the optimal way to give a dictionary to break this apart into words okay that second problem you know ends up being

a graph search problem that can be you know optimized with you know memorization or dynamic programming the second the second problem right

carries more information than getting the first problem right but with a really high false negative rate yeah and so the first problem ends up actually being a better general predictor of engineering skill.

And so is there a way if I'm like getting ready for a new, you know, I'm going to go to another company. I'm going to get ready to interview. Do you recommend people train in any particular way or is because like you're going for that sweet spot of easy questions? Like, you just have to be smart enough to do it. Like what do you tell people? Sort of in general. If I'm going to prep to do some interviews, what would I do? So, I mean, I guess there's two questions there. One is where I think for companies that I think are doing a good job interviewing and then maybe for what I think some sort of the status quo is. In general, it depends where coming from.

So a very different advice for new grads and for experienced folks. Okay. So let's break up them apart. Yeah. New grads. Okay. For new grads, I would say, you know, make sure you're solid with sort of the classic stuff that, so, you know, breadth first search, you know, hash table heaps.

I mean, it's important. Classic or computer science. Yeah. A surprising percentage of a new question ends up being slightly obscured applications of sort of those, especially sort of hash tables and breadth first search. Those two things by themselves represent probably 40% of the questions that are asked in most companies. And so you need to know those. But actually, many new grads are already pretty solid in that because they've been being drilled throughout school. The second thing is practicing writing code under stress. So working out a big problem over time is very different than you have 10 minutes or 30 minutes. Here's a marker right in a whiteboard or even here's a laptop program on it. And so just the things correlate, the skills correlate. But you can improve your performance by practicing. So totally put in 30 minutes a day, finding some of your questions online and giving yourself a time limit and sort of trying to solve them under stressful situations.

And are there good resources that people can look for? Like anything in particular? Yeah, I mean the classic ones, right? So cracking the coding interview has a pretty good list of questions. It's the other advice in that book I don't think really applies to startups very much, but the questions are good. Then there are a bunch of sites online that have lists. Interview Cake is one that I've seen that I think is high quality.

An interesting aside to this, though, is that most companies actually want you to do these things. It's not, companies would prefer that all their candidates, we prefer. We totally prefer. Now, we try to design our interview in such a way that there's no impact, right? We don't really want to be measuring if you've been cramming. on algorithms. But what companies want to measure is like max skill, max potential. So they would actually much rather see you, you know, in a state where you're well prepared in the material, rather than the state when you have the potential to understand it, but forgot about it.

Yeah. And, you know, an interesting trend that's happening in the industry is companies being more upfront about what they're asking. So Facebook, for example, has started providing a sort of an interview prep class to everyone who applies so that there's sort of going over the material. It's part of me finds it encouraging because it is moving it in a better direction, but that's also discouraging because it's like really sucky that you have to like take a class to figure out of it. Right. I just wonder if it's filtering for those types of people, right, who are just like looking for it's like, I don't know what to do. I don't know what to do. It's like, well, if you apply here, you can take the class and then like, take it. Like,

let me hold your hand the whole way through your life. Um, yeah, yeah. Okay. Um, um, Okay, so say I am going to interview at a bigger company. Is there a way to prep to do well with the brain teaser stuff? Yeah. Practice. So again, there are some words that are thrown around describing any of your questions that aren't. Brain teachers are pretty rare. Some companies probably have asked things like, you know, the golf balls in a 1747.

But that's really very rare. Much more common is application of a computer science idea to a practical problem. Okay. And there still is this leap of insight required. In many cases, I think those are bad, bad any of your questions, right? You don't, like, you know, companies should try very hard to ask questions where there is, it's not like there's one thing that has to be grasped until that's grasped.

The problem feels impossible. But that, you know, but hard application, you know, sort of practical application of computer science topic represents the significant majority of questions at big companies. And so, so when I was in college, I interviewed at one of those, like, big management consulting firms. They did have all those questions. We spent like two months, like, prepping for it. And I didn't get the job.

And I did okay on like the stupid, you know, ping pong ball questions. You don't have to feel bad. One thing, one number we have that's interesting is that the engineers who do the best at companies go on to pass about 80% of their interviews, but not 100. No, almost no one passes more than 80% of their interviews at companies. Okay. And yeah, so, you know, one big advice to everyone is just like, don't feel bad if you fail the end. It is really not a referendum on your skill. I'm very happy to have not gone in that direction.

So what about the role of, like, you know, projects, someone's portfolio of like side projects. Are there certain types of side projects that across the board are attractive to companies? Or is it like, you know, say I'm applying for a job at Stripe? And like I did, you know, X payment type project and that would be more attractive to them. So across the board, are there things that are interesting? Let me, let me talk about, answer that question. Then we'll talk about what I think, you know, the right thing for companies to do it. So companies don't actually pay very much attention to side projects.

Okay. Except for at the screening stage. So resume screen, you know, candidate applies to a company, the company decide if they're going to interview the person at all. And there's some adverse selection bias and who applies. the companies and there's this big stream of candidates. And so at any company, there's this huge dream coming in and they have to decide somehow. And so they do that based on resume screens. And it comes down to pretty dumb credential stuff. If you've worked it at a top company,

you've gone to a top school. Or in some cases, if you have a project that they catch is right, that's impressive. And so sidebrass can help a lot there. But they are very rarely given away in the actual interview. And I think it's actually probably the right decision. So people who have side projects sometimes feel bad about this. Yeah. But the reason it's the right decision is that most engineers don't have side projects. Most engineers have

been working at a company and it's all proprietary code and there's very little they can show. That's, you know, eight out of ten in that situation. And having a consistent process, consistent fair, you know, like, consistency is the first goal of design of your process so that, you know, the big problem is that

the process is not usually consistent. So if you make it consistent, then you can optimize it. And having this sort of other process we look at at projects introduces noise. And it's also just really hard to do. So you can't tell if someone spent, you know, a weekend on the project or if they've been working out for like the last 10 years. We literally see both pretty regularly when talking about their projects. It's something I did over a weekend or this has been my, you know, abiding passion for the last 10 years. Let alone like who actually contributed.

Yeah. Yeah, and, you know, things like coding quality, right? It's actually, it's startlingly hard to look at a big bit of code and decide if you think the programmer who wrote it is, is skilled. Just again, like there's so much context. You can't tell what bugs they spend hours over. Finding me, like, finding bugs. None of us are good enough to look at a code and like immediately find the bugs.

Yeah. And, yeah. And so, you know, for that reason, for all those reasons, you know, side projects are useful. So if you're, if you're applying for jobs and you're being screened out at the resume stage, doing projects probably helps. I'm doing projects a great way to obviously increase your skills and that will be reflected in better performance on interviews. But I don't think projects have a very big role in the actual interview. And so what other things should I think about if I am being screened out? Like say like, you know, I'm getting a callback from like a one out of ten.

Yep. What should I do? Flat of Trilbyte. that's the short answer um you know otherwise yeah side projects help it just sucks right

it's this it's not it's not it's not malice on the part of the companies you know like they're overwhelmed by applicants and so they use these sort of crude filters and that's I think the big thing that we're focused on is trying to figure out to how to directly measure the skill and so that we don't have to rely on on you know

filters like where someone's worked or what's called they went to and what about things like you know for example, location. So say I live in Salt Lake City and I'm interested in getting a job, possibly at Facebook. Should I put like San Francisco on my resume and just like fly out for an interview? Do you have general advice in that area? Big companies don't care at all where you're based. They have, they fly people in, you know, by the hundreds every week.

Smaller companies do show a slight preference to local candidates. And so if your goal is to work at a small, let's say, sub, you know, 20 person startup, You're probably out of, I don't know, 10 to 20% advantage if you're based in the Bay Area. Right there. Okay, cool. So from the company side, there's like a million different interview methods that people go for. Say we are, you know, they go through triple bite, they get screened. They're going to do an interview.

Whiteboarding, pair programming, all that stuff. How do you feel about it? All the methods can work, the retro here. Let me sort of give a bit of a overview here. So, as I mentioned earlier, the core problem is there's this tension between the skills that can be best in an interview, solving small problems quickly, and the skill that matters as a program, solving big projects over a long period of time. And so the first approach you can take to interviewing is just say, okay, we're going to not do it. We're going to do like trial employment to something like that. And that totally works.

If you've worked with someone for a week, you have a far better read of their skill than, I think anyone can get during a three to four hour interview. The problem is that there's a pretty strong bias in who's willing to do trial employment. And it's an adverse bias. So many of the best programmers have lots of options. And if your company requires that everyone do this trial employment period, most of them are not going to just going to say no. And obviously, anyone who currently has a job. They can't leave for a week.

Can't leave for a week? Yeah. And of course, there's also, you know, you're committing a week of time. And so obviously you need some filter before the trial, you know, employment. And so I think in the end we're left with a thing kind of like the, you know, like the famous, you know, democracy is the best one of government except for all the others. I think that, you know, interviews are the worst way to evaluate engineers except for like all the other options. Yeah. And so we're less so like you have to do it.

It's fundamentally inaccurate, but you still have to do it. And the goal is to make it as accurate as possible. You know, and so once you're on that page, we see two sources of noise. we see noise that comes from the companies being inconsistent. So I talked about that a bit earlier. You know, just, it is still too often the process that engineers are responsible of coming with their own questions. So if you're asking every candidate different questions and coming to a gut call,

they're just this far larger than anyone really realizes a source of noise. And so like if you ask, you know, pick any company that has that process. If you ask them to somehow re-engineering, re-interview their colleagues, you know, in a blind fashion, right? They would have a, you know, they would likely have a 50 to 60% pass rate, right? So 40% of their colleagues would be screened out. Yeah. Yeah.

And so the solution there is just to be really consistent. Okay. So like to make sure that you're asking everyone the same question and make sure that you're evaluating them in the same way. And I think that's more important than what you're actually asking, right? So like if the first step is be consistent, second step is tweak that over time based on what's done on the results you see. Okay.

you know, once you're doing that, I think the other source of noise we see is companies looking for different things, right? And so, as example, like, if earlier you have a company that's looking for super academic, you know, engineers, you have to look for very practical engineers, you have companies that, you know, all skill engineers, you know, know, know, all skill engineers, you know about how operating systems work. You have companies who think that, you know, they only want people to people who have experienced in compiled languages. You have companies who like want people to use enterprise languages. You have companies, it's this mess. Yeah. And so I think the important thing is, you know, is to untangle which of those are conscious decisions you're making about who you want to hire. So you're a banking company. You want a big focus on, you know, QA process and safe code. It probably makes sense to reject someone for being too much of a cowboy. You know, you're a social media company. Your goal is to move really fast. You know, maybe you decide to have a culture where you want to move fast and break things and you want to hire cowboys.

You know, those are all, those are, you know, logical decisions. Okay. But very often companies are making similar kinds of decisions almost by accident. And so sort of introspection deciding, okay, like we want to hire those people and then designing the process to look for it. And so in your examples, whiteboard coding tends to skew toward the academic. It tends to give preference to people who are really good at breaking their thoughts down into this sort of structured academic way and writing with a small amount of code. So you often have people who are actually really productive, excellent programmers who look really stupid. and bad on a waypoint interview.

And so if you're not looking for academic skills, it probably makes more sense to put people on the actual computer and see how they actually work in their environment. Okay. And so what does the underlying question for me is like, could you engineer a perfect interview? But I wonder, like, what does the interview for a job at TripleByte look like? I mean, I imagine you made it, right?

Yep. Well, so first of all, all our candidates go through a regular process. Okay. So we hire people out of our, our regular stream. And then we compete head to head with companies. So they, we just enter them to us as well as other companies,

which is kind of fun. So they first, so they first go through a regular process. And so we already, you know, have a pretty strong sense of how they are in those areas. And then just like, you know, my advice generally is to decide

what the skills that you preference. And so I think we, we preference a couple of things. We prefer, you know, data and data analysis is pretty key to our business. And so we preference people being comfortable and familiar, talking and thinking about data.

That skews a bit more academic, I think, than what many companies hire for. And then we, because we're in the business of evaluating knowledge really broadly, we then preference breadth of knowledge, I think, to a greater degree than most companies need to. And so what does that mean in practice? Like, what questions would I be?

Would I be looking at? Yeah. So we, again, so everyone goes through first our, our standard process. Yeah, yeah. And so that gives us a pretty good read on just, you know, proactive programming output, general knowledge of computer science, general knowledge of system design.

And then we then, in our, we do an additional follow-up, you know, on-site with the candidates. And that goes much more into depth into data. Or if we're hiring them for a different role, we sometimes hire folks who are not working in your data. So if we're hiring to be sort of, say, a front-end developer, that would be sort of into depth, into front-end development. And so here's a, you know, here's a spec for a front-end project.

You have two hours. Build that. Or if there can be a back-in specialist, here's a back-end spec. Gotcha. Okay. And so as an engineer, should I be paying attention to like every new thing that's coming out? Is that going to be of importance when I'm doing an interview? Or should I be paying attention to whatever, a medium amount of it?

Well, yeah, there's an interesting longer-term answer. So one thing, a class of people we see, interestingly, we see people. We see people who have, who thought about that same thing 10 years ago. Right. Made the decision to not keep up. Now the industry has changed and now these folks are maybe still using, let's say, CGI, and don't understand modern webstack and are indeed in a weak situation in interviews. So I think the answer to your question is day one is not so important.

Very few companies, especially generally only smaller ones, are like directly evaluating flashy new tech. However, if you make the decision too, too forcefully today and don't keep up to date and you end up being totally behind 10 years from now, then you probably are going to pay a price. Yeah, I mean, especially if you're actually interested in starting your own thing at some point. Like being on the edge really, really matters. Okay. I mean, maybe this is kind of difficult to answer, but I wonder about like employee retention, engineer retention. Are there any qualities that you've found like you can vet someone and say like, okay, I think the actual. average is like 18 months or something for someone to stay around. Are there qualities that

correlate to longer term employment? I haven't looked at our data on this recently, so this is going to be a little bit sort of off the cuff. Yeah. I mean, just the obvious things, if candidates who are excited about the mission and the actual company have a higher probability of staying than candidates who are chasing the highest paycheck. Of course, they're counter examples. Sometimes there are awesome engineers who are looking for a place to really commit, but I also want to make a fair way. So it's all right. Like these things are all really complicated.

But yeah, looking for, I think, I think the number one thing I would just say is looking for engineers. How are excited about the company and the job. Okay, cool. So kind of just wrapping up, I wonder, are there any books or things that if I'm kind of like an engineering manager, I'm going to be running a bunch of interviews that I should really dig into and I can get a lot out of? I have not actually found any books that I think are very useful. I think this is going to sound arrogant maybe. But I think engineering, interviewing is this field where it's so easy to say things that sound profound that are not true.

I truly believe that like 80% of what's written out about interviewing doesn't actually hold up. So, for example, it sounds like a really idea that a lot of engineers love, right, is the statement that interviews don't make any sense at all. And you just look at work someone has done in the past. And, you know, we tested this a bunch. We tried scoring engineers and having talked about past projects. projects and scoring them and trying, like, even like a full hour, like going to depth in the project, talking details, scoring it. Just talking skill, ability to spin a tail, ended up dominating the actual engineering rigor. And this was far less predictive of job performance

than giving them a relatively simple programming assignment. I mean, and that kind of sucks. I don't really like that. That's the case. And you can find so many articles out there on the internet about sort of, you know, this, it's stupid that we're asking engineers to do these interviews. Why don't we just have them talk about their past experience? And just if you test it, it doesn't hold up. And just for the sake of like keeping the standard, right? What do you tell people do in that way when they're conducting an interview? Well, yeah, I mean, standardize, right?

So you'd be really careful about helping people, interestingly. So certain candidates are a lot better at a listening help without necessarily realizing that you're helping them. It's the thing we've had to battle with a bunch, actually. And so we have, it helps because we, again, we're doing thousands of interviews, and so it's easier for us to do this. But sort of being, we have kind of a decision tree of all different ways any of you can go and like what help we're allowed to give and what help we're not allowed to give. It's a big sorts of noise. Outside of doing a thousand interviews and standardizing it, I'm not sure I have a really good fix for it, but be aware that some really compassionate candidates will be like. Okay, so, okay, so what's a common area where I might ask for help without you even realizing that I'm getting help?

one is just being brave enough to ask. So saying like being really friendly. Yeah. And then saying something with confidence that's that's sort of like sort of right. And then getting, there's this natural instinct to like add on to incorrect the error. And as the interviewer, it's really easy to do that and not realize that you're like steering the person through the problem.

So if you're going after interviews, you should do exactly that. I have a blog post on how to fare for interviews. And I do like, I recommend trying to do that actually. Really? Oh, that's awesome. Yeah, I want to add a sort of a side to that, though, which is actually the negative side of that, which is that interviewing can turn into hazing. Another interviewing is not just devaluation. It's also like this right of the shared right of entry into a company.

And some companies develop this culture around the interviews being hard and unpleasant. And as the interviewer, it's really easy to forget how much harder it is. But you're the one you're answering the question. much easier to feel smart when you're asking the question. And sometimes candidates get really flustered and can't answer a question. And it seems like it could be really frustrating as the interviewer if they're like, this thing is obvious in front of them and they're just missing it and they're wasting your time. And you can get a little bit angry inside. And it's just really important to stay away from like the hazing, but like taking out anger on them by like, you know, I'm generally

against cutting interview short, actually, I think it's, I think, I think, except we're in the case where the candidate is in pain, I think it's not worth doing. I think you save some time, but, but you damage reputation, you know, they, they, they just like it. It's embarrassing. Okay. But definitely staying away from the hazing, staying away from like the... So does that just mean, like, crazy brain teasers? Does that mean, like, cutting them off in conversation? What does that mean? Yeah, I mean, well, it means all those things, right? So it means crazy brain teasers, being mean, thinking, sort of, sort of getting slightly angry and aggressive

in how you answer their questions. Because you're frustrated by how poorly they're doing. And a trick that we use that I think helps in that case is sort of, in the case where a candidate is totally failing in the interview, like flipping a switch in your brain and going from like evaluation into teaching mode. Your like full goal now is just to explain and as friendly as possibly. Generally, you're only other person failed, right? This happens when the person has already essentially failed,

at least the problem, if not the interview. Sure. And so you're like, you're already having in their brain found, okay, this person is not passing. So I'm going to spend the remaining 15 minutes being friendly and explaining the answer to this question. Sure. Rather than continuing to try to elicit responses from them.

And what about just the dynamic? Like, do you advise one-on-one interviews or how many people to per interviewee? Yeah. Interview panel definitely increases the stress. Yeah. So we maxed out at two to one. So training is important, right?

So if you're trying to keep it consistent, you need to have continual cross, like, people need to watch each other's interviews. And so, but two, sort of one interviewer, one shadower is enough to do that. You know, going beyond that increase of the stress and I don't think it really helps. Cool. So if people want to follow up with you and ask you questions, how can they reach out to you? Sure. My email is Amin at drillbite.com.

That's A-M-M-M-O-N. Cool. Thanks, man. Thank you, Craig. All right. Thanks for listening. Please remember to subscribe to the show and leave review on iTunes. After doing that, you can skip this section forever. And if you'd like to learn more about YC or read the show notes, you can check out blog.commodator.com. See you next week.