Y Combinator Startup Podcast - #66 - Breaking Down HackerRank's Survey of 40,000 Developers with Vivek Ravisankar

Episode Date: March 30, 2018

Vivek Ravisankar is the CEO and cofounder of HackerRank, which was in the Summer 2011 batch.They surveyed 40,000 developers on things including their favorite frameworks, what they want in a job, and how they learned to code. You can read the report here.The YC podcast is hosted by Craig Cannon.

Hey, how's it going? This is Craig Cannon and you're listening to Y Combinators podcast. Today's episode is with Vivek Ravisankar. And Vivek is the CEO and co-founder of HackerRank, which was in the summer 2011 batch. HackerRank is a technical recruiting platform that assesses developers based on coding skills. And you can find them at hackerRank.com. All right, here we go. All right, Vivek, why don't we start with what you guys do and then we'll rewind to before you even did YC? Yeah, sure.

Some Vivek, one of the founders and CEO of HackerRank, our mission at HackerRank is to match every developer to the right job with the underlying driver being skill. I mean, so resumes are just a very poor correlation to skills. So that's how we got started. The way, what we do as a product, as a developer, you could come on to HackerRank to practice and improve your skills across different dimensions, whether that's algorithms or artificial intelligence or database skills. And when you're applying to a company, instead of going through the traditional route of

uploading a resume and it's mostly going to be a black hole, you take a company's coding challenge. So if you want to apply to Airbnb, you take an Airbnb's coding challenge. If you want to apply to VMware, you take a VMware's coding challenge. And if you have the required skills, basically we automate the whole code and give you a report, then you get called in for an interview. So it's great for developers because it's a merit-based process. is great for companies because now you're able to track the right developers for their interview process.

And what might one of those challenges look like? For instance, Airbnb, are we being put into the code base in any way? What would it look like? Yeah, I think that it varies. So we've built a platform where it's super custom for companies to go ahead and create. So you could have a simple problem solving challenge to something close to a real world where you could have a, you could give an access to our GitHub repo. Okay. And people can go ahead and modify the code, fix a bug on your, on your repo to go ahead and do it. So that's a wide range of problems that you could give on the platform. Okay, gotcha.

So you guys haven't always been doing this, correct? In its exact instantiation. Yeah. So let's go backwards. Before I see, what were you working on? Yeah, sure. So Hari is my co-founder. Right.

We knew each other from college. We used to do a bunch of things together in college. Okay. And we started the company technically. We quit our jobs. I used to work at Amazon as a developer before this, where I did a lot of technical interviews, which is kind of where I saw the problem.

You know, the resumes were not a good correlation to skills. So we quit our job. The way we started to do, we thought the way to solve this problem of resumes, not correlating skills, was to help prepare students for their interviews. So we started Interview Street. That was our first version.

And what it did was mock interviews. So let's say you had an interview call with Amazon or Google next week. You could come onto our site and attend a mock interview with either somebody from Amazon or somebody, an ex-Amazon person who can give you a walk-through of here other things. We'll do a mock interview and give you what are the adios that you need to improve. Where do you stand in all of those things? And the students would be paying these people for their time? That's right. Okay.

So it's going to be, you know, it's going to be hard for people to comprehend the value here in the U.S. Because we were doing it in India. But we were charging $350, Indian currency, from the student. And we used to pay $250 to the interview work. So $100 goes to our pocket. Right. So you could almost imagine that to be something on the lines of, I mean, technically it's $6 and $5. Yeah.

But as you mean... You earn different amounts. Yeah, different amounts. You could say a student would pay $200, something like that, and an interviewer would get about $150 here. So then if you then think about it in that way, it was pretty expensive for students. And we didn't have any good payment gateways or anything of that in India.

So we used to set up campus ambassadors in different colleges. And Hari and I used to do a trip every month to visit all of these colleges to collect money from these campus ambassadors. Cash. Yeah, cash. So our accounting was all pretty funny. And from an interviewer perspective, $250 or like $150 or something on those lines here,

is not very, it's not enticing, right? If I have to spend an hour of my time doing a mock interview and giving you feedback, is it worth it? So I still have the Excel, which projects us to make, I think somewhere on the lines of $100 million or 100 crores, whatever it's like $50 million, something along those lines. Given the trajectory in the beginning. Yeah.

I think we made about $4,000 or maybe about that's like an equivalent of $100 here. Yeah. Okay. After a year or a year and a half. So clearly you could just drag the cells in the Excel in whatever way you want and get the numbers, but it's very hard. to execute. So that failed. I mean, we had a lot of lessons that we learned from that. Most importantly, I mean, the students didn't have cash. There was a lot of scheduling. And I think one of the things that

YC stresses on the most, I wish we had actually done YC at that point. I think this was done by Paul Bouquet, make something that 100 users really, really love versus making 1,000 people or 10,000 people sort of okay. I think we should have focused on, you know what, we're just going to on these two companies, mock interviews, versus any tech company, anywhere in the world, and we were scrambling for interviewers from Amazon, Google, Microsoft, all over. We should have just focused, you know what, these are the three companies that we're going to focus.

If you want to get a job here, you can come in to and make it really happy and then figure out a way to scale. Looking back. Yeah, of course. But did people like the product? Yeah. Whenever people did the interviews, they really liked it. In fact, there were a lot of really interesting testimonials of people who actually

attended the interviews, got their job. They knew what are some of the interviews. the areas that they need to improve, they need to prepare, and all of those things. Yeah, that's great. Yeah, I mean, I remember all the interviews I did during college, like if I could have just gotten the email of someone at one of these companies, it would have been invaluable to me. We used to do that sort of informally in college, right? When we were in the final year of college, we used to train people for sort of mock interviews and others. So this was more like an extension. Why can't you do this on over the web across everyone? Interesting.

So you weren't necessarily off the mark around finding something that people really like, but you found that you kind of hit a ceiling somewhere. And it wasn't very focused either. So there were a lot of, there were different cohorts of people. Some of them were students. Some of them were a couple of years into their work experience. And some of them wanted to do technical interviews. Some of them wanted to do sort of a automobile interview.

And then we had to go ahead and scramble and figure out, can we get somebody from the automobile division to come and take the interview? We had no contact. So we had to figure out a way to scramble through language. in and Facebook. I think it was sort of okay. But it really taught us to stay strong. Okay. And so at what point

do you decide, you know what, we have to change this. This isn't working. Yeah, it's probably about 18 months. Well, we had applied to YC. Oh, you did? Okay. We'd applied a Y combinator with this idea. Of course, we got turned down. Then we, I think it was about 12, 15 months, something on those lines when, I mean, clearly this wasn't going anywhere. The Excel was completely off the trajectory of

the Excel graph and where we were. And then we switched our business model or idea to something different. So there is a big trend that is still present in India, which is once you graduate, you have to do your master's in a

school here. It's huge. I mean, pretty much, you know, I don't know, 80, 90% of people would attempt their GRE, which is your standard way of getting here, your Tofell, they will prepare their statement of purpose, they'll do all of those. And the application form is pretty expensive. Right. And so you could only apply to maybe five or six universities because it was so expensive and you couldn't just, you couldn't just apply to all the schools in the U.S. So what we did was we connected people who were already studying in these schools,

who can review your statement of purpose, your resume, and tell you if you have. had a chance to come here to apply to the school. So that was our pivot. That actually took off a bit, but then after three months or so, the traffic was down to zero. Turns out you can only apply twice a year. So we didn't know what to do for the rest of the year. We'd actually apply to YC even with this idea.

There you go. We got turned down. So I think this whole thing about mock interviews, applying, helping people apply for your master's, prepare. Maybe a year and a half to two, something on those lines that we worked on here. Okay. And then what sparked you to start working on where you are now? Yeah, we were close to bankrupt. Yeah. Okay. So there was a real need. And, you know, of course, we had really good jobs.

And Amazon was a great company to work at. Were you in India working on Amazon? All of this is in India. Okay. And so then we realized, okay. So just to pause you really quickly, how are you making the connections of people here if you were based in India? Sure. So what we did, was we used to take the directory of people from our school who went head and did master's at all of these different universities

and who were working in different companies. So we used to go through the directory, keep calling people, email them, hey, are you interested? We are from the same university, put that university touch in to get them on board. So it was just friends, networks, second degree. And I used to make, I don't know, 50 calls every day to get this going. Okay. And just for context for me, people from India want to do a master's here because it's like a gateway to getting a job here or what?

I don't know how it started. Maybe it was just, I think it is, it's sort of a checkbox. Okay. I'm not sure that this changed now, but it was definitely, I mean, I graduated eight years back. It was, or nine years back. It was definitely a pride. Okay, you know what? going to do my master's here. And there's, of course, some glamour associated with the great American dream coming here and getting working in the Silicon Valley. Silicon Valley is sort of this mythical place for all technology developers to be here. So it was more like a checkbox.

I still think it is the case that you would have to apply. If you had good GPS and if you had good grades, the question is, why aren't you going, why aren't you applying for a master's? That's funny. Even among the very talented smart kids going to great schools in India. Yeah, for sure. I think it's why aren't you applying for your master's? Why aren't you doing your PhD here in one of,

in Carnegie Mellon or Stanford or Purdue? And there are a lot of, actually, I might, Hardy and I might actually be the odd man out. Because you didn't do it. Yeah, we didn't do it. Yeah, exactly. And I'm happy because I'd apply to six schools and all of them turned me down.

I was so dejected when the last school turned me down, but in hindsight it's great because I don't think we would have started the company if I'd come here. No, probably not. It is the conservative move. Yes, and you have to, once you graduate, you'd have to get a job because you have to pay back the loan, which is pretty big. So you do two years of masters, then you do two to three years of working in different

companies to pay back the loan. And there's all of these visa things. And by the time you want to start, you're already, I don't know, close to five, six years invested in this. It's really common. And the friction to or the inertia to go ahead and start another company when you're happy working at Amazon or Google is going to be hard. Yeah, definitely. And so, okay, so you're basically out of money. Yeah, we're close to close to bankrupt. Yeah, yeah.

And yeah, so we had a clear choice, which is we could go back and get a job. I think we could have got that. But something we just wanted to keep going. We just wanted to try this out once more. And I have no idea what kept us going at that point in time. And there was some conversation we were just walking around, which was,

it sounds obvious. Hey, if you have to make money, you have to sell something to people who have money. Otherwise, there is no way that you're going to make any money. We were selling to students and others who were already in whatever they had student debt and they had very little money. So that's when we said, hey, can we invert this,

which is can we build a product for companies to help them identify the right developers based on our skills? Of course, we didn't have the mission written down, like match every developer to the right job. But looking back, I think that was the code of every decision that we made. And we felt, you know, we'll still stay true to our core because it doesn't, it's, mock interviews is one way of doing this. But if you actually give a platform that can enable developers from, it doesn't matter

what school, did you go to which company you went, but you have a chance to showcase your skills and get the dream job that you want, then that's great. You know, we should build a platform for that. You know, that's what we got started, and that got the early attraction. So that was the third time we applied to YC. And yeah, YC called us that time. Oh, all right. And was it because you were showing significant growth at the time, or the idea was just working out?

I mean, I don't know. I definitely didn't want to ask, why are you guys calling us now? Because I didn't want to point a chance. Yeah. Is this a mistake? Are you sure? So that was our first trip to the U.S. Oh, man.

Yeah. And we had some struggles there as well. Hari, my co-founder, couldn't get his visa. to come here. And I still think it's the stance of YC, but YC cares about co-founders a lot. And so, so I got my visa, Harry couldn't get. And so I flew in here just for the 10-minute interview. This was, yeah, six years back. And fortunately, we got in. And I remember, you know, I think there was a panel of interviews. It was PG, Sam, Harch, Jessica, and Paul Bouquet. There were five people interviewing me. Whoa. Okay. And, you know, that's the first, I landed in the morning and the

evening was the YC interview because I couldn't get my visa on time. So you were the last day of interviews? Yeah, last day of interviews. Probably the last one. Yeah, yeah, could be the last one as well. Could have been. And then you see all these five people as a creator of Gmail. You're obviously a little stressed.

And then you don't have a co-founder. And PG asked, where is your co-founder? I said you're going to have his visa. But it was a good, intense 10 minute. And frankly, I didn't think we'll get into YC. So it was, and maybe it worked to our advantage because it was just completely open and honest about everything. Because I didn't fake any of my answers or try to impress them. Look, this is the problem.

I really want to come here and go ahead and do this. And it probably worked out to our advantage. Interesting. Yeah. I mean, so what you were working on before has this separate difficulty in the application because you have to educate someone about what it's like in India versus here where we have the exact same problem. like, oh, we're this company, we have all these jobs. Yeah.

We don't have enough engineers. Like, let's sort this out. Yeah, yeah. I think there is a, there is a different way things. I mean, yeah, you're right. The environment is different. I mean, I remember I used PG's, so I didn't have, so I used to sit in the Pioneer Way office, one combinator office for most.

Actually, our first office was right opposite to the YC Pioneer Office. Yeah, yeah. It was the RethinkDB's old office. Okay. Which we actually took. So it was right opposite. So I used to be in Y Combinator for most part. I remember I took PG's charger, Mac charger, by mistake, home.

And this was the second day or the third day. And I was so kind of, I didn't know what to do because, oh my God, what is PG going to think? And then I started emailing. I'm really, really sorry. I took your charger. He said, no problem. Just keep it there tomorrow. And I said, okay, are you going to come in tomorrow?

and he said he's not going to come in tomorrow. Because the way, at least I was brought up, for most part in India, is if you keep a charger or if you keep the Amazon box right outside the door, somebody is going to take it away, right? And even the tiny things. And then you have to learn about how the environment is different,

how people talk, how people react to it. And the dinner time here is 6.30. So I remember sending a calendar invite for dinner to a person that I want to recruit for 9 p.m. And he said, that's kind of my bedtime. So I can't meet you in 9 p.m. So there are all of these things that you have to learn. And it's not just the macro part of the developer, how it works, but even the tiny things about how you interact, how you work with people and all of those things you have to learn. Yeah.

Has that been incorporated into your product? Because, you know, I imagine you have all these international people applying to U.S. companies. Yeah. How do you help educate them on the cultural norm? Yeah, no, it is a really good question because right from how you sort of the types of sentences or words that you use, varies quite a bit between what you would say to somebody in Asia, what you would say somebody in UK and what you would say somebody here. So all of those nuances we need to take care of because you want, there is a different cultural aspect on how you sell, how you talk, how you get people. people going here. It's really difficult. How long did it take you in particular? I think it's a

far-over learning. I don't think I've perfected it. I've gotten my dinner time now at 7 p.m. I've shifted a little. But I think it's a constant learning. I don't think it's ever going to stop. Yeah. My friend moved to India for a couple years and he explained the exact thing opposite. Yeah. It was tough for him. Yeah. I mean, it's just the way you grow up is different. I mean, there's nothing that's right or wrong. It's just different. And you have to get used to it. And I remember PG and Sam correcting me from everything to even the tiny nuances. So in India, you would probably say programmer. You would add the A at the end, but here people will not be able to follow. It's programmer, right? So if you have to say, and when I was doing my

demo day pitches, all of these little things mattered because it was just a five-minute pitch or, no, I think it was a two-minute or a five-minute pitch in the demo day. And every little thing matters a lot because when you're talking to investors. So it's not just the accent, but how you talk, how you schedule meetings, how you follow up, and everything is different. Yeah. Demode is tough universally because, I mean, you're pitching amidst the group. I think you said your batch was like 60 or 70. Yeah, I think our batch was 63 or 65. Yeah.

And so that's enough companies for one to just fall into like monotony, like, mur, murm, right? No one pays attention. But interestingly, I really enjoyed it because I've never been exposed to, exposed to such a large group of investors. And I couldn't even sleep the previous day. I was looking forward to it. Look, I wanted to pitch and get a Cidysay going. So I frankly enjoyed it.

I was looking forward to it. That's very cool. So you raise money after YC and you're committed. You're like, okay, we're going to stay in California at this point? Yeah, I mean, after I came here to Mountain View, and I don't know, maybe that's just partially the reason. The first city, well, I technically landed in San Francisco, but immediately took a cab and came to Mountain View.

It was, I just, I think everything, everything looked great for most part. I mean, right from the kind of the people that you interact with, the advice that you get, the weather, the food, sort of the perfect mix for me coming from India. You had the weather similar to Bangalore and the food similar to Bangalore and Chennai, and then you had all the people that I really wanted. order to talk and get advice from in terms of building a company. So it was a perfect mix at least for me. So I was very much determined to stay here and build a company. Man, what was that like, you know, flying here and then seeing all the people who you,

like, you read their blog post? You know, PG's an easy example. Yeah. What was that experience like? Yeah, it was kind of surreal. I didn't know it was, as I said, I thought I was going to just stammer and flunk the interview. But like I mentioned, I didn't think we'd get in. Actually, we were the first Indian-based company to get in a Y Combinator. Really? Yeah. And so that was another quote-unquote, I don't know if it helped or didn't help because we thought, okay, they've never chosen a complete Indian-based company. So our odds are low.

That really helped because I was just completely free and open. But it was great. It was great to meet all of the people that you admire far off from reading their blog post or Twitter or watching their videos. Yeah. And what were your greatest learnings from the experience? From YC? Yeah. I think the intensity was very, very high because there was a dinner every week.

And I mean, I noticed that each founder had their own set of people that they interacted with. Of course, I interacted with a lot of them, but you talk to a few people and then you hear their updates every week that's, and you want to push harder. And you have a fixed deadline, three months down three months. you have a demo day, which you have to go and present to investors. So you have a fixed demo, a fixed timeline to make sure that you have a good prototype. You have possibly paying customers to go ahead and do it. And I think the bar is continuing to increase at every demo day. So I think the intensity, I think the learning is, I still think Hardy and I have that intensity for most part, which is good.

Wow. How do you keep it up? Because people talk about this all the time. They're like, it's so cool having a cohort that you can kind of benchmark. yourself against. Yeah. But once you're out, you're kind of out. Yeah.

I don't know. Maybe we're just, it's DNA or we're just consistently hungry to win. Okay. But I think the challenge is, can you extend that across the entire company? I don't think we've done that. We've done an okay job at that. Right. So I think that's something that we still need to get better at.

That's a challenge. So to fast forward massively to now. Yeah. The reason we started talking about doing the podcast in general was that you guys, release this developer skills report. So just for folks who haven't seen it, can you explain it? Yeah, sure. So we have a developer community that's over, I think it's 3.4 million as of last week.

Wow. That's slightly over 10% of the world's developer population who come to Hackerang to solve challenges to practice their skills and get better. And one of the things that we wanted to do was to learn more about our developers. So we sent a survey. Actually, it had 40 questions. There was a lot of debate. You know, we'll develop,

firstly, we'll develop a survey. And then you're sending a survey to developers with 40 questions, something on those lines. What are the odds that somebody was actually going to fill this? So we sent this to a million of our active developers. Whoa. And it was amazing. We got close to 40,000 people complete the survey.

That was huge. I think Stack Overflow had 60,000, and they have 20 million developers or so. So if you compare the ratio, the engagement and how many people completed the survey was giant. And hopefully we can beat Stack Overflow's number next year. Yeah, what was your email subject line? How did you get so many people to convert? That's a good question. We did A-B test, but maybe Nicola or our marketing team, Ritika,

they might have a better idea on what was the subject line. But we did A-B tests a few things. And we also tested on the timing, you know, when to send to which cohort, depending on the country, depending on the geography and all of the others. Because you're fairly distributed, right, in terms of your user base? Yeah. So maybe it's about 26 to 28% in Automatica, about 40-ish percent plus in Asia-Pacific. And then you're talking about 15 to 20% in the UK,

and the remaining is the rest of the world. Just wherever. Okay. So for each of the GOs, we also. optimize the subject line, when to send them, and how do we actually position this? So we did a bunch of things on those lines. Was it dialed in for language or all in English? I know. It was all in English. I don't think we changed any language. So the questions were waiting from when did you learn to code to what do you look for when you're switching for a job or if you're a hiding manager,

what do you look for in a candidate? So we had a wide variety of questions and some really, really interesting insights. So we had, we engaged a firm who could parse the data, who could do real analysis on this. And we're also likely going to put this up on our website for all of our developers to go ahead and do the analysis with our data. So something might happen maybe in the next couple of quarters. Yeah, it was, it was a really interesting insights. And we published this earlier this year. And we got over half a million developers to read the survey. I think this was a, this was a huge milestone and exceeded kind of at least my expectations in both the quality of the content, the design, the reception that we got. And of course, this podcast wouldn't have happened because of

that. So it exceeded my expectations in all dimensions. So, all right, I guess the most important question is, what programming language is the most popular? I think it varies across different industries. And, you know, for, I think the one that people want to learn a lot is Python and the that's starting to get a lot more popular is go. But it varies across different industries from what financial services want versus the fast-growing internet companies want versus retail.

All of these companies are transforming into software companies. So each of them have a different variation of it. Okay. And so for a developer who's curious and sees this survey, do you kind of funnel them through a flow on your site? Like if you're interested in getting into banking, you ought to be learning X. Yeah, it's a good question.

So we are trying to do with the rivers, which is what do you want to learn? What language and skills do you want to learn? And then we'll help you recommend. You know, where can you most fit? Where does Hacker Ranks see you the best fit or the best job for you based on all the data that we've collected? Okay, gotcha. So let's go through some of the facts. Yeah, sure.

Some of the fun facts. So I saw there were in the beginning of the report. It's a lot of stuff about age. Yeah. So when do you? people start learning how to program, when does it all begin? Yeah, I mean, I think the most, the fun insight was one in every four developers, learned to code before they learn to drive. Yeah. That was a, that was a really interesting insight that we were able to get. And it's,

it's not super surprising for me as a developer, but when we announced this, it was just, I think it was the most popular insider. We even started printing T-shirts and in our company. That was, that was, that was an interesting one. And it just shows the fact that self-learning. is starting to become big. People are learning on your own. You don't necessarily need to go to a particular school to do your four-year degree and then understand how to code

or how to build applications. And it's just going to, I think, over, if you just extrapolate the trend line, I think this will be two and four or three and four developers will learn to code before they drive in the next decade or so. Yeah. And did you collect any data around boot camps,

coding boot camps? Yeah, we did collect. I think the, We collect the data on how much the hiring managers or others valued the boot camps and kind of learning. It was mixed, to be frank. I think there were certain hiding managers who valued it a lot and certain hiring managers who didn't. And my guess is probably depends on the quality of the boot camp. I mean, there are a lot of them who, there are a set of bootcams who say, you know what, come here.

It doesn't matter whether you know how to code or not. I'll help you become a great React developer in 90 days. I mean, that's not how it works. and it almost seems like a negative signal for sometimes, which is, oh, you didn't know how to code 90 days before and now you're claiming that you're a React expert, you know, that puts me off. So I think it depends upon the quality of bootcams. And then on the other extreme, you have this school, which I was personally really impressed by, it's called 42. It's a school, actually PG tweeted about it. Is that one in Fremont?

That's right. They just opened another one in Fremont. where I had a chance to visit, where it's, you, of course, had to take a minimum level of coding proficiency challenge or a test. Yeah. And then you are allowed to explore whatever types of computer science topics that you really like and then figure out which one you have a natural attraction or a passion for, whether it's security or machine learning or a front-end developer, and then continue to get

better at that, and then will help you connect to the right job. So that, that seemed to be a different approach than a lot of people. But it's massively different, right? Because if I understand it correctly, they're giving people room and board. That's great. And it's like years long. And it's free. And it's over the course of years.

That's right. Right. Yeah. So, yeah. It is, it is very different. It'll be interesting to see how that experiment works because I think it was started by a really a big billionaire in France or something who just wants to help the world. So I'm not sure how it might work from a business value proposition, but at least in terms

of a net improvement to the world to create more developers and to help improve technology. I think that's a great start. So I think that's why the boot camps had a mixed one from the hiring managers at least. Okay. And then what are the differences between, you know, startups hiring and big, you know, like Amazon type companies? Yeah. I think the one common thing was everybody stressed on the importance of problem solving

skills. Okay. In respect to whether you were a smaller. a company or a bigger company and others. And the one thing that the startups or the execs cared about the most, because at least in a startup, you might have a recruiter or not, but most likely you're doing the recruiting on your own. They cared more about your contributions or your open source contributions or your

profiles or portfolio much more than what a large company cared about. But at the heart of it, people wanted strong problem solvers. And also one of the things that was present in the survey was how you wanted to dive in deeper on the knowledge of that particular stack, at least for a startup. Because, of course, you have very low or pretty much no bandwidth to train somebody on the new stack that you're working on versus a large company where you have a little more bandwidth to go ahead and train. So you're okay with a generalist who can come and learn a new language versus in a startup. I like you to know the particular language or the stack before you come in. So those were some of the nuances of the differences that you can see between a startup and a large company. So in terms of structuring a job offering to someone, or rather just like a job posting, what are the things that end up appealing to developers?

Yeah. I mean, this was very interesting because I assumed differently, and of course, always data over opinion. And one of the things that we learned was how work life balance beats all of the different perks that people offer, whether that's free food. or we have a ping pong tournament every week or things along those lines. And then I actually didn't believe this data, firstly. So I wanted to do another survey for people who did the work-life balance who actually opted in.

What would you really mean by work-life balance? And what we learned was a lot of developers have their own projects that they want to learn, that they want to build on the site. and the environment that you're working should give a good amount of balance and probably even support, if you can do, on the things that they're learning on the site

or they're building on the site versus what they're doing for the company. So that was kind of an interesting insight about work-life balance. And one of the things that I'd like to do at Hacker, and we're still discussing on how do you position this in the right ways, if you have any of the site projects

that you're working on, will fund the infrastructure cost for that. So we're still figuring out how do you position this? How do you give this that way? And I'm pretty sure after people hear this podcast, they're going to email the developers at least. But that's one way of helping or encouraging people to do that as well. Oh, interesting.

And so what are the other ways that companies signal that they have that work-life balance figured out? Or is it just like folks come in for the interview, they get a sense for it, and then they say yes or no? Yeah, that is one. But I also think if you're actively contributing to open source, for example, we are improving a lot of the open source code editors that are there right now. We've not done a great job in letting the world know about all the improvements that we've been doing.

But that's one way. And of course, interviews help a lot in understanding more about the developer culture. Okay. And do you guys have any rules around working on side projects? Because I know that at larger tech companies, there are totally issues. and like, you know, I don't think they really flex that muscle very often if ever. But yeah, you know, like companies like Apple are infamous for not letting people do side projects. Yeah, yeah.

I mean, that's probably one of the advantage when you're small, which is, I mean, as I don't think we have any rules. I don't think we should put any rules because it's just freedom or creativity to go ahead and do it. And frankly, if it's going to be conflicting with your core idea, and you're scared about it, maybe you should take the developer's idea and implement it. Maybe it's better. Or otherwise, you don't need to be scared and you can just have the developer work on the projects that they want.

So I don't think we should put any rules, but I agree. I mean, a large company like Apple or Amazon or Google will need to put in constraints. Yeah. Although Google hacked this by putting a 20% rule where you could 20% of the time you can work on your projects. I think that's how Gmail was born. on where I think Paul Bouquet worked on his 20% free time to build Gmail.

So that's how they hack it. It's not really technically a site project that you do outside. You use all the Google infrastructure and libraries and eventually contribute to Google. So that's one way you could try and do. Interesting. What else is attractive to a developer? Yeah. And the other thing that we had on the survey was really on continuous learning.

I think there are just so many new things that come up, layers of abstraction, which is obviously very enticing for a developer, new frameworks. I mean, there's a framework for Java that's getting launched every month or every two months, which is kind of driving crazy because you just have to keep migrating from one hour to the other. And the other thing was continuous learning.

Can I learn new things? Can I learn how to get better? So that is a forever learning attitude or culture. What else did you learn about how developers are actually educating themselves. So obviously some percentage are going to boot camps. Some are doing CS. What is everyone else doing? Yeah. It was really interesting. Even though you had a CS degree, a lot of developers said they actually learned a lot from YouTube over books or sites like Hacker Rang. It's not a plug for Hacker Rank, even though I think the title might say this or whatever.

But that was really interesting because that's a new, that's, this. It's going to change. That's a paradigm shift, right? Because if you look 10 years back, and paradigm shift in recruiting, because if you look 10 years back, you're going to use GPAs and universities and the companies that you worked at as proxies. Yeah. But now the proxies are changing, right? It doesn't matter if you went to a great schooler and had a great GPA. If you learned on your own and built a lot of things on your own, that is what is needed.

That's all is enough. So the way that you look for in a candidate, who do you look at? for how to assess, how do you get people in the door, and how do you remove biases that's been there for, not 10 years, that's been there for 100 years. So do you go to the school or do you work in this good company? How do you remove those bias and truly focus on the skill? I mean, it's a lot of work. It's going to, but that's a big paradigm shift.

And it's a very good and healthy trend. If more developers can learn on their own for free, you don't have to pay $100,000 for tuition. It's just good overall for the world to do that. Interesting. And so how would someone stand out? Like, what are the kinds of projects that end up appealing to a company? Because I know if you do a lot of these like boot camps, like, great, you made your own CMS.

Like, so did 10,000 other people. Yeah. Did you get any insight as to, you know, when someone does care about these projects, what are the kinds of projects that appeal to companies? Yeah. I think it's hard to, I think there's one part of it, which is your sort of motivation or your drive, which can really come through, which is, I mean, if you look at, you're right, if you just forked repo and made some, made a couple of edits in one file, it really doesn't, you can't claim that you actually done, contribute to open source or anything. But I think one is definitely

indicates a drive. The second is also, and which is probably hard to just look at a hackerang profile or a GitHub or any of the profiles, you can't just look and figure it out is the depth of thinking and what are all the things that this developer wants to do more. And that's something probably you can get from the interview. But the bigger question really is, would you be willing to bring this person on site and go through the interview process based on their skills and not on their pedigree? And it sounds obvious, but we're trying to educate our customers and everybody who are hiring developers that this is the right way to go.

And we're sort of leading and pushing the movement. movement for this and hopefully we can be successful at that. Cool. And how does this affect a situation where you're hiring remote developers? Are there any other particular cues that you pay attention to? Yeah, what do people say? Yeah, I think the communication aspect is always a big deal. How do you get everybody in sync? I think some of the companies have done an amazing job. GitLab is a YC company. I talk to their CEO and it's fully remote. It's crazy. how

and it just shows how much effort that you need to put in to make sure that people are in sync with each other. I do think and you're seeing this happen a lot. GitLab is just an example but I know a lot of others,

a lot of other companies were starting to do. I think the future would be an ability for you to build out engineering teams wherever you want, not just restrict yourself to the Bay Area and a good common connective tissue that make sure that things don't break because of bad communication or because

of time zone differences and others. I think that's where the world is going to move, which means ability to know their skills and how they can contribute when you're not in the office and all of the others is going to play a very, very important part. Like ability to understand both of this at a deeper level is going to play an important role. Yeah. And have you guys conjectured or started working on anything that might be able to test someone's like, efficacy as a remote developer because it's one thing to have the technical chops. It's another thing to be able to work in like a co-working space or in your bedroom or whatever when you're in a different time zone and like get shit done.

Yeah, yeah. I mean, I think there are, there's one part which is the IQ, which is how good are you in all of these different skills where do you need to improve? And the other part is EQ, which is what your personality is. We've been doing some experiments or partnerships with a couple of them, but we've not had a real breakthrough on that yet. But I do think that is going to be, that's going to be helpful.

I mean, the reason I'm saying that is there are a lot of companies who use Hackerang to build out remote teams in India and other parts of the globe. There is a company, which is probably familiar at booking.com, one of the most popular travel companies in Amsterdam. They are able to hire developers

all over the globe using Hackerang, right? And they're doing it in a very efficient way because they understand the skills, strengths, and weaknesses before you come on site and before you fly to Amsterdam. So I think we're starting to see this movement, at least happening in a lot of companies, on the IQ part of it. And now if you can do a better job or if you can work with a partner

or if you can do something even internally at Hackerang to do the EQ, it can be a very strong solution. And so this is all done through programmatic exams. There aren't any in-person interviews to, like, assess EQ? There are some companies who are working on this programmatic way. It sounds very interesting. I've not we've not like partnered with them deeply

to really understand how well it'll fit in but it's certainly a very very interesting opportunity okay so in addition to maybe figuring that out in the future how else do you see paradigm shifts happening for developers in the future you know obviously they'll be all over the world yeah what else is going to be happening in you know five years

I mean there are a couple of things one is I think it was there on Hacker News it's as much as there are APIs and different levels of abstraction that have made it easier to get things started, to set up your environment is, and this is for somebody who's a newbie, who's just learning to code. It's very intimidating to set up something and get it running and compile. I mean, you could have, I think that's one part where things needs to get better. you know, here's an environment, here's where you can go ahead and start to code, because a lot of them give up. Oh, yeah.

Oh, I did this thing online and then I tried it out on my local. It didn't work because I don't know if you did you have this library and there was a dependency failure and you have to go and figure it out and it just becomes a big mess and you give up for most part. So I think that is still, you know, we still have to do a better job if we have to increase the number of developers in the world. You can't have people giving up after your third lesson or the fifth lesson and saying, you know, it's not taking me anywhere. And also there needs to be a purpose. I mean, the purpose can't be you're going to get a $100,000 job. The purpose has to be much more whatever is in your mind, you can go ahead and build on your own, right? I mean, you have to encourage the fact that you can build versus I'm going to get you a six-figure salary to do it. I think that's one part that needs to get better if you really have to create more developers in the world. Why does that have to get better? Because $100,000 salary, I mean, I agree with you. Yeah. But to be devil's advocate on this, like, that's a lot of money to a lot of people. Yeah.

Why does that not work? I think it will work for a subset of people, obviously. Sure. But I think in a very longer term, the really great developers are people who did it, not because they'll get a job, but because they really wanted to build something. They really wanted to get this, whatever the idea that they had in mind out. Okay. And I worry that if you had a, and this is just my hypothesis,

if you had a monetary value associated with it, you might just focus a lot on that and you will reach a ceiling pretty soon. This is just my hypothesis. So versus if you're, if you really fall in love with the idea of building or creating new things, the ceiling is sort of infinite. Okay. Because you can, there's always new technologies.

There are always new things. that you can go and build. So it's more about what is the type of behavior that you're trying to create. It's like a kid. You're trying to get into the developer world trying to do it. What is the type of behavior that you're trying to create and why you're doing what you're doing?

So that should be taken care. And the second big bucket is really the paradigm shift that I was talking about, which is more companies should come forward and encourage and say, you know, what we're about skills, we don't care about resumes. We don't care about what school did he go to as long as you have the skills will be willing to do.

I still think the proxy of your GPAs and companies exist at a lot of companies. Yeah, absolutely. I mean, I think that, like, people constantly disqualify themselves before they even enter a race. Yeah, yeah. And we see it with YC all the time. Like, you know, people are constantly asking questions. Like, you know, I raise money. Can I do YC?

Yes. Like, I haven't raised money. Can I do YC? Yes. Like, I don't have a co-founder. Can I do YC? Yes. And you often encounter this thing where it's like,

Like, you can just apply. Yeah, you can just go for it. Yeah. And, like, moving things toward a more meritocratic system, that makes a huge difference. Yeah. What else does a developer value in the hiring process, not just, like, about a company? Yeah. What do they want out of that?

Yeah. I think the, in the counterintuitive thing, at least whenever I've talked to a lot of companies, is you should also understand that developers are interviewing you. Right. It's not just a one-way thing. And developers are interviewing you in a lot of, a bunch of different things right from the process. How tight and efficient are you actually doing their peers?

And also the challenges or the questions that you're asking and how deep. Sometimes I've seen this at a lot of companies. People are scared to ask hard questions because it might turn off. It might work the other way, which is, I'm wondering, is this the level? I mean, if you just ask the easy question, are these the kind of peers or developers that I'll be working on? And of course, there's a different nuance to the hot question. You don't want to ask a complex math problem which has a trick answer.

If you didn't know this particular formula, you can get it. The hard in the sense of relevant to the business, relevant to the job, and how do you go about doing it. The other thing that I'm starting to see happen across a lot of different companies, and even Hackerenc has incorporated this, is the business acumen, the level of business acumen that you have as a developer. Usually it is, it's always developers are builders. You have this whatever requirements or spec you can go ahead and do it. But if you look at a lot of the great companies, like Google or Facebook or others, they're all developers at the heart of it. And the reason, of course, they had great products, but they also had really strong business acumen. And really great developers have that.

Or what, I think Steve Jobs, incorporated as a 10x developer, have that level of very strong business acumen. So that's something that people want to know what your strategy is. People want to know how they can contribute, how what they're building helps in the business strategy. Oftentimes, we think it's just about users who are using and MOUs and others. But no, I like to know how we're building the company. And that's something that we're trying to do a lot. It's something that you should do in the interview process as well as my recommendation. Interesting.

So in other words, ask questions about product, not just technical stuff. Yeah, and about the business strategy. You should be able to challenge hacker rank or challenge the company on, you know, why are you doing this? And here is a different idea. Here's another thing that you can do better. Oh, man. Now you're just dressing people out. Like more stuff to prep for?

Yeah, but I think it's fun. I like it. I like, you can just sit in a whatever, like the Amtcha critic, you know, you can say, oh, why did you enter this geography? I think that's a bad move. Or your business plan edition is not as confused. with your free user edition. I mean, you can have a good conversation. And frankly, and here is, I mean, all of these things are, the nuances are super important,

which is it doesn't matter the questions that the candidate asks, whether that's the right or the wrong thing for the business. You have to judge the quality of thinking. It could be completely off from your business strategy that you would actually implement because you have way more data and you've been in this, doing this for six years or something on those lines. But it's the quality of thinking and what's the framework of asking questions

that you should be judging people on? And I found those questions and developers really be at a much higher productivity at our company. What about the brain teasers? Are they still in fashion? Do developers like them at all? No, I think we shouldn't be using that. I don't know who created that.

Some company created that, clearly. I have to go back and look at the history. It's probably like McKinsey or something. Yeah, it could be, right? It is, I think the underlying goal is the right one, which is basically they're trying to think about the computational thinking, which is, I don't know, how many tennis balls will fit in a Boeing 747. I think they're trying to, again, this is the question,

they're trying to figure out the quality of your thinking. How do you think about it? What kind of data that you will analyze or how do you, how do you structure the problem? How do you break it down into sub-problems and go ahead and do it? But I think there are better ways that you can assess the computational thinking or the problem-solving skills, which is interesting because that's what the hiring managers care about the most

in a much more developer-friendly, coding-friendly way of helping people do that. Cool. Okay. So in closing, I'm curious about your experience as a founder hiring developers. Like, we've covered a bunch of stuff for like, if you're a company, this is how you hire your like 1000 and first developer. But if it's just you and your co-founder and you raise a little bit of money or you're making enough money that you can finally hire someone,

what are the things you're thinking about, what are the things you're testing for early on? Yeah, I think Peter Thiel talked about this. I think maybe one of the podcasts or one of the interviews, which is the way PayPal got built was they went and hired people who the others thought weren't qualified, but actually they were really, really good because they didn't have the traditional degree

or the traditional background in doing it. I think the first 20 people or so at PayPal were very much on those lines. That's kind of how we at least built the first five to ten hires of developers, and it's really helped us a lot. If you look at it, I mean, Hardy and I didn't go to the IITs,

which is the equivalent of Stanford, and the first five to ten developers that we hired were, people who are very smart, who are very skilled. And it's almost this quote-unquote untapped pool. And it's the secret source that they have. And then that really set everything right from their motivation and drive to do great things. I mean, the first engineers are still with our company and doing amazing things, Akshaya and Shib. And they're doing some great things right now. They're leading a team. They're managing a team. And the motivation and the drive to do

really great things, the ability to learn, the ability to assess, making sure that the peer or whoever is going to join, the company is as good or better than them. All of that has helped a lot. And I feel that's an important thing that, I come back to this whole proxy thing. The current ways of doing proxy on GPS and degrees is bad, and you have to figure out your first 10 people, almost from that untapped pool. I think it helps a lot in your drive and motivation for the long term. That's just my hypothesis, but I just have one data, from Hackeroy. No, I mean, I agree with you, but like, the short answer is that you have to do the work to find the great people. And it's like in that arbitrage of you don't have the right credentials, but you have all the skills. So you're worth working. Yeah. Also, I think the, I think at some level, people should be really passionate about the mission. And, and again, the nuance comes here, right, which is you can't expect somebody to be more passionate than you. That's never going to happen. But as long as they have a, they have a real, feel or connect.

You know what? We need to solve this problem and here are some of the ways that you can actually do it. You know, that's another thing that you need to test in the first and developers. Awesome.

All right, man. Well, thanks for coming in. Yeah, thank you for having me. All right. Thanks for listening. So as always, you can find the transcript and the video at blog.

dot ycombinator.com. And if you have a second, it would be awesome to give us a rating and review wherever you find your podcast. See you next time.