12,000 engineers evaluated

By Triplebyte on Nov 7, 2016

We launched Triplebyte with the goal of building the first credentials blind hiring process for engineers. Our mission is to give anyone who has the right skills, the opportunity to work at the best technology companies in the world regardless of what school they went to or which companies they've worked at.

We've now evaluated over 12,000 engineers without using their resumes. We've done this by designing a two step process. The first is an online programming test. If you do well on the test, the next step is a technical interview with our interviewing team where the interviewer knows nothing about your background. We've now interviewed over 2,000 engineers and 15% made it through to the final step of being introduced to the companies we work with.

To put that into context, companies at the size of Airbnb or Dropbox would expect to do technical interviews with approximately 50 engineers a month. We're already interviewing 3x that number every month and we've built software to track every tiny detail of what happens during these technical interviews. This means we're getting data on how to accurately interview an engineer, faster than anyone. In total we've now done over 3,000 hours of technical interviewing, or 127 full days.

Once an engineer makes it through our process, we match them with companies they'll be a good technical fit for. As we wrote before companies disagree significantly about the types of engineers they want to hire. We're optimizing our matching process for accuracy so we gather as much data about the technical preferences of the current engineering team and use that to match engineers with them. We get this by having the current engineering team complete a technical questionnaire that gives us a fingerprint of their hiring preferences. This matching model is working really well. At our partner companies like Dropbox and Cruise, we're seeing offer rates on our candidates of over 60%. That's more than 2x better than the average they see on their own candidates (the industry average is about 25-30% onsite to offer rate i.e. about 1 in 4 engineers who make it to onsite will receive an offer) and our candidates are going straight to an onsite interview, skipping recruiter and phone screens.

What's really exciting about such a high offer rate is that we're achieving it without doing any culture fit screening. Our process only looks at technical skills and that's the data we use for matching engineers to companies. That shows the way for companies to hire more engineers is to get better at identifying candidates with the right skills early on, not doing more culture fit screening early on.

We can also beat companies on the most important metric of all - the number of internal engineering hours they have to spend per new engineering hires. Sequoia recently estimated that it takes a company 82.5 total hours to hire an engineer. Around 30-35 of these are engineering hours. We're able to deliver an engineering hire at an average “cost” of 15 internal engineering hours.

We're only able to interview so many engineers each month because our programming test can accurately identify good engineers. After many iterations our test now has 70% precision (precision means of the engineers our test identifies as being good, how many of them actually are). This means we can identify great programmers with much higher accuracy than a resume screen and only interview the good ones.

What's unique about the programming test is that we've iterated on the questions we ask using actual data from interview outcomes at the companies we work with (i.e. we correlate performance on specific questions with performance on the onsite interviews at companies like Dropbox). We only keep the questions that have high signal and continually replace low signal questions with new ones. This process is the only way you can build a test that is actually accurate at screening for what companies want.

The data set we're building is unique and we use it to improve the accuracy of our process over time. The data set is unique because:

We track a lot more data from each interview than a typical company would. We've built specialized software to track data points like how long it takes to complete each section of a problem and what the interviewer is thinking every 5 minutes throughout the interview.

We also get data on how our engineers perform on the onsite interviews at the companies we work with. We use this data to build a predictive model specific to each company, based on our engineering genome, which we continually update over time.

It's really exciting for us to see that a credentials blind technical evaluation can work at identifying good engineers and matching them to the right companies.

Get offers from top tech companies

Liked what you read? Here are some of our other popular posts…

How to Interview Engineers

By Ammon Bartram on Jun 26, 2017

We do a lot of interviewing at Triplebyte. Indeed, over the last 2 years, I've interviewed just over 900 engineers. Whether this was a good use of my time can be debated! (I sometimes wake up in a cold sweat and doubt it.) But regardless, our goal is to improve how engineers are hired. To that end, we run background-blind interviews, looking at coding skills, not credentials or resumes. After an engineer passes our process, they go straight to the final interview at companies we work with (including Apple, Facebook, Dropbox and Stripe). We interview engineers without knowing their backgrounds, and then get to see how they do across multiple top tech companies. This gives us, I think, some of the best available data on interviewing.

Bootcamps vs. College

By Ammon Bartram on May 19, 2016

Programming bootcamps seem to make an impossible claim. Instead of spending four years in university, they say, you can learn how to be a software engineer in a three month program. On the face of it, this sounds more like an ad for Trump University than a plausible educational model.

But this is not what we’ve found at Triplebyte. We do interviews with engineers, and match them with startups where they’ll be a good fit. Companies vary widely in what skills they look for, and by mapping these differences, we’re able to help engineers pass more interviews and find jobs they would not have found on their own. Over the last year, we’ve worked with about 100 bootcamp grads, and many have gone on to get jobs at great companies. We do our interviews blind, without knowing a candidate's background, and we regularly get through an interview and give a candidate very positive scores, only to be surprised at the end when we learn that the candidate has only been programming for 6 months.

How to pass a programming interview

By Ammon Bartram on Mar 8, 2016

Being a good programmer has a surprisingly small role in passing programming interviews. To be a productive programmer, you need to be able to solve large, sprawling problems over weeks and months. Each question in an interview, in contrast, lasts less than one hour. To do well in an interview, then, you need to be able to solve small problems quickly, under duress, while explaining your thoughts clearly. This is a different skill. On top of this, interviewers are often poorly trained and inattentive (they would rather be programming), and ask questions far removed from actual work. They bring bias, pattern matching, and a lack of standardization.