Interviewing is a skill that takes a lot of practice. Lately I have been spending nearly 50% of my time talking to developer candidates and I still suck at it. Despite all my self-doubt however, there’s one thing that I’m quite confident about:

We do the best coding tests.

How do I know? Here are some quotes from different candidates from their answers to rejection emails:

“I’m sorry to hear that, but I must admit it was great experience for me”

“The exercise was very useful”

“I’m obviously disappointed by the outcome, but I’m also happy to have gone through the process […] the overall process [was] indeed a good exercise”

“It was a nice experience”

“This was [a] very inspiring experience for me.”

“Thank you for your time and for allowing me through your process. It was really great.”

It is not a big secret that you absolutely have to get your developer candidates to write code during the interview process. Technical interviews without code are useless. Every once in a while I run into an interviewee who outright refuses to write a single line of code in the process. That’s an easy reject.

The ultimate coding test

Unfortunately, FizzBuzz can get you only so far. Here are a few more skills you want to assess besides the ability to write a simple program that works:

ability to write readable, well-structured code

understanding and adapting to an existing codebase

clearly communicating and reasoning about the code

writing robust, meaningful tests

being resourceful

being trustworthy with your commitments

familiarity with the platform or language you are hiring for (yes, this comes last on my list)

This is why creating a good coding test is hard. Luckily I know a place that is full of publicly available programming challenges that can measure all above points. It’s called GitHub.

Indeed, our coding exercise is simply this: “Contribute to an open source project of your choice.”

This exercise, done right after a short intro call and before the technical interview, ticks all the “skill” boxes above, and more.

Besides providing us with the best possible test for the actual job, the resulting the code will have real value and, regardless the outcome of the hiring process, the whole community benefits from it. Often the ripple effect is greater than the contribution itself; most candidates have never contributed to open-source before. We use countless open-source tools ourselves (I’m sure you do too) and it feels great to add to the potential workforce.

The rules

Candidates are absolutely free to choose the project and the feature. Some choose tiny projects that I have never heard of, some choose well known, popular libraries. Some decide to work on brand new features, others prefer refactoring the existing code or fixing a few bugs.

There’s a handful of rules, though:

It can’t be your own project. Writing software from scratch or adding to a codebase you have written is much easier than figuring out someone else’s code.

It can’t be a past contribution. On the interview that follows we scrutinize the code and “when I wrote it I didn’t know it was for an interview” can’t be an excuse.

It has to be large enough so that we can talk about it for a good 20-30 minutes at least. This, of course, is very subjective. The point is to avoid too small and uninteresting contributions.

There is no deadline but there is a commitment. If you fail to honor it, you are out.

Once the developer candidate has completed the contribution (that is, has an open pull request or patch available), we schedule the technical interview. First we start with the code review, then in the 2nd half of the call we talk about more generic topics, often making references to the code. It’s great to have something so tangible.

Lessons so far

We have been doing open-source coding interviews for less than a year and we still have a lot to learn. Here are a few lessons we have learned so far.

First of all, some people flake. Either because they find it too hard to complete the task or because they were not that excited to join us in the first place. Whichever it is, we don’t mind them churning. It’s a great filter.

Second, the code itself matters the least. Of course, it’s not irrelevant at all but we gather way more information from other sources. The most important is the discussion about the code. We like people who can explain and defend design decisions intelligently. There is also a lot to learn about the candidate from which project and issue they chose, how they communicate with the core contributors, and how they follow up on the status with us. All that information is very useful when selecting your future teammate.

Third, evaluating is hard. Each contribution is different and to make the right decision you need to understand the code well, regardless of the platform or the specifics of the given project. My teammates often assist me during the code review for that reason. On the other hand, we all greatly enjoy these interviews.