Thank you for writing such a heartfelt critique of the Fizzbuzz screening question. I’m not writing to argue with your observations: I think we both agree with your points.

First, I think we agree that is not true that 199 out of 200 programmers cannot actually program. Anyone who draws this conclusion from observing that 199 out of 200 (or any lesser ratio down to 1 out of 2) applicants for a programming job cannot program has omitted a very simple analysis.

Second, I think we agree that it is not necessary to ask screening questions, and that interviewers who fail to take advantage of alternatives when they are easily available is unprepared.

Third, I think we agree that some people place far too much emphasis on a screening question like Fizzbuzz, trying to extract all sorts of inferences about someone’s professional software development capabilities form what is—in every respect—a toy problem with almost no relationship to the work we do on a daily basis.

Fourth, I think we agree that some excellent candidates will feel anxiety in an interview situation, and that their “performance” on a test in front of a panel of interviewers will fall short of their performance under workplace conditions. And I certainly agree with your premise that workplace pressure is often not the same thing as performance anxiety.

All of my comments below are based on the premise that we are talking about “blind” applicants, people who have responded to a posting or people with CVs up on a job board.

Can 199 out of 200 programmers actually write programs?

Let’s perform this simple thought experiment (if you know anyone who is single and actively seeking a partner, they can carry out the experiment for real and report their results to us).

We register with a dating or personals site. We use the search facility to perform a fairly simple screen for “qualifications” like age, gender and gender orientation (presuming you care). Now, we contact every single person matching our search. How many are worth a second date? How many are, let’s be honest, completely unready to date?

After a few months of “dating,” people can become very cynical. “Where are all the good men/women/womyn?” They plaintively ask. The answer, we all know, is that good people are rarely looking. They are already in relationships. If they become single, after a reasonable period to recover from their experience, they date very briefly and then get into another relationship. They have a good network of friends and family that introduce them to eligible mates, so they have very little need for dating web sites.

People with “issues,” on the other hand, spend a lot of time dating and searching for mates but relatively less time in relationships. Their friends and family are reluctant to introduce them to potential partners because, deep down, their friends and family know they are not ready.

The net result is that on a dating site, the majority of the people looking for mates are not the people you want to meet. And if you try to judge people as a whole from dating sites, you will get a poorly skewed distribution. It is not a representative sample.

The fact is, the vast majority of people are wonderful partners. You cannot judge people as a whole from the people you find looking for blind dates.

And the vast majority of programmers can program.

It’s understandable that someone interviewing programmers from the available pool (like résumé boards or blind submissions to the company’s jobs@bigco.com email address) will become cynical and wonder “where are all the good programmers?” But it’s wrong to conclude that the vast majority of programmers are incompetent based on observations of people looking for programming jobs.

Is a screening question mandatory?

If there is no reason to worry that someone may not be an outstanding, qualified programmer, it is appropriate to screen them before investing time and energy in a full interview. Just as an applicant ought to be carefully screening the companies where she submits her CV. Are there alternatives to a screening question like Fizzbuzz? Of course there are.

Before I touch on a few, let me describe one that is not. Years of experience indicated on the CV, even if not falsified, is not evidence that an applicant can actually perform the work in a competent fashion. They must still be screened and interviewed.

It’s a paradox related to the dating analogy: if someone has five or more years of experience, why are they looking for a job? Why haven’t they compiled a network of friends and contacts? For five straight years recruiters have been calling them every week offering them a job. Why haven’t the recruiters found them a new position?

This person may be an incredible find who is looking due to unusual circumstances. Or they may be a wonderful developer with poor networking skills. But they may also be incompetent, and that’s the reason their friends, contacts, and recruiters haven’t placed them in a job.

So what alternatives do we have to asking a screening question? The best alternatives are entirely within the candidate’s control.

First, a candidate can belong to the “Insider’s Club.” Very simply, candidates who do not wish to be asked screening questions should only go on interviews where there has been a warm, personal referral from a trusted friend. I don’t mean Joe told you that Bob’s company is hiring and Bob sends your CV to HR. I mean you know Greta well and she takes you to lunch with Gwen, who is the hiring manager.

Not everyone is a great people person. That’s why it helps to have nature or nurture on your side. But everyone can become a better networker through practice. And nearly everyone should. Not just to avoid screening questions, but because a regular lunch with Greta is fun, and playing Ultimate with Joe’s team on Tuesday nights is a good workout.

If an interviewer still insists on screening such warm referrals… I don’t know what to say about that.

Second, I like your proposal that candidates bring samples of their work to interviews:

Having candidates bring in samples of code and having the interviewer and interview team review such, and question why decisions were or weren’t made is an excellent way of getting insight into the person’s problem solving skills.

For the last decade I have carried a portfolio to interviews with code samples, product brochures and awards, references, and other opportunities for the type of interaction you suggest. I would love it if every candidate brings such a portfolio to me when I’m interviewing them.

Even though I suggest it in nearly every invitation to an interview, most candidates show up with nothing except an eager expression on their face. I wish they would listen to you on this subject. No matter, by the time they have arrived I have already Googled them to see what they have on their home page, what they’ve written on their blog, and what they’ve contributed to OSS. Just as you suggest.

Alas, I often come up empty. (Not always: one candidate arrived with a book… that he had written! Another brought an actual shrink-wrapped Adobe product he had a part in developing). But note that you have a blog… I would never need to ask you about Fizzbuzz, I would much rather ask you about what you have written.1

So candidates can avoid a lot of screening questions by blogging, contributing to open source, participating intelligently in on line professional forums, and bringing samples to interviews.

If an interviewer wants to ask me to write Fizzbuzz after reviewing my code that implements tail call optimization in Scheme… let’s just say that I will be amused.

Root of minus one, what I actually do in many cases, especially with new graduates, is mail people a problem. They can work on it in their own time without any pressure and get back to me when they have a solution. I have posted one such problem on my blog: designing a deck of cards.

I’m still waiting for the candidate who has Googled me before an interview and comes prepared with code for a deck of cards. That person I would hire: even if they copied the code off the internet, they have initiative and will make a contribution somewhere!

The work at home problem is still used to screen people out, so it’s kind of unfair to lump it under “ways to avoid a screening question.” But it isn’t a screening question in the sense that I plan on drill down and ask candidates questions about their code in a face to face meeting.

In some ways it’s worse than a screening question in the sense that it requires a larger investment on the part of the candidate and there’s more pressure to do well. It’s not just enough that it work, it must somehow please the interviewer.

I personally find that tough, more on that a few headings below.

Is the screening question a significant predictor of talent?

If someone doesn’t have a warm referral, and they don’t have the type of material you suggest, I need to know if they’re the person on their CV. Something dead simple like Fizzbuzz is impossible to screw up in an interview. I actually don’t believe you would fail this in an actual interview, even with your disclaimer.

But let’s pretend that I was too busy to Google you and you forgot to bring some samples and I ask you to write some code or diagram something. Let me be very clear: I’m not judging you on how well you write it, or whether you are nervous. Or even a trivial “nit” like whether you start at zero or forget that numbers divisible by fifteen should print “fizzbuzz.”

As Shelley points out, NASA, the military, and other organizations training people for high risks jobs spend so much time in simulation because they want the tasks to be so ingrained that in a stress situation, the people’s responses are almost automatic.

Programming Interviews Exposed takes you through actual programmer interview situations—fifty of them—so that you’ll be prepared to win the job. If you want to ace your next interview, start with this book.

And I’m not going to put you on a pedestal if you write it in Haskell, or use recursion, or somehow torture Perl’s regular expression engine into doing the work. Although some interviewers insist that you write such tests in the lingua franca of their office, I always ask a candidate to write programming tests in whatever language they find easiest. PHP is acceptable.

All I’m trying to do is separate you from the people who don’t know that you want a loop (or recursion, or a range initializer, or whatever) and can’t even get started. This is a very Boolean gate.

When it’s done we’re not going to have along chat about your approach. I may even stop you before you’re finished if it looks like you know what you’re doing. I don’t want to waste time on it: once I know you aren’t impersonating the person on your CV, I want to forget about Fizzbuzz as fast as humanly possible and get to the real interview.

So really, the important thing about a screening question is to remember that it’s only a screening question. It’s not a be-all and end-all opportunity to judge a candidate’s coding style, or ability to elicit requirements, or cultural compatibility.

I’m not going to narrow my eyes into slits and demand that you “find the bug.” I’ll glance over your one page of code and then turn it face down. Or I’ll erase the whiteboard. And move on.

Do good people really get anxious enough to fail screening questions?

I agree that good people can fail a screening test from time to time. Do you want to hear a story about the time I was given a lukewarm referral into a company and somehow I was unable to write a program to print out the first one hundred prime numbers? My contact cursed me for choosing that day to have a brain cramp and devaluing his credibility as a judge of talent. We all have our days… this kind of thing keeps my feet firmly on the ground.

Every single technique has some kind of false negative. Interviewing is a statistical model: you employ a number of techniques in the aim of making sure you don’t accidentally toss out a great person or hire a duffer by mistake.

From my experience, the most important thing is not to fall prey to the mistake of making up your mind in the opening sixty seconds of an interview and using the remaining time to justify your prejudices. There are people with my experience—perhaps even me—who are duds. There are people with a funny accent or perhaps with experience in a programming language you despise who are brilliant thinkers and excellent communicators.

A good interview requires an open mind.

This is why the title of my own post on the subject of Fizzbuzz was Don’t overthink Fizzbuzz. It’s just one extremely brief part of a process that takes multiple meetings and many hours when I am looking for a great team mate. But that being said, I still look for people who can finish the problem, no matter how elegant or haphazardly they do it.

I can tell you what frightens me more than another “print the first one hundred primes” problem:

What frightens me is an interviewer cross-examining me with no apparent system or objective. Perhaps someone who has already pre-judged me as superficial because I brought code samples on a Macintosh instead of a Dell. Or who ignores my Java experience to spend the entire interview grilling me about how I could use an allegedly “unsafe” language like Ruby.

I personally find testing of any kind—whether a formal test like a programming problem or an informal test like an interview—produces anxiety. What increases the anxiety for me is not knowing how the test is graded. It’s like being asked to write a program without requirements from the client. That’s very stressful.

Quite honestly, this is a major flaw with the way some people administer screening questions. They ask you to write a program that does such-and-such. But they don’t say anything else. Are they looking for unit tests? Build scripts? Do they want golf-like brevity of code? Or do they want you to baby. talk. the. code. so. that. any. body. can. mouth. the. symbols. one. by. irritating. one.? Do they want comments from top to bottom? Or just comments to explain why not what?

Faced with this anxiety, a lot of candidates feel like they are playing “Guess the program I’m thinking of.” They worry that the interviewer is thinking of the one perfect answer (I wonder if it’s by strange coïncidence exactly what the interviewer would write themselves?). The candidate can think of a few things to write, but the main problem is choosing the right thing to write.

I personally think that the anxiety can be reduced by being very specific about the problem and what is being tested. I think it’s not just anxiety-provoking but outright harmful (to the interviewee and to the interviewer) to say little or nothing beyond the desired output (fizz… buzz… primes…) but attempt to draw conclusions about a candidate based on the code they wrote on the spur of the moment with no further guidance.

I agree it’s ridiculous to expect that the code for Fizzbuzz would resemble production code for a mission-critical web application, and one of the sources of anxiety for some candidates might be trying to “complexify” such a toy program to make it look like it belongs on BigCo’s heavy iron.

Thank you

Once again, thank you for taking the time to remind everyone not to glorify screener questions like Fizzbuzz. Although I have written this partly in support for your points and partly in defence of my own interviewing practices, your post has reminded me to be extra observant the next time I am interviewing. I do not want to let a good person become nervous: even if they “pass,” the extra pressure will put the meeting off stride.

I also want to thank you for your implied criticism of the tone of many blog posts. Yes, I have been frustrated with the quality of applicants the last few times I have been interviewing. But nevertheless, I need to be more clear about the difference between saying that 199 out of 200 applicants cannot code and 199 out of 200 working programmers cannot code.

Cordially,

Reginald Braithwaite

p.s. I am going on my very first diving holiday later this month. Your links to the ocean’s wonders are a timely discovery. Thanks! I am going with my partner, whom I happened to meet on a dating site! So I know that good people are looking and that they deserve the benefit of the doubt until proven otherwise, no matter what the previous 199 dates were like.

For example (and I say this gently, I am very pleased with your post), I might probe into whether you are the type who “does not suffer fools gladly,” based on the use of phrases like “arrogant twats” and “idiocy travels up the food chain” in your own post. Just as, I am sure, you would have plenty of questions to ask me if you took a few minutes to peruse my own blog.

Thanks... I was actually kidding, I am a little hard core about software development. If we establish that you have an outstanding track record for shipping great software, what do I care about whether you keep your pen sharpened?

Of course, some behaviours, like arrogance, are costly to a team. Usually this reveals itself over time in the results.

Your track record suggests that an interviewer has far better things to discuss with you than what you think of people who ask candidates to simulate children's nursery games :-)

Having just recently discovered your blog in the past couple days thanks to Fizzbuzz, I've placed it on my list of blogs to read.

Personally, I despise tests during interviews, I never feel like they are true measures of my ability and worth, and invariably I come up with a 10 times better answer while I'm driving home from the interview.

At least I'm now prepared for any interviewer who asks me to code Fizzbuzz, I'm going to compile a portfolio of Fizzbuzz solutions in every known language just by googling, they're all out there now. :) Might just put in a few samples of some of those other questions (got any code to calculate prime numbers?)

Seriously, though, the portfolio of code/projects is a great concept and I think I may just create on, though I have no immediate plans on leaving where I'm at now, it could prove useful in the future.

I don't know about other tests, but I know that I have no idea how to measure anything except "can this person actually code something simple" with a problem like Fizzbuzz.

But it's still a measure of ability, the ability to code.

Might be useful when doing an interview at the entry level on a kid fresh out of school, but otherwise I'd be wary personally if I felt inclined to give it to someone showing experience on their resume. Someone who engenders that sort of doubt is probably a candidate to avoid.

I recommend the portfolio, even if you aren't looking. The act of putting it together is also useful for planning your career inside your current organization.

I will do so, if for no other reason than to have it ready just in case. One thing I have learned over the years, you may be paid lots of money and have everyone tell you that you're indispensable but when push comes to shove, they'll drop you like a stone if the circumstances are right.

I just wish I had some samples of stuff I did 15 years ago, which is a bit different from what I do now but still applicable to a portfolio. (43 years old and spent over half of that as a programmer, 25 years more or less, that portfolio could be a mult-volume set)

"It’s a paradox related to the dating analogy: if someone has five or more years of experience, why are they looking for a job? Why haven’t they compiled a network of friends and contacts? For five straight years recruiters have been calling them every week offering them a job. Why haven’t the recruiters found them a new position?"

Here again, you're making assumptions based on your personal experience, and nothing more. There could any number of reasons that someone with more five years of experience is actually, I don't know, applying for a job rather than hitting up their friends--including moving to another area, changing job types, or being among friends all of whom are looking for work because the entire department just got outsourced to India.

You're basing so much of your assumptions and criticisms of people and the industry based on a very narrow view of it. Narrow, and somewhat intolerant view.

Perhaps the real issue isn't that the people can't program, but that you can't interview.

if someone has five or more years of experience, why are they looking for a job? Why haven’t they compiled a network of friends and contacts? For five straight years recruiters have been calling them every week offering them a job. Why haven’t the recruiters found them a new position?

That's odd, the very next sentence on my copy of this blog post reads "This person may be an incredible find who is looking due to unusual circumstances."

Few people know less about software development than Joel Spolsky. He's like Dvorak-- pronouncements generated out of pure ignorance.

Reginald-- you have already stated that anyone who sends you a resume is not a really good programmer. And in doing so you have exhibited the primary trait that makes for bad programmers, and the trait that I screen for when interviewing:

Arrogance.

Give me a humble beginner over a joel spolsky any day. And I know I'll ship faster with better quality code.

Great post. It's definitely important to distinguish between software developers who are *interviewing*, and the *entire profession* of software development. I thought we were both pretty clear about that in our respective entries.

Still, I find it mightily discouraging that so many programmers would come to a job interview and fail to write FizzBuzz. It's not a meaningful challenge, nor is it meant to be one. It's like asking a truck driver if he knows where the gas pedal and gearshift are. If you find this offensive, it shouldn't be offensive because it's an arbitrary test-- it should be offensive because it implies the field has almost no barriers to entry.

So from that perspective, I don't really understand Shelley's problem with it.

What I believe some reasonable people are saying is that there are other ways of filtering people who cannot program, and that one or more other ways are superior to asking a candidate to write Fizzbuzz in an interview.

I agree there are other ways, and I have named a couple in this post that I prefer. I certainly believe that I can make the same determination of whether someone has Fizzbuzz competence without actually asking them to write the program.

And if you read my words very carefully, you will realize that I never said that I require people to write Fizzbuzz, but that I said "let's pretend that I ask it in an interview."

I think we all agree that people ought to be able to write Fizzbuzz, and the only things to debate are:

1. How many people applying for jobs can or cannot, and;

2. Whether the best course of action as an interviewer to ask candidates to write it in an interview.

Thanks for your concern. But please don't worry, I am not denying my argument. Although the post you mentioned was really an observation, not an argument.

But to the point in question, I never actually said that I insist that every interviewee do something as simple as Fizzbuzz.

I do ask something like that (but not Fizzbuzz) of a few candidates who are completely cold: I don't know them and I don't know the companies where they've worked.

As indicated in another blog post, I ask some candidates to do a homework problem.

But many of the people I interview come to me through my blog or network. There is no need to screen such people at that level.

With those people, we will eventually get to Monopoly or something like that couched in business terms ("If you were designing a _____ to be used by _____ under _____ conditions, what would you consider important? How would you approach it?").

So in actual fact, everyone I interview has to do some kind of juggling at some point or another, but I try to find the most appropriate path for each candidate.

On to a point Jeff Atwood has made. The issue with Fizzbuzz is not the problem itself, or whether we should use that exact screening test, but why there are a large (debate amongst yourselves how large) number of job applicants who lack the skills to solve it.

I have been known to ask questions as simple as fizzbuzz during a phone screen, because these questions actually _do_ screen. Within ten minutes, I can tell whether someone has written any substantial amount of Java in the last year or two.

Of the fifty-some-odd candidates I have screened, about a fifth have no detectable skill at Java, despite a strong presence on the resume. I can accept someone who wants to change languages, and will usually give them a good chance. I can also accept someone who is a bit rusty after a year of Perl. I cannot accept someone unable to write up something a step above Hello World when they claim a decade of experience in the language.

Someone who has done everything in their graphical IDE and not written a single command line program might have forgotten the signature for the main method. They are still going to be able to declare a method signature, declare a variable or two, and write up a simple loop through a data structure. This is not rocket science; this is something worth five minutes of a phone screen, and not much more.

My alma mater teaches Java in the first year, and C++ for the next three. New grads coming to interview usually do the questions in C++, and that is just fine. They rarely are foolish enough to claim four years of experience just because they used the language for a year three years ago.

In short - interviewers ask questions like this because we have seen that they tend to filter out the people who either cannot code, or who lied on their resume. Not 'stretched the truth a bit', but out and out claimed familiarity with something they do not know. We have no way of knowing that a given candidate is one of the good ones, so we ask these foolish-appearing questions even of the good candidates, and then move on to the ones that tell us 'how good' if the first question works out.

It's like asking a truck driver if he knows where the gas pedal and gearshift are.

During the interview for my current job, I was asked to write an output line in C# that said "Hello World." Out of all the candidates who applied for the posision, I was (I was later told) the only one who did it correctly. Most wouldn't even try. This is out of only six other people, but it still opened my eyes.

The way I see it, you have to get some kind of sense of their ability somehow. Reading the resume just won't cut it.

I'm not strictly a developer (I'm an actuary) but we do lots of software work.

By very different paths we ended up at a similar place.

Interviews and such are obviously the first step. Unfortunately we got our fingers burnt quite badly with someone who claimed skills that it turned out he just didn't have. In fact he struggled with basic computer concepts (such as the difference between values and the formatting that represent those values on the screen). This from a supposedly experienced VBA developer.

What we ended up with was the normal interview process. We would then make up a short list. Two or three candidates would make it to this stage and we would set up a custom test based on the candidate's supposed skills. The first tests would be something they should know - so an sysadmin would have to set up a time-limited account for himself. Then there would be a set of short tasks that get more and more difficult. Failing a task doesn't necessarily fail the interview. I normally sit looking over the person's should and see what they get up to. A good sign is when they find their way to the documentation or the internet (googling is a skill). A bad sign is when they bang their heads repeatedly on the same obstacles like a fly trying to fly through the window pane.

The point I'm trying to make is that we learned that we need to see how people work and how they approach problems in some real world situations. Those tests tend to take a lot of time to set up but we have found that it was completely worth it

Just yesterday I was given a slightly more complex variant of the fizzbuzz program.

I had to say what the program was outputting.

Such a task is poorly thought through and puts unneccessary pressure on the interviewee.

There is *nothing* wrong with posing technical tests, but they need to be carefully thought out.

Do you as employer want someone who can work effectively in an environment where:

- two strangers are staring at him (as was the case for me yesterday) - only the most cursory explanation of the task is given (are there any catches?/typos) - you have to read code from the printed page in Arial.

...Or do you want someone who has a demonstrable ability to learn, strong technical skills, can communicate well and work as part of a team.

I have a degree in Computer Science from a well respected university, and I have developed complex software systems before at a well respected company (as shown on my CV).

Frankly such a test says more about the company than the candidate.

The irony is, of course, I completed the test fine, but I could still *feel* I had failed it.

Maybe I took too long. Maybe my approach was not to their liking.

Time will tell, but these gimmicky tests serve only as a focus for recruiters' insecurities and unfounded bias, leading to poor recruiting decisions.

Do you as employer want someone who can work effectively in an environment where:

- two strangers are staring at him (as was the case for me yesterday)- only the most cursory explanation of the task is given (are there any catches?/typos)- you have to read code from the printed page in Arial.

...Or do you want someone who has a demonstrable ability to learn, strong technical skills, can communicate well and work as part of a team.

I cannot discuss a test that someone else gave you, I have no idea whether it was something I would use or not. However, I can say without hesitation that when hiring developers:

I want someone who can write a simple program with extremely obvious requirements and no hidden catches like FizzBuzz in an environment where I am observing the process,

and,

I you want someone who has a demonstrable ability to learn, strong technical skills, can communicate well and work as part of a team.